import time from concurrent.futures import ThreadPoolExecutor, as_completed def foo(a): time.sleep(a) return a def calc(): with ThreadPoolExecutor(max_workers=500) as pool: results = [pool.submit(foo, 1), pool.submit(foo, 10)] for future in as_completed(results): return future.result() print(calc())

In this code, I put two function calls foo in the pool, which as a parameter receive the number of seconds they need to sleep.

This code will display 1 , as the result of the very first executable function.

But the problem is that we get this result only after 10 seconds, until the second function is executed.

How do I make such a functional: all calling functions in this pool should be completed until the end, but as soon as we received at least one answer, the calc function should immediately terminate without waiting for all calls in the pool to be completed.

    1 answer 1

    with statement clears up for ThreadPoolExecutor : calls pool.shutdown(wait=True) on the output. She cannot return until the tasks in the pool are executed.

    To fix this, you need to replace return with yield and use print(next(calc())) .

    If subsequent results are not needed, then it is necessary to provide for the possibility of canceling already running tasks.

    Obvious code with concurrent.futures.ThreadPoolExecutor did not work (see code at the end of the question). Here is a similar example c multiprocessing ThreadPool that works:

     #!/urs/bin/env python3 import contextlib import logging import time from multiprocessing.pool import ThreadPool as Pool @contextlib.contextmanager def logged(message): logging.info('before {}'.format(message)) try: yield finally: logging.info('after {}'.format(message)) def foo(a): with logged('sleep'): time.sleep(a) return a def calc(): with Pool(2) as pool: for result in pool.imap_unordered(foo, [10, 1]): with logged('result'): yield result logging.basicConfig(format="%(relativeCreated)5d %(threadName)s %(message)s", level=logging.DEBUG) with logged('calc'): print(next(calc()), flush=True)

    Result

      4 MainThread before calc 8 Thread-2 before sleep 8 Thread-1 before sleep 1010 Thread-1 after sleep 1010 MainThread before result 1010 MainThread after result 1 1109 MainThread after calc

    The unit is shown after one second. Subsequent tasks canceled.

    Non-working code with concurrent.futures pool:

     #!/urs/bin/env python3 import contextlib import logging import time from concurrent.futures import ThreadPoolExecutor as Pool, as_completed @contextlib.contextmanager def logged(message): logging.info('before {}'.format(message)) try: yield finally: logging.info('after {}'.format(message)) def foo(a): with logged('sleep'): time.sleep(a) return a def calc(): with Pool(2) as pool: for future in as_completed(pool.submit(foo, i) for i in [10, 1]): with logged('result'): yield future.result() logging.basicConfig(format="%(relativeCreated)5d %(threadName)s %(message)s", level=logging.DEBUG) with logged('calc'): print(next(calc()), flush=True)

    Result

      43 MainThread before calc 43 Thread-1 before sleep 44 Thread-2 before sleep 1045 Thread-2 after sleep 1045 MainThread before result 1046 MainThread after result 10053 Thread-1 after sleep 1 10054 MainThread after calc

    The unit is shown only after 10 seconds.

    The variant with multiprocessing ThreadPool works in this case even if you use return . with statement calls pool.terminate() in this case:

     #!/urs/bin/env python3 import contextlib import logging import time from multiprocessing.pool import ThreadPool as Pool @contextlib.contextmanager def logged(message): logging.info('before {}'.format(message)) try: yield finally: logging.info('after {}'.format(message)) def foo(a): with logged('sleep'): time.sleep(a) return a def calc(): with Pool(2) as pool: for result in pool.imap_unordered(foo, [10, 1]): with logged('result'): return result logging.basicConfig(format="%(relativeCreated)5d %(threadName)s %(message)s", level=logging.DEBUG) with logged('calc'): print(calc(), flush=True)

    Result

      10 MainThread before calc 13 Thread-2 before sleep 13 Thread-1 before sleep 1014 Thread-1 after sleep 1014 MainThread before result 1015 MainThread after result 1 1114 MainThread after calc

    The unit is shown after one second.