Python 3.2+concurrent.futuresterm

concurrent.futures.ProcessPoolExecutor

process pool

A pool of processes from concurrent.futures. It gets around the GIL — every process has its own interpreter. Use it for CPU-intensive work.

Syntax

with ProcessPoolExecutor(max_workers=N) as ex:
    future = ex.submit(fn, *args)
    results = ex.map(fn, iterable)

Arguments and return values travel between processes through pickle, so lambdas, nested functions and unpicklable objects cannot be passed — the target function must live at module top level. On Windows and macOS processes start via spawn, and on Linux since 3.14 via forkserver by default, meaning the child re-imports your module: without an if __name__ == '__main__' guard the pool breaks — the children fail with RuntimeError and the parent gets BrokenProcessPool. Process startup plus serialization is expensive too, so for small or I/O-bound work ThreadPoolExecutor will come out ahead.

Examples

from concurrent.futures import ProcessPoolExecutor
def heavy(n):
    return sum(range(n))
if __name__ == '__main__':
    with ProcessPoolExecutor() as ex:
        results = list(ex.map(heavy, [10**6] * 4))
        print(results)

See also

Open in the full glossarydocs.python.orgReport an inaccuracy