all Python 3multiprocessingterm

multiprocessing.Pool

pools and managers

A pool of worker processes: map(), imap(), starmap() and apply_async() hand out tasks and collect results. The classic way to run a function over data in parallel.

Syntax

with ctx.Pool(processes=None) as pool: pool.map(func, iterable, chunksize=None)

Leaving with calls terminate(), not close() and join(): apply_async results not collected yet are lost, so fetch them inside the block. The function and its arguments go through pickle, so a lambda will not work; an exception in a worker is re-raised in the parent by .get(). For new code concurrent.futures.ProcessPoolExecutor is often handier: Future objects, asyncio integration and BrokenProcessPool when a worker dies (Pool may hang then). Pool wins where you need imap_unordered, maxtasksperchild or chunksize.

Examples

import multiprocessing as mp
def square(x):
    return x * x
if __name__ == '__main__':
    with mp.get_context('spawn').Pool(2) as pool:
        print(pool.map(square, range(5)))  # → [0, 1, 4, 9, 16]
        print(pool.starmap(pow, [(2, 3), (3, 2)]))  # → [8, 9]

import multiprocessing as mp
if __name__ == '__main__':
    with mp.get_context('spawn').Pool(2) as pool:
        r = pool.apply_async(divmod, (1, 0))  # падает в работнике
        try:
            r.get(timeout=10)
        except ZeroDivisionError as e:
            print('raised in parent:', type(e).__name__)  # → raised in parent: ZeroDivisionError

import multiprocessing as mp
import time
def slow():
    time.sleep(2)
    return 'done'
if __name__ == '__main__':
    with mp.get_context('spawn').Pool(2) as pool:
        r = pool.apply_async(slow)
    print(r.ready())  # → False — выход из with сделал terminate()

multiprocessing.Pool

Syntax

Examples

See also