gh-64192: Make `imap()`/`imap_unordered()` in `multiprocessing.pool` actually lazy - #136871

Merged

encukou merged 35 commits into

python:mainfrom

obaltian:feature/add-buffersize-to-multiprocessing

Jul 29, 2026

Commits on Aug 14, 2025

draft: impl lazy input consumption in mp.Pool.imap(_unordered)

Oleksandr Baltian
authored and
obaltian
committed
Use semaphore to synchronize threads

Oleksandr Baltian
authored and
obaltian
committed
Update buffersize behavior to match concurrent.futures.Executor behavior

Oleksandr Baltian
authored and
obaltian
committed
Release all `buffersize_lock` obj from the parent thread when terminate

Oleksandr Baltian
authored and
obaltian
committed
Add 2 basic `ThreadPool.imap()` tests w/ and w/o buffersize

Oleksandr Baltian
authored and
obaltian
committed
Fix accidental swap in imports

Oleksandr Baltian
authored and
obaltian
committed
clear Pool._taskqueue_buffersize_semaphores safely

Oleksandr Baltian
authored and
obaltian
committed
Slightly optimize Pool._taskqueue_buffersize_semaphores terminate

Oleksandr Baltian
authored and
obaltian
committed
Rename `Pool.imap()` buffersize-related tests

Oleksandr Baltian
authored and
obaltian
committed
Fix typo in `IMapIterator.__init__()`

Oleksandr Baltian
authored and
obaltian
committed
Add tests for buffersize combinations with other kwargs

Oleksandr Baltian
authored and
obaltian
committed
Remove if-branch in `_terminate_pool`

Oleksandr Baltian
authored and
obaltian
committed
Add more edge-case tests for `imap` and `imap_unodered`

Oleksandr Baltian
authored and
obaltian
committed
Split inf iterable test for `imap` and `imap_unordered`

Oleksandr Baltian
authored and
obaltian
committed
Add doc for `buffersize` argument of `imap` and `imap_unordered`

Oleksandr Baltian
authored and
obaltian
committed
add *versionadded* for `imap_unordered`

Oleksandr Baltian
authored and
obaltian
committed
Remove ambiguity in `buffersize` description.

Oleksandr Baltian
authored and
obaltian
committed
Set *versionadded* as next in docs

Oleksandr Baltian
authored and
obaltian
committed
Add whatsnew entry

Oleksandr Baltian
authored and
obaltian
committed
Fix aggreed comments on code formatting/minor refactoring

Oleksandr Baltian
authored and
obaltian
committed
Remove `imap` and `imap_unordered` body code duplication

Oleksandr Baltian
authored and
obaltian
committed

Commits on Aug 31, 2025

Merge branch 'main' into feature/add-buffersize-to-multiprocessing

obaltian
authored and
Oleksandr Baltian
committed

Commits on Sep 3, 2025

Merge branch 'main' into feature/add-buffersize-to-multiprocessing
obaltian
authored

Commits on Jul 17, 2026

Commits on Jul 18, 2026

Update Lib/multiprocessing/pool.py

obaltian
and
encukou
authored
Update Doc/library/multiprocessing.rst

obaltian
and
encukou
authored
Apply suggestion from @encukou

obaltian
and
encukou
authored
Make `buffersize` argument kwarg-only
Sasha Baltian
committed
Use set for `._taskqueue_buffersize_semaphores`
Sasha Baltian
committed
Make `buffersize` arg check inline
Sasha Baltian
committed
Simplify `.imap()` and `.imap_undered()` tests
Sasha Baltian
committed
Fix iterator instantiation in in `.imap()`
Sasha Baltian
committed
Mark kw-only arg in `imap`/`imap_unordered` in doc
Sasha Baltian
committed
Merge branch 'main' into feature/add-buffersize-to-multiprocessing
obaltian
authored