Pool.imap(buffersize=...) blocks all other tasks on the pool until the iterator is consumed

Bug report

Bug description:

While an imap() or imap_unordered() iterator created with buffersize is waiting to be consumed, no other task submitted to the same pool is sent to the workers. If the program needs that other work before it reads more of the iterator, it deadlocks.

import multiprocessing as mp

def sq(x):
    return x * x

def neg(x):
    return -x

if __name__ == '__main__':
    with mp.Pool(2) as pool:
        a = pool.imap(sq, range(10), buffersize=4)
        b = pool.imap(neg, range(10), buffersize=4)
        print(list(zip(a, b)))   # hangs forever

Without buffersize, this prints the ten pairs immediately.

A second form, with a single buffered iterator:

with mp.Pool(2) as pool:
    a = pool.imap(sq, range(10), buffersize=4)
    pool.apply_async(sq, (7,)).get(timeout=5)   # raises TimeoutError

ThreadPool and imap_unordered() behave the same way.

Cause: the pool has a single task handler thread (Pool._handle_tasks). It iterates the generator returned by _guarded_task_generation(), and that generator blocks in sema.acquire() when the buffer is full. While it is blocked there, the thread cannot take anything else from _taskqueue.

Thread dump during the zip hang:

Thread [Thread-2 (_handle_tasks)]:
  File "Lib/threading.py", line 515 in acquire
  File "Lib/multiprocessing/pool.py", line 405 in _guarded_task_generation
  File "Lib/multiprocessing/pool.py", line 551 in _handle_tasks

Main thread:
  File "Lib/multiprocessing/pool.py", line 922 in next

In the zip example, next(b) waits for a task that is never submitted, and the task handler waits for a to be read, which never happens.

Possible directions:

Do not throttle inside the task handler thread. For example, submit the next item from the consumer's thread in IMapIterator.next(), similar to what Executor.map(buffersize=...) does in concurrent.futures.
Or keep the current design and document that a pool should not be shared between a buffersize iterator and other work.

I would like to work on this. Before I write a patch, I would like to know which direction maintainers prefer.

Related: gh-158675 (results dropped after close()), which comes from the same blocking sema.acquire() in the task handler.

buffersize was added in gh-64192 and is new in 3.16, so no released version is affected.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

gh-158677: Fix Pool.imap(buffersize=...) blocking other tasks on the pool #158792

@PrakharAgarwal17

@PrakharAgarwal17 thanks for looking into this.

I had a patch in progress too and have just opened it as a draft: gh-158792.

I started with the same approach as you, submitting from the consumer's thread, but moved away from it. next() then has to read the next input item before returning, so it can hold a ready result while the iterable is slow, and it deadlocks if the next item depends on that result being handled.

The draft keeps the iterable in the task handler thread and makes the handler skip an iterator whose buffer is full instead of waiting on it.

On your questions:

I left close() unchanged and kept Pool.imap(buffersize=...) silently drops results after close() #158675 separate, so each fix can be reviewed on its own.
With this approach, the question no longer comes up: the initial fill still happens in the task handler thread, so imap() returns immediately as before.

Your review and testing on the draft would be very welcome. ;)

@picnixz

@picnixz, I closed my draft PR gh-158792 so we can agree on the approach here first.

The approach I used there is different from what I originally suggested in the report.

The task handler keeps consuming the input, as it does today. If one iterator hits its buffer limit, the handler skips it for now, processes other work, and comes back to it once a result is consumed. So from the caller's side, nothing changes.

I initially tried having next() submit the next task, similar to Executor.map(buffersize=...). The issue is that this also makes next() consume the input in the caller's thread. If the input is slow, next() can block on the iterable even though a result is already ready.

I'll wait for your take before pushing anything.

pythongh-158677: Fix Pool.imap(buffersize=...) blocking other tasks o…

Navigation Menu

Repository navigation

`Pool.imap(buffersize=...)` blocks all other tasks on the pool until the iterator is consumed #158677

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Pool.imap(buffersize=...) blocks all other tasks on the pool until the iterator is consumed #158677

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

PrakharAgarwal17 commented on Oct 3, 2026

oyiakoumis commented on Oct 4, 2026

oyiakoumis commented on Oct 4, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`Pool.imap(buffersize=...)` blocks all other tasks on the pool until the iterator is consumed #158677