Skip to content

Add concurrent.futures.InterpreterPoolExecutor #124694

Closed
Closed
Add concurrent.futures.InterpreterPoolExecutor#124694
Assignees
ericsnowcurrently
Labels
3.14bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytopic-subinterpreterstype-featureA feature request or enhancement

Description

@ericsnowcurrently
ericsnowcurrently
opened on Sep 27, 2024

Feature or enhancement

Proposal:

While we wait on PEP 734, there's nothing blocking us from adding a new executor for multiple interpreters. Doing so would allow people to start trying them out. (I wish I had thought of this for 3.13!)

My planned design:

  • mostly built on top of ThreadPoolExecutor
  • callables (and args) passed as initializer or to submit() or map() will be pickled (same as ProcessPoolExecutor)
  • initializer and submit() arg can be a script instead of a callable (a script for map() doesn't make sense since there are no args)
  • __init__() will take a new "shared" arg that corresponds to what Interpreter.prepare_main() takes in PEP 734, to allow each worker interpreter to share some data

In the future we could support a more efficient "sharing" scheme than pickle. When (or if) PEP 734 is accepted, we can refactor the executor to use the interpreters module rather than using _interpreters directly.

Open questions:

  • reconstitute and raise uncaught exceptions from running in the interpreter (like we do for ProcessPoolExecutor)?
  • (optionally?) exec functions against main instead of the their original module, i.e. use the body of the function as a script?

CC @brianquinlan

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

Activity

  1. ericsnowcurrently
    added
    type-featureA feature request or enhancement
    stdlibStandard Library Python modules in the Lib/ directory
    topic-subinterpreters
    3.14bugs and security fixes
    on Sep 27, 2024
  2. ericsnowcurrently
    self-assigned this
    on Sep 27, 2024
  3. github-project-automation
    added this to Subinterpreterson Sep 27, 2024
  4. github-project-automation
    moved this to Todo in Subinterpreterson Sep 27, 2024
  5. ericsnowcurrently
    added a commit that references this issue on Oct 16, 2024

    gh-124694: Add concurrent.futures.InterpreterPoolExecutor (gh-124548)

    a5a7f5e
  6. ericsnowcurrently
    closed this as completedon Oct 16, 2024
  7. github-project-automation
    moved this from Todo to Done in Subinterpreterson Oct 16, 2024
  8. ericsnowcurrently
    added a commit that references this issue on Oct 18, 2024

    gh-124694: In test_interpreter_pool, Restore the Asyncio Event Loop P…

    322f14e
  9. ebonnal
    added 2 commits that reference this issue on Jan 12, 2025

    pythongh-124694: Add concurrent.futures.InterpreterPoolExecutor (pyth…

    843c78d

    pythongh-124694: In test_interpreter_pool, Restore the Asyncio Event …

    128a602
  10. donbarbos
    added a commit that references this issue on May 20, 2025

    pythongh-124694: Document missing ctxkwargs argument in ThreadPoolExe…

    837ead9

  11. henryiii commented on Jul 31, 2025

    @henryiii
    henryiii
    on Jul 31, 2025
    Contributor

    Does the pickling work? I've been trying several things on pickable arguments, but I keep getting concurrent.interpreters.NotShareableError: args not shareable. My first thought was this:

    # /// script
# dependencies = ["numpy"]
# requires-python = ">=3.14"
# ///

import numpy as np

from concurrent.futures import InterpreterPoolExecutor


def identity(*args):
    return args[0]

if __name__ == "__main__":
    x = np.random.rand(10003)
    xs = np.array_split(x, 4)
    with InterpreterPoolExecutor(max_workers=4) as pool:
        results = pool.map(identity, xs)
        final = np.concat(results)

    assert np.all(x == final)

    But I get that error. Then I tried explicitly pickling:

    # /// script
# dependencies = ["numpy"]
# requires-python = ">=3.14"
# ///

import numpy as np
import pickle

from concurrent.futures import InterpreterPoolExecutor


def identity(*args):
    return pickle.dumps(pickle.loads(args[0]))

if __name__ == "__main__":
    x = np.random.rand(10003)
    xs = np.array_split(x, 4)
    xsp = (pickle.dumps(a) for a in xs)
    with InterpreterPoolExecutor(max_workers=4) as pool:
        results = pool.map(identity, xsp)
        final = np.concat([pickle.loads(r) for r in results])

    assert np.all(x == final)

    But I still get this error:

    Exception: AttributeError: module '__main__' has no attribute 'identity'

The above exception was the direct cause of the following exception:

concurrent.interpreters.NotShareableError: object could not be unpickled

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/henryschreiner/git/scikit-hep/boost-histogram/example.py", line 21, in <module>
    final = np.concat([pickle.loads(r) for r in results])
                                                ^^^^^^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 639, in result_iterator
    yield _result_or_cancel(fs.pop())
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 311, in _result_or_cancel
    return fut.result(timeout)
           ~~~~~~~~~~^^^^^^^^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 450, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 395, in __get_result
    raise self._exception
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/thread.py", line 86, in run
    result = ctx.run(self.task)
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/interpreter.py", line 85, in run
    return self.interp.call(do_call, self.results, *task)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/interpreters/__init__.py", line 233, in call
    return self._call(callable, args, kwargs)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/interpreters/__init__.py", line 215, in _call
    res, excinfo = _interpreters.call(self._id, callable, args, kwargs, restrict=True)
                   ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
concurrent.interpreters.NotShareableError: args not shareable

    Also, I'm not sure why the Exception: AttributeError: module '__main__' has no attribute 'identity' without a traceback is at the top. Other executors are fine with code.

    I was actually trying to use this in a test for boost-histogram, trying to pass a numpy array in and a boost-histogram object back out, but it fails even with byte strings from pickle above.

    Edit: I do think it's an issue with the import like seen at the top, and the traceback is a red herring. The tests version of this list Exception: ModuleNotFoundError: No module named 'tests' at the top. So it's not able to import the function to run. I thought __main__ support was fixed?

    It seems the subinterpreter's don't inherit the current sys.path? Using pytest 7's pythonpath setting doesn't work, they aren't picking up the added test paths.

    If I put the function inside my package, I get ImportError: NumPy: dtype is already registered which I think that's occurring when pybind11 is registering my dtype. I've opened an issue pybind/pybind11#5781; if I manually skip over that, it looks like numpy doesn't support subinterpreters yet, so I'd not really be able to do much with it yet.

  12. zware
    added 2 commits that reference this issue on Oct 22, 2025

    pythongh-124694: Add versionadded note to InterpreterPoolExecutor

    3ea6467

    gh-124694: Add versionadded note to InterpreterPoolExecutor (#140468)

    421a475
  13. miss-islington
    added a commit that references this issue on Oct 25, 2025

    pythongh-124694: Add versionadded note to InterpreterPoolExecutor (py…

    ea7e129
  14. zware
    added a commit that references this issue on Oct 25, 2025

    [3.14] gh-124694: Add versionadded note to InterpreterPoolExecutor (G…

    5961115
  15. StanFromIreland
    added a commit that references this issue on Dec 6, 2025

    pythongh-124694: Add versionadded note to InterpreterPoolExecutor (py…

    7bf047f
  16. ambv
    added 3 commits that reference this issue on Oct 23, 2026

    gh-124694: Document missing ctxkwargs argument in ThreadPoolExecutor (#…

    9a4187a

    [3.15] gh-124694: Document missing ctxkwargs argument in ThreadPoolEx…

    68ac574

    [3.14] gh-124694: Document missing ctxkwargs argument in ThreadPoolEx…

    be986a6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.14bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytopic-subinterpreterstype-featureA feature request or enhancement

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions