[tpython] Handle Python interpreter embedding in TPython, not cppjit by guitargeek · Pull Request #23616 · root-project/root

guitargeek · 2026-10-05T19:27:56Z

cppjit is a Python extension library, not an embedding library. Remove
the PyConfig/Py_InitializeFromConfig fallback from cpyrt's private
Initialize() and let it fail gracefully when the interpreter has not
been initialized by the embedding application. TPython::Initialize()
now owns interpreter embedding for the Cling-first case, replacing the
cpyrt::Scope_Check(nullptr) side-effect hack.

Along the same boundary, move the 'access to the python interpreter'
functions out of cpyrt entirely, since their only consumer is TPython
and they use none of cppjit's proxy machinery:

cpyrt::Exec/ExecScript/Prompt are inlined into TPython::Exec,
TPython::ExecScript and TPython::Prompt. This also removes the
duplicated static gMainDict in API.cxx; TPython now consistently
uses its own.
cpyrt::Import is inlined into TPython::Import, which additionally
stops leaking the module reference. The attach of the imported
module to the cppjit module (to prevent a proxy of the C++ proxy)
goes through a new small exported getter, cpyrt::GetThisModule().
The dead class-iteration loop (which computed a class name that was
never used) is dropped; namespace/class creation already happens on
the TPython side via TClass::GetClass.

Collateral changes:

Utility::IncludePython() now explicitly includes in the JIT
preamble, which previously arrived transitively through cpyrt/API.h.
Dropped test_api.py::test02_interpreter_access, which exercised the
removed cpyrt::Exec API.

🤖 Done with the help of AI

cppjit is a Python extension library, not an embedding library. Remove the PyConfig/Py_InitializeFromConfig fallback from cpyrt's private Initialize() and let it fail gracefully when the interpreter has not been initialized by the embedding application. TPython::Initialize() now owns interpreter embedding for the Cling-first case, replacing the cpyrt::Scope_Check(nullptr) side-effect hack. Along the same boundary, move the 'access to the python interpreter' functions out of cpyrt entirely, since their only consumer is TPython and they use none of cppjit's proxy machinery: - cpyrt::Exec/ExecScript/Prompt are inlined into TPython::Exec, TPython::ExecScript and TPython::Prompt. This also removes the duplicated static gMainDict in API.cxx; TPython now consistently uses its own. - cpyrt::Import is inlined into TPython::Import, which additionally stops leaking the module reference. The attach of the imported module to the cppjit module (to prevent a proxy of the C++ proxy) goes through a new small exported getter, cpyrt::GetThisModule(). The dead class-iteration loop (which computed a class name that was never used) is dropped; namespace/class creation already happens on the TPython side via TClass::GetClass. Collateral changes: - Utility::IncludePython() now explicitly includes <vector> in the JIT preamble, which previously arrived transitively through cpyrt/API.h. - Dropped test_api.py::test02_interpreter_access, which exercised the removed cpyrt::Exec API. 🤖 Done with the help of AI

- Deduplicate the PyGILRAII helper, which had one copy per TU, into a single private header, PyGILRAII.h. - Drop the write-only mainThreadState global: the PyEval_SaveThread() result was never restored because ROOT intentionally never finalizes the embedded interpreter; say so and discard it explicitly. - Warn in TPython::Import() when the cppjit module is unavailable and the imported module cannot be attached to it (previously a silent skip with a confusing double-proxy symptom downstream). - Fix a stale comment that still referred to cpyrt.h instead of cpyrt/API.h. 🤖 Done with the help of AI

The compiled ROOTTPython library was built with -DPy_LIMITED_API, but TPython now embeds the Python interpreter (PyConfig, Py_InitializeFromConfig) and serves the interactive prompt (PyRun_InteractiveLoop), none of which are part of the limited API: embedding interpreters is out of the stable ABI's scope by design. This broke all non-free-threaded, non-Windows CI builds, where the flag was active.

github-actions · 2026-10-05T22:44:57Z

Test Results

24 files 24 suites 3d 21h 8m 1s ⏱️
3 883 tests 3 881 ✅ 0 💤 2 ❌
83 112 runs 83 110 ✅ 0 💤 2 ❌

For more details on these failures, see this check.

Results for commit 9159ea6.

♻️ This comment has been updated with latest results.

vepadulano · 2026-10-06T07:01:38Z


 //
-// Access to the python interpreter and API onto cpyrt.
+// API onto cpyrt: access to cppjit proxies from Cling and C++.


Does cppjit want to have comments mentioning explicitly Cling?

It did before, the Cling name drop was just moved from API.cxx. But you're right, this is the moment to remove that mention.

vepadulano · 2026-10-06T07:04:15Z

+      // saved thread state is never restored: ROOT never finalizes the
+      // embedded interpreter.


// saved thread state is never restored: ROOT never finalizes the // embedded interpreter.

Has this always been the case? What are the implications?

This behavior dates from the thread-safety rework in 96fc4cc (Sept 2024). Before that, Initialize never released the GIL, which only worked because TPython use was usually single-threaded. This is why it's required: Initialize() runs implicitly on the first TPython call (e.g. CPPInstance_FromVoidPtr), and all API functions acquire/release the GIL via PyGILRAII, which expects the caller to not hold the GIL afterwards. If Initialize didn't release it, any second thread using TPython would deadlock in PyGILState_Ensure.

Note that on the long term (next release) I want to deprecate the implicit initialization and require the user to initialize the Python interpreter themselves. Then we avoid confusion from implicit initialization like this one. That would also deter analyzers from over-using TPython while leaving it accessible to framework experts like in ATLAS/Athena.

vepadulano · 2026-10-06T07:05:12Z


+   // attach to the cppjit module to prevent the creation of a python proxy
+   // for the C++ proxy
+   if (PyObject *thisModule = cppjit::cpyrt::GetThisModule()) {


So, TPython also depends on cppjit's C++ API. Is this wanted also in the future?

We don't know, that's a a separate discussion for another PR and the Tuesday meetings. I wanted to clearly separate the API usage that is just for Python interpreter embedding (independent from the rest of cppjit) from the API functions that actually use the cppjit type system. ALTAS uses the latter inside event loops in Athena, so this performance requirement makes is hard to not have a cppjit C++ API exposed via TPython....

vepadulano · 2026-10-06T07:05:38Z

+   // attach to the cppjit module to prevent the creation of a python proxy
+   // for the C++ proxy


This sentence is not clear to me, can you elaborate?

vepadulano · 2026-10-06T07:06:21Z

-   std::vector<std::string> args(argc);
-   for (int i = 0; i < argc; ++i) {
-      args[i] = argv[i];
+   FILE *fp = fopen(name, "r");


I don't see the corresponding fclose, maybe use an RAII struct to avoid confusion

Right, I now see the comment at line 401, that's just confusing.

guitargeek added 2 commits October 5, 2026 21:26

guitargeek self-assigned this Oct 5, 2026

guitargeek added the in:Python Interface label Oct 5, 2026

guitargeek requested a review from vepadulano as a code owner October 5, 2026 19:27

guitargeek requested a review from bellenot as a code owner October 5, 2026 20:39

guitargeek added the AI-Assisted label Oct 5, 2026

vepadulano requested changes Oct 6, 2026

View reviewed changes

Navigation Menu

Repository navigation

[tpython] Handle Python interpreter embedding in TPython, not cppjit - #23616

[tpython] Handle Python interpreter embedding in TPython, not cppjit#23616
guitargeek wants to merge 3 commits into
root-project:masterfrom
guitargeek:cppjit-tpython-boundary

guitargeek commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026 •

edited

Loading

Uh oh!

vepadulano Oct 6, 2026

Uh oh!

guitargeek Oct 6, 2026

Uh oh!

Uh oh!

Uh oh!

vepadulano Oct 6, 2026

Uh oh!

guitargeek Oct 6, 2026

Uh oh!

vepadulano Oct 6, 2026

Uh oh!

guitargeek Oct 6, 2026

Uh oh!

vepadulano Oct 6, 2026

Uh oh!

vepadulano Oct 6, 2026

Uh oh!

vepadulano Oct 6, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// saved thread state is never restored: ROOT never finalizes the
		// embedded interpreter.

		// attach to the cppjit module to prevent the creation of a python proxy
		// for the C++ proxy

Conversation

guitargeek commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Oct 5, 2026 •

edited

Loading