Repository navigation
[Python] Remove ROOT Pythonization CPython extension - #23660
Draft
guitargeek wants to merge 12 commits into
Draft
guitargeek wants to merge 12 commits into
guitargeek wants to merge 12 commits into
Conversation
guitargeek
changed the title
Test Results 22 files 22 suites 3d 21h 9m 48s ⏱️ For more details on these failures, see this check. Results for commit c3be4a1. ♻️ This comment has been updated with latest results. |
guitargeek
force-pushed
the
pyz_gone
branch
3 times, most recently
from
October 9, 2026 21:55
adfdf65 to
8f25bc8
Compare
TCling::ToString() forwarded to cling::Interpreter::toString(), which builds and compiles a statement calling cling::printValue() on every call. Printing many objects was therefore slow, and every call left compiled code behind (see root-project#12817, which PyROOT worked around with its own cache of jitted printers). Compile a printer function on the first call for a type and reuse it. Unloading code clears the cache, because a type that is reloaded with changes needs a new printer. 🤖 Done with the help of AI
The __str__ that prints objects with cling::printValue() was implemented in C++ in GenericPyz.cxx, with its own cache of jitted printer functions to avoid the cost of TInterpreter::ToString(). Now that ToString() caches a printer per type itself, the pythonization is a call to it. Null objects now fall back to cppyy's generic __repr__ rather than the class's own one. The std::vector pythonization defines __repr__ in terms of str(), so str() of a null std::vector proxy recursed until a RecursionError. 🤖 Done with the help of AI
…s [upstream] The executor for a function pointer return type wraps the pointer in a std::function, compiled from the return type and signature it takes from the type name. CreateExecutor(TCppType_t) took them from the type name with all spaces removed for the executor lookups, so a signature with a multi-word type did not compile: std::function<void(int,bool,constchar*,constchar*)> ... TypeError: conversion to std::function failed which broke e.g. SetErrorHandler() in ROOT, which returns the previous handler. Take them from the type as spelled instead. The new test is separate from test26_function_pointers, which does not run on macOS. 🤖 Done with the help of AI
IOHandler.cxx captured the stdout and stderr of C and C++ code in notebooks by redirecting the file descriptors into pipes, and ran notebook cells through TInterpreter, reporting C++ exceptions. Both are now Python: the capture with os.dup2() and pipes, the cell execution through gInterpreter, with cppyy translating C++ exceptions. The capture used to read the pipes only after flushing C stdio, so a cell that wrote more than the pipe holds (64 kB on macOS) hung: the writer blocked on the full pipe holding the stream lock that the flush waited for. Now a thread per stream drains its pipe as long as the capture runs. A process started during the capture can keep a pipe open, so the end of a capture waits at most a second for the rest of the output. Other differences: - each IOHandler has its own capture, instead of all sharing one, which the first one to be deleted destroyed; - deleting an IOHandler ends its capture; - output that is not valid UTF-8 is decoded with replacement characters instead of raising; - ProcessLine() and Declare() release the GIL (set in the facade), so the ROOT kernel streams output while a cell runs. Tested by running the JupyROOT notebooks of roottest with Jupyter installed, as they are disabled in builds without it. 🤖 Done with the help of AI
…pstream] C++ code that reads memory owned by a Python object, like a data source over NumPy arrays, needs to keep the object alive for as long as it may read, which can be past the lifetime of any Python reference to it. ROOT had its own CPython extension function for this, wrapping a reference in a std::any. ll.as_std_any(obj) returns such a std::any. It and every copy C++ makes of it hold a reference to obj. As C++ may copy or destroy them on any thread, the reference count is changed with the GIL taken; after Python is finalized, the reference is left be. 🤖 Done with the help of AI
The NumPy arrays of an RDataFrame made from them, and the TTree of an RDataFrame made from a TTree, were kept alive with a reference in a std::any from the PyObjRefCounterAsStdAny() function of the PyROOT extension module. cppyy.ll.as_std_any() does the same, and also takes the GIL to release the reference, which the destructor of the RDataFrame may do on a thread that does not hold it. 🤖 Done with the help of AI
TBuffer::Buffer() returns a char*, which the Python bindings see as a string, with the address of the buffer gone. Pickling ROOT objects from Python needs that address to read the streamed bytes. 🤖 Done with the help of AI
Pickling of C++ objects, by streaming them with ROOT I/O, was implemented in CPPInstancePyz.cxx of the PyROOT extension module, which installed a C function as the __reduce__ method of all cppjit instances. It is now Python: _reduce() streams the object with TBufferFile, reading the streamed bytes with ROOT::Internal::GetBufferAddress(), and _expand() reads it back. For that, the __reduce__ hook of cppjit holds a Python callable instead of a C function, set with libcppjit._set_reduce_method(). Pickles of the previous implementation can't be read back: they name the expand function of the removed extension module. 🤖 Done with the help of AI
PyROOT's PyOS_InputHook processes GUI events while the Python prompt waits for input, updating a web canvas first so that its pending drawing is part of them. The hook lives in the Python bindings' CPython extension module, which is going away: the ROOT part of it is now a compiled helper in Core, which needs no Python, so the Python bindings can install it from Python code. 🤖 Done with the help of AI
…[upstream] C++ libraries commonly report warnings to a callback. ROOT's Python bindings turn them into Python warnings, which only a compiled handler can do: it has to take the GIL, decline the warning while a Python exception is pending (such as a previous warning that the filters turned into an error during the same call into C++) and leave the exception a raised warning sets pending until C++ returns to Python. A ctypes callback prints and clears that exception instead, and a cppjit callback trampoline makes cling parse Python.h, which is too slow for every import. warning_handler() hands out such a handler as a ctypes function pointer, to be passed to the library's registration function. It takes an optional category, RuntimeWarning by default, and the module name the warnings filters match against, so that warnings from one library can be filtered as a group. 🤖 Done with the help of AI
…its [upstream] A C++ GUI toolkit bound with cppjit has to process its events while the interactive Python prompt waits for input, which is what PyOS_InputHook is for. The hook has to be compiled: it must take the GIL, so that event processing never runs alongside other Python threads, and it must run no Python code, since a ctypes callback swallows the KeyboardInterrupt of a Ctrl-C arriving while it runs and the Ctrl-C is lost. set_input_hook() installs a hook that calls the given compiled C++ function with the GIL and then the hook that was installed before; None restores the previous hook. 🤖 Done with the help of AI
Its last two functions installed the error handler that turns ROOT warnings into Python warnings and the PyOS_InputHook that processes GUI events while the Python prompt waits for input. Both have to be compiled, and cppjit now provides them: _backend.warning_handler() hands out the handler for ROOT::Internal::SetWarningHandler(), and _backend.set_input_hook() installs the hook that calls the new ROOT::Internal::ProcessGUIEvents(). ROOT's Python package now consists of Python code only, on top of the cppjit extension, and no ROOT library outside cppjit includes Python.h. 🤖 Done with the help of AI
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The goal is to have no C Python API code in ROOT at some point, so ROOT becomes Python version independent. All the C Python extension code should be confined to cppjit, which will be externalized from ROOT one day. Like this, the ROOT PyPI wheels won't have to be built against different Python versions as well, among other niceties.
This is a draft PR for now to check how realistic this goal is. The individual commits will be submitted as separate PRs after this big PR is green.