Repository navigation
RDataFrame in MultiThread with RNTuples #23136
Description
Check duplicate issues.
- Checked for duplicates
Description
Dear ROOT developers,
We have an analysis framework for CMS data that uses ROOT RDataFrame in MultiThread. In view of the expected adoption of RNTuples, we tried to use an RDataFrame script to convert CMS data stored in ROOT files as flat ntuples (NanoAOD, TTree structure) to RNTuples, which we can then read with our usual framework.
However, when running on the RNTuples the program opens a number of files equal to the number of available threads, waits for all of these files to be processed, and only then opens the next batch. As a result, the execution is limited by the slowest file-thread combination of the batch.
We have prepared a reproducer [linked below with instructions and comments] that generates a set of files as RNTuples, processes the events, and reproduces the issue. We have observed the same behavior on lxplus, on our institute's cluster and our laptop.
On lxplus and our institute cluster we also get the warning
Warning in <RIoUring>: io_uring is unexpectedly not available because:
Error initializing io_uring: Operation not permitted
Warning in <RRawFileUnix>: io_uring setup failed, falling back to blocking I/O in ReadV
Thank you in advance for your help!
Francesco Orlandi and Federico Ferri (@ferriff)
Reproducer
On lxplus:
- copy
reproduce_problem.ccandj.jsonin a folder - compile:
clang++ reproduce_problem.cc $(root-config --cflags --libs) -o reproduce_problem - execute:
./reproduce_problem -g -n 3 -j j.json
Options:
g(activate generation of files, necessary only the first time)n(number of threads for MT execution)j(json file name containing the input files name)
The program generates 100 files: the odd-numbered files contain 9 entries, while the even-numbered files contain 10000 entries. During the execution, we use CPU time by performing some mathematical computations for each entry. For each “batch”, we print the file names and the number of entries contained in each file. Together with the progress bar from RDataFrame, this shows the problem.
ROOT version
6.40.02, built on Jun 20 2026
Installation method
lxplus
Operating system
Linux
Additional context
No response
Activity
vepadulano commentedon Sep 2, 2026
Dear @fraorlnd ,
Thank you for the report of the issue and the reproducer! I have started working on it and I believe the root cause is the same one that I was trying to address recently with a PR I opened as a draft #22631 .
I wonder, would it be possible for you to build a custom version of ROOT from the branch of my PR? If so, I would appreciate if you could run your tests again and confirm that the performance improves.
Otherwise, I will continue the work on my side with your reproducer and let you know of any updates.
ferriff commentedon Sep 3, 2026
Dear @vepadulano ,
Thanks, I tested the PR cherry-picking your commit on top of 6-40-04 and it does fix the problem of idle threads waiting for the slowest busy one before reading another bunch of files. Tested on the reproducer as well as on real-life RNTuples.
However, interestingly, even if the threads are all busy, the total processing time in the new fixed version is comparable with the old version (actually a little slower). The displayed event rate is consistent with this: for the old version, it gets lower when the threads are waiting but the peaks are significantly higher than in the new version, which displays a constant rate as expected. I varied the number of processing threads in the new version, trying to find a sweet spot that would reach the peak rates of the old version, but with no luck.
Hope this helps. Feel free to contact us directly on mattermost if you wish more feedback.