Return large run results through shared memory instead of the pipe

[kavyawadhwa134]

Why

Profiling (session wonderful-dirac-dd7816, tools/profile_breakdown.py) showed that pickling the awkward output of Application.run through multiprocessing.Pipe took 40–81% of the wall time on macOS (~60–80 MB/s, 190–310 MB per 8-thread run). This had to be fixed before a GPU offload could be evaluated fairly.

What

• New geant4_python_application/_transfer.py:
  • Child (pack): if a result holds ≥ 1 MB of ak.Array data, it calls ak.to_packed + ak.to_buffers (the same steps awkward's own pickling uses) and copies every buffer (64-byte aligned) into one multiprocessing.shared_memory block. Only the forms, lengths, offsets, behavior and the block name/size go through the pipe.
  • Parent (unpack): attaches, unlinks right away, copies the block out, closes it, and rebuilds the arrays with ak.from_buffers.
• application.py: the child calls pack before pipe.send (and discards the block if the send fails). _send_and_recv calls unpack.
• Fallbacks to the old pickle path: results under 1 MB, results without awkward arrays, and any failure to allocate or write the block. On Linux, posix_fallocate reserves the pages up front, so a too-small /dev/shm (64 MB by default in Docker) makes it fall back instead of crashing the child with SIGBUS.
• Crash cleanup: under spawn, fork and forkserver, the parent and child share multiprocessing's resource tracker (checked). A block orphaned by a killed child is unlinked once the processes exit.
• Public API and returned data are unchanged. Comparing a real 4-thread em_calo output field by field (types, values, behavior), the result is identical to a pickle round-trip.
• Developer guide §60 updated. New tests/test_transfer.py: round-trip of strings, records, nested lists, option types and with_name; unlink after unpack; discard; allocation-failure fallback; and an end-to-end run large enough to use shared memory, at 0 and 2 threads.

Benchmark (em_calo, Apple M2 8 GB, Geant4 11.4.3 MT, medians of 5)

	1 thr before → after	4 thr before → after	8 thr before → after
wall Application.run	0.76 → 0.45 s	3.10 → 0.84 s	4.20 → 1.84 s
pickle + pipe (transfer) share	38% → 2%	75% → 8%	75% → 16%
events/s	13 → 22	13 → 48	19 → 43

G4 transport is now 50–91% of public-API wall time, compared with 18–58% before. The 8-thread samples are noisy (1.56–5.21 s), which fits memory pressure on an 8 GB machine.

Testing

• Full suite on macOS against the 11.4.3 conda-forge build: 55 passed, 2 skipped (both skips existed before; test_transfer.py is the last file, rerun separately after a test-size fix).
• Linux is not run locally; CI covers it.

Follow-up commits

• Worker death mid-transfer: the parent names each block (g4<random>_<counter>) and unlinks it after every call. A block written by a worker that is SIGKILLed before sending is removed right away instead of at interpreter exit.
• Test hooks: GEANT4_PYTHON_APPLICATION_SHM_MIN_BYTES forces either path (0 / huge). _transfer.stats counts results by path. A test-only pause runs after the block is written.
• New tests:
  • shm vs pipe on the same seeded run give identical arrays (fields, nesting, dtypes, parameters);
  • results over 1 MB use shm;
  • no blocks are left in /dev/shm after a normal run, a run that raises, a failed unpack, or a mid-transfer SIGKILL;
  • an 8-thread PbWO4 result much larger than a 64 MB /dev/shm falls back to the pipe without SIGBUS.
• CI: Ubuntu / Python 3.10 remounts /dev/shm at 64 MB, reruns tests/test_transfer.py, and fails if the small-shm test is skipped.
• tools/profile_breakdown.py committed, updated for the new transfer.
• macOS: 59 passed, 3 skipped (the new small-shm test skips on macOS).

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 28fb7269-4b84-4fcc-b071-903c31ae579f

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}