Skip to content

NeXus output on native HDF5, replacing the frozen napi (libNeXus) - #2895

Open
willend wants to merge 8 commits into
mccode-dev:mainfrom
willend:nexus-native-hdf5
Open

willend wants to merge 8 commits into
mccode-dev:mainfrom
willend:nexus-native-hdf5

Conversation

@willend

@willend willend commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

McCode's NeXus output stops using libNeXus. The NeXus C API (napi.h) has been frozen and deprecated upstream. It is replaced by about 300 lines of native HDF5 code in the runtime, keeping the same NX* function names, so --format=NeXus needs only HDF5.

This is the full replacement that #2889 was the pre-study for. #2889 fixed the napi-based writer and added Test_NeXus_output. This PR keeps those fixes and passes the same test.

The libNeXus dependency is removed from packaging, devel tools, CI, toolchains and docs (commit 1318d48ff); the conda-forge recipes are the remaining follow-up.

What changes

  • mccode-r.h.in no longer includes napi.h. It declares the napi subset McCode and the Monitor_nD libraries use: NXhandle, NXstatus, NXlink, NX_OK/NX_ERROR, NX_CHAR/NX_FLOAT64/NX_INT32, NX_UNLIMITED, NXACC_RDWR/NXACC_CREATE5, and NX_COMP_NONE/NX_COMP_LZW/NX_COMP_LZW_LVL0..9. Existing -DNX_COMPRESSION=NX_COMP_LZW_LVL1 flags keep working.

  • mccode-r.c implements the 19 functions used, on HDF5:

    • files: NXopen, NXclose;
    • groups: NXmakegroup, NXopengroup, NXclosegroup;
    • data sets: NXmakedata, NXcompmakedata, NXcompmakedata64, NXopendata, NXclosedata, NXputdata, NXputslab64;
    • attributes and queries: NXputattr, NXgetinfo64, NXgetgroupinfo;
    • links: NXgetgroupID, NXmakelink;
    • error reporting: NXMDisableErrorReporting, NXMEnableErrorReporting.

    Components that use nxhandle/NX* directly keep compiling.

  • Build: NEXUSFLAGS (in PlatformDefaults.cmake, the Debian toolchains, MCUtil.cmake and InstallMCCODE.cmake) link only HDF5. Debian packages depend on libhdf5-dev instead of libnexus-dev. An existing mccode_config.json that still passes -lNeXus keeps working: the runtime's own NX* take precedence.

Design

  • Open groups are a stack of HDF5 handles, at most 32 deep. HDF5 resolves names relative to the current group, so no path buffers are needed: long component or monitor names are written, not skipped or overflowed. A failed open leaves the stack unchanged, and closing below the root is reported.
  • The file layout follows libNeXus 4.4.3:
    • groups carry an NX_class attribute;
    • strings are fixed-length, scalar attributes replace existing ones, and one-element string data sets are kept;
    • compression uses chunked storage with shuffle and deflate, as NX_COMPRESSION sets it;
    • links are hard links with a target attribute;
    • the root attributes are unchanged, including NeXus_version = 4.4.2.
  • Error suppression nests. In napi, one nested disable/enable pair silenced error reporting for the rest of the run.
  • Chunks are kept below HDF5's 4 GiB limit inside NXcompmakedata64.
  • HDF5's automatic error printing is turned off at file open, as napi did. The MPI per-node file/VDS code relied on that side effect.

Output differences against the libNeXus build

The new and old builds were compared with h5dump -p and h5diff, with timestamps masked. Covered: serial runs, list-buffer flushes (2.5M events), --IDF, MPI with 2 ranks (node files and the VDS), Test_NeXus_comp_par_runtime, and a McXtrace instrument. Group and data-set layout, chunking, filters and data were identical, down to byte-identical file sizes. The remaining differences:

  • Long monitor paths (over about 1000 characters) are now written. Their group can't be linked from /entry1/data because NXlink keeps napi's 1024-character targetPath; this is reported.
  • No root NX_class is written; the libNeXus 4.4.3 builds in use never wrote one.
  • The file_time offset is now correct during daylight saving time; napi was one hour off.
  • Errors napi hid are now printed as NeXus error: ….

Limitations for component authors

  • Only the 19 functions listed above exist. Code calling another napi function, or including napi.h itself, will not build.
  • NXopengroup doesn't check NX_class.
  • NXgetgroupinfo fills name and class only at the root.
  • NXputdata writes the whole data set; use NXputslab64 to append.

CI (willend/McCode, workflow_dispatch, merge commit 1e3c36752)

mcstas-basictest and mcxtrace-basictest are both green. The NeXus output regression test passed in all 12 classic jobs: Linux gcc-13, clang, Debian and conda; macOS clang; Windows MSVC. The antlr jobs skip it by design.

Follow-ups (not in this PR)

  • Drop nexus/libnexus from environment.yml, meta-pkgs/windows/*.yml and mccode-create-conda-yml (now hdf5), remove 3rdparty/nexus-package
  • conda-forge feedstock recipes (outside this repo): replace libnexus by hdf5
  • Source-build CI jobs: install libhdf5-dev / Homebrew hdf5 instead of libnexus-dev / nexusformat, no nexus CMake dirs
  • Code generator hints say -DUSE_NEXUS -lhdf5; manuals and examples updated
  • Toolchains: unused nexus dirs removed (Debian), NEXUSINCLUDE points at the HDF5 headers (macOS, MinGW); mccode-build-conda no longer passes nexus dirs
  • Not tested: ASan (it hangs at startup on macOS; Linux needed), GPU builds, macOS x86_64

🤖 Generated with Claude Code

willend and others added 5 commits October 22, 2026 17:33
Implement the subset of the NeXus API used by McCode and the Monitor_nD
libraries (NXopen/close, make/open/closegroup, (comp)makedata(64),
open/close/putdata, putslab64, putattr, getinfo64, getgroupinfo,
getgroupID, makelink, error reporting) directly on HDF5 in mccode-r.c,
with the same names so components using nxhandle keep compiling.
The written layout (groups, NX_class, string types, chunking, shuffle +
deflate level, links and 'target' attributes) matches libNeXus 4.4.3,
verified with h5dump -p/h5diff on serial, list-flush, --IDF and MPI (VDS)
runs for McStas and McXtrace.

The open groups are a stack of HDF5 ids: no fixed path buffers, a failed
open does not change the state, over-closing is reported, nested error
suppression nests, and chunks are kept below the 4 GiB HDF5 limit.

Fixes in the NeXus writers:
- nxstr used the value as printf format: string parameters, titles or
  instrument text with '%' were garbled (e.g. "a%sb" -> "aa%sbb") or UB.
  Values are now passed as "%s", without the 1024-char buffer.
- Monitor_nD BINS output wrote uninitialised MCDETECTOR fields (Date,
  xylimits, ... seen as stack garbage in MPI runs); zero it.
- uninitialised NXlink used for /entry/data links when a group failed to
  open; unbalanced closegroup in mcdatainfo_out_nexus and the IDF group.
- existing files were opened with mode NXACC_CREATE5|NXACC_RDWR (7).
- nxhandle was defined in the header (duplicate symbol with -fno-common
  when the runtime is not embedded); siminfo_out() wrote to the NeXus
  handle as a FILE*.
- drop a debug printf in mcdetector_out_3D_nexus and the duplicated
  zmin/zmax geometry attributes in Monitor_nD.

Output changes: no root NX_class is written (as libNeXus 4.4.3), the
file_time offset is now correct during DST, NeXus_version stays "4.4.2".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
They belong to the UB_matrix work on ub-matrix-injector and were picked up
here by accident from a shared working tree. Nothing on this branch uses them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Test_NeXus_output.instr with test_nexus_output.py, run by the basictest
workflows after the MPI list test. It runs the instrument with
--format=NeXus, serial and with MPI, and checks mccode.h5 with h5py:

- string values containing '%' are stored as given (instrument
  parameter and component parameter literal)
- all string attributes and data sets are printable (no uninitialised
  memory, as in the Monitor_nD BINS output)
- no debug output on stdout
- a monitor file name of 1000 characters does not crash the run
- --append adds entry2 to mccode.h5

With the current napi-based writer the first four fail.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Conflicts with the napi fixes of mccode-dev#2889 in mccode-r.c/mccode-r.h.in are
resolved in favour of the native HDF5 layer, which already contains them.
The napi-only workarounds merged from mccode-dev#2889 are dropped here: the native
layer has no 1024-character path buffers (long monitor names are written,
not skipped) and NXcompmakedata64 already keeps chunks below 4 GiB.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@willend
willend marked this pull request as ready for review October 26, 2026 18:14
willend and others added 3 commits October 26, 2026 19:14
NeXus output needs HDF5 only since the native HDF5 writer:

- conda: hdf5 instead of libnexus in mccode-create-conda-yml (used by
  the conda CI jobs, so these now build without libNeXus present); drop
  'nexus' from environment.yml and the Windows meta-pkg environments
  (they already list hdf5)
- remove 3rdparty/nexus-package (it packaged libNeXus) with its sync.sh
  clone and the GhostBSD mkdist line
- mccode-build-conda: no -DNEXUSLIB/-DNEXUSINCLUDE, the conda NEXUSFLAGS
  already use $CONDA_PREFIX/include and lib
- basictest workflows: install libhdf5-dev (Linux) / hdf5 (Homebrew)
  instead of libnexus-dev / nexusformat, no -DNEXUSLIB/-DNEXUSINCLUDE
- toolchains: Debian ones set NEXUSFLAGS fully, drop the unused
  NEXUSLIB/NEXUSINCLUDE; macOS/MinGW NEXUSINCLUDE points at the conda
  include dir holding hdf5.h instead of include/nexus
- code generator: hint '-DUSE_NEXUS -lhdf5' instead of '-lNeXus'
- manuals (McStas, McXtrace), ILL_H16_IN5_Mantid and the macOS
  dependency README: HDF5 instead of libNeXus, -lhdf5 instead of -lNeXus

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant