Stream missing annexed content with remfile, cached by annex key

[CodyCBakerPhD]

This PR and the one stacked on it (#1956) are an alternative to #1941 and #1933. The goal is the same: dandi validate on a DataLad clone of a Dandiset, streaming file content without downloading it. Here, dandi leaves the git-annex and HTTP work to datalad-fuse's adapter, and datalad-fuse reads NWB/HDF5 content with remfile.

Changes

• dandi.support.annex_stream.AnnexedReadableFile is a Readable for a locked annexed file whose content isn't present locally.
  • It is backed by datalad-fuse's RemoteFilesystemAdapter, used directly with no FUSE mount. One adapter is shared and closed at exit.
    • caching=False: nothing is written into the clone.
    • backends="remfile,fsspec": NWB/HDF5 files are read with remfile even if a dataset configures datalad.fusefs.backends otherwise.
  • What the adapter does:
    • reads the key from the target of the broken symlink;
    • gets the URLs from git annex whereis, in the order git-annex lists them. For a DataLad Dandiset that is the DANDI Archive's API download URL, which redirects to S3, then the direct S3 URL;
    • opens the first URL that can be read. .nwb/.h5 files go through datalad-fuse's remfile backend.
  • What dandi adds: only a thin io.RawIOBase wrapper around the file object the adapter returns, so it works with h5py (which reads with readinto()) and pynwb.
  • Why remfile: it reads in the calling thread, in requests sized for HDF5's access pattern. To read the structure of a 16 MB NWB file in the tests it downloads under a tenth of the file. fsspec with its default cache downloaded 64 MB, four times the file.
• get_annexed_readable(path) returns None when:
  • datalad-fuse (with its adapter) or remfile isn't installed;
  • the path isn't a broken symlink to annexed content;
  • git-annex isn't initialized in the repository;
  • datalad-fuse can't handle the repository, e.g. one without commits;
  • the key records no size;
  • git-annex knows no URL for the content.
• No implicit git annex init. Before handing a file to datalad-fuse, dandi checks git config annex.uuid. So the adapter, which would otherwise run git annex commands that initialize it, is never used in such a clone. This also avoids loading a DataLad dataset for every broken symlink of a fresh git clone. The tests check that a fresh clone keeps no annex UUID.
• Caching by annex key: get_metadata, get_neurodata_types and nwb_has_external_links use annex_fingerprint as their custom_fingerprint.
  • It wraps fscacher 0.5.0's annex_key_fingerprint, which keys on the path paired with the annex key.
  • Locked annexed files are cached by key, whether they are streamed or present locally. Only content-hash backends count, so WORM and URL keys fall back to stat().
  • Unlike Memoize readable fingerprints #1941, this needs no fingerprint hook on Readable.
• Dependencies:
  • datalad extra: datalad-fuse with remfile, for now pinned to a commit (see below). git-annex is also required.
  • fscacher >= 0.5.0.
• Tests:
  • make_git_annex_dandiset() builds a real git-annex repository: content added, one or more URLs registered, then dropped.
  • A range_http_server fixture serves files with range requests from a separate process (dandi/tests/range_http_server.py). No thread of the test process serves requests, so a garbage collection there can't run h5py finalizers while h5py's lock is held waiting for a response. The server exits normally when stopped, so coverage records its process.
  • New tests cover:
    • the file object's seek/read/end-of-file behavior;
    • reading through whichever URL works;
    • the error when no URL can be read;
    • small reads of a large NWB file;
    • an uninitialized clone left untouched;
    • every case in which the content isn't streamed;
    • caching by key.
  • New skipif.no_git_annex condition.
  • A new CI job (mode: datalad-fuse, ubuntu, Python 3.12) installs git-annex and .[datalad]. The other jobs skip these tests.

Depends on unreleased datalad-fuse

This uses the remfile backend and pluggable backends of datalad/datalad-fuse#131, which datalad-fuse hasn't released yet. It works with that PR as it is. Until it is released, the datalad extra pins the PR's head commit in datalad/datalad-fuse:

"datalad-fuse[remfile] @ git+https://github.com/datalad/datalad-fuse@b4dbee656eeb8ed54b1396c8d7e5c8b369b64a4a"

With the pin, pip install "dandi[datalad]" gives working streaming, and CI just installs .[datalad].

Before merging, the pin must be replaced by a requirement on the first datalad-fuse release with #131. PyPI rejects distributions with direct URL dependencies, so a release of dandi with the pin would fail to upload.

With a released datalad-fuse that lacks #131, nothing breaks: get_annexed_readable() returns None and the content is reported as missing.

An init_annex option of the adapter (datalad/datalad-fuse#141), to stop it from initializing git-annex in an uninitialized clone, would be useful to other users of the adapter, but dandi doesn't need it: it checks annex.uuid itself.

Why datalad-fuse's remfile backend

An earlier version of this PR streamed through datalad-fuse's fsspec adapter. That needed three workarounds in dandi:

• a custom fsspec cache, so HDF5's scattered reads didn't download most of the file;
• pausing garbage collection while streamed files were open;
• dandi's own git-annex initialization guard.

The garbage collection pause was needed because fsspec fetches data in a separate I/O thread. If a collection runs in that thread while h5py holds its global lock waiting for the data, any finalizer that calls h5py deadlocks. A patched h5py wasn't enough: hdmf 6.2.0's HDMFIO.__del__ calls is_open(), which waits for the same lock.

remfile fetches data in the reading thread, so with datalad-fuse's remfile backend none of these workarounds is needed. dandi keeps no streaming code of its own beyond a file-object wrapper.

Compared to #1941

746 added lines, 257 excluding tests and the test server, against #1941's 990. It drops AnnexKey, AnnexRepo, the URL-log parsing, the fake-annex fixtures and the Readable.get_fingerprint hook. In exchange, streaming requires git-annex and datalad-fuse with remfile.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

---

Generated by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 99.56710% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.16%. Comparing base (44fd1a6) to head (b75c824).

Files with missing lines	Patch %	Lines
dandi/tests/range_http_server.py	97.46%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1955      +/-   ##
==========================================
+ Coverage   90.83%   91.16%   +0.33%     
==========================================
  Files          92       96       +4     
  Lines       14190    14648     +458     
==========================================
+ Hits        12889    13354     +465     
+ Misses       1301     1294       -7     

Flag	Coverage Δ
unittests	91.16% <99.56%> (+0.33%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.