Disable automatic GC while a remote asset is open, to avoid a deadlock

[CodyCBakerPhD]

Reading a remote NWB asset through RemoteReadableAsset can hang forever: a deadlock between h5py's global lock and garbage collection in fsspec's IO thread. This is the cause of the timeouts in #1450, and it made test_reextract_metadata time out in the lowest-deps job of #1955.

The deadlock

• h5py holds its global lock while it reads from a Python file object. A read of a RemoteReadableAsset waits for fsspec's IO thread to fetch the data.
• If a garbage collection runs in that IO thread meanwhile, it can finalize an unreferenced object that closes an HDF5 file. For example, hdmf's HDMFIO.__del__ calls HDF5IO.close(), which calls h5py's File.close(), which waits for h5py's lock.
• Each thread then waits for the other. The aiohttp timeouts added in a6109be cannot end this, because the event loop thread itself is blocked.

The main thread's stack is the one in #1450 and in the CI failure of #1955: h5py.File() → fsspec's readinto() → sync() → Event.wait().

Whether it happens depends on when a collection happens to run, so it comes and goes with unrelated changes. On #1955 it happened on every run of the lowest-deps job, and never on master.

Changes

• RemoteReadableAsset.open() wraps fsspec's file object in a new _GCPausedFile.
  • Automatic garbage collection is disabled while any such file is open.
  • The previous state is restored once all of them are closed (or garbage-collected).
• No new dependencies.

Tests

• test_gc_paused_file and test_gc_paused_file_gc_disabled (new dandi/tests/test_misctypes.py) cover nesting, closing, dropping an unclosed file, and GC being disabled beforehand.
• test_asset_as_readable_open_pauses_gc checks that a remote asset opened from the test archive pauses GC.
• Locally, with the lowest-deps versions from CI (hdmf 4.1.0, h5py 3.16.0, fsspec 2022.11.0):
  • A script that runs a collection in fsspec's IO thread, only if automatic GC is enabled (as automatic collection would), deadlocks without this change and completes with it.
  • test_reextract_metadata passes, also with the latest versions.
• test_nwb2asset_remote_asset now gets past the read. It still fails, on its expected metadata (strain), so its xfail is left as is.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

---

Generated by Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.88%. Comparing base (0d5f96e) to head (f137d02).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1967      +/-   ##
==========================================
+ Coverage   90.76%   90.88%   +0.12%     
==========================================
  Files          92       93       +1     
  Lines       14159    14238      +79     
==========================================
+ Hits        12851    12940      +89     
+ Misses       1308     1298      -10     

Flag	Coverage Δ
unittests	90.88% <100.00%> (+0.12%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.