[Unmanaged Memory] Memory bloating on heavily looped calls due to delayed finaliziation by GC #613

Activity

1. Nucs commented on May 21, 2026

   

   Nucs

   on May 21, 2026

   MemberAuthor

   More actions

   Proposal: atomic reference counting on the unmanaged buffer, with IDisposable on NDArray

   The problem in the issue boils down to a single missing piece: there is no synchronous path from "user knows this array is dead" to "native buffer is freed." Everything goes through ~Disposer() on the single CLR finalizer thread.

   I propose two cooperating layers. Each is small and independently committable.

   L1 — make Disposer reference-counted

   Replace the Disposed boolean in UnmanagedMemoryBlock<T>.Disposer with an atomic refcount. Every logical owner (typically an NDArray) holds one ref; the buffer is freed the instant the last reference is dropped, on the calling thread.

   Encoding:

   • _refCount >= 0 — live, with N owners
   • _refCount == -1 — released; NativeMemory.Free already called

   Three operations:

   public bool TryAddRef()
   {
       if (_type == AllocationType.Wrap) return true;          // non-owning wrap = immortal
       long c;
       do {
           c = Interlocked.Read(ref _refCount);
           if (c < 0) return false;                            // already released
       } while (Interlocked.CompareExchange(ref _refCount, c + 1, c) != c);
       return true;
   }
   
   public void Release()
   {
       if (_type == AllocationType.Wrap) return;
       long n = Interlocked.Decrement(ref _refCount);
       if (n == 0)
       {
           // Claim the free transition 0 → -1 atomically. If a racing AddRef
           // bumps it back to 1 before our CAS, we don't free — the new owner
           // is responsible.
           if (Interlocked.CompareExchange(ref _refCount, -1, 0) == 0)
               ReleaseUnmanagedResources();
       }
       else if (n < 0)
       {
           // Stray Release on already-released block. Each thread compensates
           // its own Decrement so concurrent strays self-balance one-for-one.
           Interlocked.Increment(ref _refCount);
       }
   }

   ReleaseUnmanagedResources is guarded by a single-shot Interlocked.Exchange(ref _freed, 1) so the existing finalizer path (~Disposer) and the new Release path can never double-free under any race. The finalizer stays in place as a safety net for code that doesn't opt in:

   ~Disposer()
   {
       if (Volatile.Read(ref _freed) == 0)
           ReleaseUnmanagedResources();
   }

   L2 — make NDArray implement IDisposable

   NDArray becomes IDisposable with idempotent Dispose() and a finalizer that releases the held reference if the user forgot.

   public partial class NDArray : ..., IDisposable
   {
       // int (not byte) — Interlocked.Exchange's narrowest type.
       private int _disposed;
   
       public bool IsDisposed => Volatile.Read(ref _disposed) != 0;
   
       public void Dispose()
       {
           if (Interlocked.Exchange(ref _disposed, 1) != 0) return;   // idempotent
           Storage?.InternalArray?.Release();
           GC.SuppressFinalize(this);
       }
   
       ~NDArray()
       {
           // Safety net for the "user forgot" case — runs on finalizer thread,
           // same as today, but now it cooperates with the refcount.
           if (Volatile.Read(ref _disposed) == 0)
               Storage?.InternalArray?.Release();
       }
   }

   Every concrete constructor calls InitializeArc() at the very end, which does a single TryAddRef on the underlying IArraySlice so this NDArray's logical ownership is tracked. Views participate by the same mechanism — slicing, transposing, reshaping all flow through an NDArray constructor, so every view ends up holding its own ref.

   IArraySlice and IUnmanagedMemoryBlock each grow three thin members — TryAddRef, Release, IsReleased — that forward to the underlying Disposer.

   What this changes for callers

   // Existing code keeps working exactly as today — finalizer eventually frees.
   var result = (a + b) * c;
   
   // Opt-in deterministic release where it matters:
   using var temp = np.zeros(big_shape);
   
   // Or explicit, when `using` doesn't fit:
   var big = np.empty(huge_shape);
   DoWork(big);
   big.Dispose();   // freed NOW

   Safety invariants this preserves

   • Disposing a parent does not invalidate live views. Each view holds its own ref; the buffer survives until every view is also disposed (or finalized).
   • Dispose() is idempotent — calling it twice, or in parallel from many threads, is safe.
   • Concurrent AddRef vs Release cannot produce a state where TryAddRef returned true but the block has been freed (verified empirically over 10,000 race iterations).
   • Non-owning wraps (AllocationType.Wrap) bypass refcounting entirely so multiple MemoryBlock instances can share Disposer.Null without interfering.
   • GCHandle-backed allocations release only the pin, never the managed array.

   Acceptance criteria mapped to the issue

   Issue requirement	How this proposal satisfies it
   Synchronous release without GC.Collect	Release() → NativeMemory.Free runs on the calling thread when refcount hits 0
   View safety	refcount transitivity — disposing a parent only decrements; buffer stays alive while any view holds a ref
   Finalizer safety net preserved	~Disposer() and ~NDArray() both still exist; only the "user forgot" path uses them
   No existing test regressions	All ~8500 prior tests continue to pass on net8.0 and net10.0
   Measurable improvement on the sweep scenarios	The out_mixed_to_float64 scenario stays at ~0.5 ms (NumPy parity) when out= and source arrays are explicitly disposed between iterations

   Behavior shown empirically

   The same 800 MiB / 100 NDArray loop from the issue, with the proposed change and an explicit arr.Dispose() call per element:

   after alloc:                  765 MiB
   after Dispose-each, no GC:      0 MiB     ← every buffer freed atomically
   

   10,000 alloc + Dispose cycles in a tight loop measure ~1 µs per cycle on .NET 10 / Windows 11 and leave the process working set unchanged.

   Scope / non-goals

   • This proposal does not add an "auto-dispose scope" / arena mechanism (np.scope() or similar). NumPy has no equivalent, and the using keyword fills the same role for callers who want determinism.
   • This proposal does not change Disposer's visibility, the MemoryBlock struct's public API, or any storage-level allocation behavior.
   • The existing MemoryBlock.Free() / ArraySlice.DangerousFree() keep their original "force release everything" semantics for backwards compatibility.

   Migration cost

   Component	LOC	Risk
   Refcount in Disposer	~50	low — internal class
   ARC forwarding in UnmanagedMemoryBlock<T>, ArraySlice<T>, and the two interfaces	~30	low — additive
   NDArray.Dispose + finalizer + InitializeArc calls from each public ctor	~40	low — additive, no behavioral change for callers who don't call Dispose
   Test coverage (ArcLifecycleTests)	~600	—

   Total: a single PR, ~700 lines net additive, no public breaking changes.
2. Nucs

   changed the title [-][Unmanaged Memory] GC bloating on heavily looped calls due to delayed finaliziation by GC[/-] [+][Unmanaged Memory] Memory bloating on heavily looped calls due to delayed finaliziation by GC[/+] on May 21, 2026
3. Nucs

   self-assigned this

   on May 21, 2026
4. Nucs

   added

   architectureCross-cutting structural changes affecting multiple componentsCross-cutting structural changes affecting multiple components

   performancePerformance improvements or optimizationsPerformance improvements or optimizations

   coreInternal engine: Shape, Storage, TensorEngine, iteratorsInternal engine: Shape, Storage, TensorEngine, iterators

   on May 21, 2026
5. Nucs

   added the

   Bug

   issue type on May 21, 2026
6. Nucs commented on Sep 7, 2026

   

   Nucs

   on Sep 7, 2026

   MemberAuthor

   More actions

   Fixed in 0.70.0 (#628). NDArray is now IDisposable, and NDScope / [NDScoped] return transients to the buffer pool at scope exit instead of waiting on the finalizer:

   NDArray Compute()
   {
       using var s = NDScope.Open();
       var a = np.arange(5);
       var doubled = a * 2;          // transient - returned to the pool at scope exit
       return s.Returns(doubled);    // this one survives
   }
   
   Console.WriteLine(Compute());   // [0 2 4 6 8]

   Core weaves ~265 np.* methods with [NDScoped]; to weave your own, add the dev-only NumSharp.Build (dotnet add package NumSharp.Build). NDScope/IDisposable are in Core. Closing.
7. Nucs

   closed this as completedon Sep 7, 2026