Repository navigation
Conversation
…ill holding a connection, so a read after it gets the bytes queued before it, not freed memory (#605) In the default shared receive mode what a connection queues, and what a reader takes from that queue, points into the reactor's one receive slab, and Teardown freed it unconditionally. A connection its handler still held kept its queue across the stop: one still in the table, and one its client had closed, which had left the table. A read after the stop served that queue, and a reader still held what it took before the stop, so both handed out slices of freed memory, and reading them faulted the process. The slab now has holders: the teardown and every handler that has not let go. Each accept adds its handler's hold, the handler's DecRef gives it up once per life, and the teardown gives up its own where it freed the slab. The last to let go frees it, on its own thread; the kernel stopped using it when the teardown unregistered the ring. A stop whose handlers all let go during it, as one parked in a read does, frees it where it did. Incremental mode counts its handlers too and has no shared slab. Tests in StopParkedTests (E2E): a read after the stop of bytes queued before it, for a connection open at the stop and one its client closed before it, and what a pipe reader took before the stop, each get intact bytes from a slab still allocated; the slab is freed once the handlers the stop woke let go, and outlives the stop until the last of two held connections' handlers lets go. An incremental-mode control reads intact bytes on main too.
This was referenced Nov 14, 2026
Owner
Author
|
Closed in favour of #634: a stop that drains, then frees everything in one place, instead of a fix per after-stop path. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #605.
Touches core:
src/ioxide/Reactor/Reactor.cs,src/ioxide/Reactor/Reactor.Runner.cs,src/ioxide/Reactor/Loop/Reactor.Loop.SharedRing.cs,src/ioxide/Reactor/Transport/Tcp/Reactor.Tcp.csandsrc/ioxide/Connection/Tcp/TcpConnection.cs, because the shared receive slab, the teardown that frees it, the accept and the handler's release are all the reactor's own, and no satellite can tell when the last reader of that slab is done.What was wrong. In the default shared receive mode every item a connection queues, and every slice a reader takes from that queue, points into the reactor's one receive slab, and
Teardownfreed that slab unconditionally. The stop empties the queue only of a connection whose handler has already let go (ClearinReturnReleasedTcpConnections). A connection its handler still held kept what was queued. That is one still in the table at the stop, and also one its client had closed before the stop, whichCloseFromRecvtakes out of the table, so the stop never sees it. A read after the stop serves that queue before it reports the close, soReadAsyncwithTryGetItem,TcpConnectionPipeReader(and kTLS RX on top of it) andTcpConnectionStreamhanded out slices of freed memory. So did every slice a reader took before the stop and still held.Reproduced at main with a scratch program (a 1 MiB slab, which glibc maps on its own and unmaps when freed): reading 1000 bytes queued before the stop faulted the process with
AccessViolationExceptionfor a connection open at the stop, for one its client closed before the stop, and for aTcpConnectionPipeReadersequence held across the stop, 3 of 3 runs each. Without reading the bytes, each read completed at once with all 1000, from inside the slab the reactor had freed. Incremental mode read them intact, as the issue says.What changes. The slab has holders, counted in one field,
_bufSlabHolders: the teardown, and every handler that has not let go yet. Each accept adds its handler's hold. The handler'sDecRefgives it up, once per life, behind the gate that already makesDecRefonce per life. The teardown gives up its own hold where it used to free the slab. Whichever lets go last frees it, on its own thread (ReleaseBufSlab). The kernel stopped using the slab when the teardown unregistered the ring, which it still does before the ring closes. While the reactor runs, the teardown's hold keeps the count above zero.A stop whose handlers all let go during it frees the slab where it did. A handler parked in a read is one: the stop wakes it on the reactor thread and it lets go there, before the teardown's own release (a scratch check found the slab already freed when the join returned, 3 of 3). The buffer ring's control area is still freed at the stop, since nothing queued points into it. Incremental mode counts its handlers too, rather than branch on the mode, and has no shared slab to free.
No reader changes. This is the issue's second option, which also covers what a reader took before the stop. The first, emptying the queue at the stop, can't be done from the teardown, since the queue's one consumer may be reading it on another thread.
What a caller notices.
RecvSlotsxRecvBufferSize, 128 MiB of address space by default) stays allocated after a stop until the last handler still holding a connection lets go, and is freed on that handler's thread. A handler that never lets go keeps it, as the stop already keeps every held connection's write slab, and in incremental mode its buffer ring and slab.DecRef. None per request.Tests (E2E,
StopParkedTests):TcpConnectionPipeReadertook before the stop and still holds stays intact, in receive memory the stop kept;Each one checks that the slab is still allocated, and that a slice lies inside it, before it reads a byte, so on main they fail with a message rather than a fault. Taking out each part of the fix fails its own tests:
the receive slab the read points into was freed at the stop, under a connection its handler still holds; 6:the receive slab was freed at the stop, under two connections their handlers still holdDecRefthe receive slab outlived a stop whose handlers have all let go; 6:the receive slab outlived the last handler to let go3 and 5 pass on main. With the fix all pass.
Suites. On the head, built and run pinned to the safe CPUs: Unit 263/0, E2E 736/0 (730 before, six added), Tls 269/0 with the
tlsmodule loaded, so the kTLS tests ran and nothing was skipped, Chaos 105/0, Http 104/0 and File 13/0. Pg and Redis weren't run: they need their servers, and they drive ring client ops, not accepted connections.Overlaps. None of the open PRs #594 to #604 touches these files, and
git merge-tree --write-tree origin/fix-<n> HEADexits 0 for each. Two of them work around this bug in their own readers, and either merge order works:HopDuplexPipereceive pump once its reactor has stopped, and gives the freed slices as the reason. Its guard still ends the pump; without it,CopySliceno longer reads freed memory.The parallel issues in the same teardown:
DecRefgives up the slab hold afterReleaseRef, whatever that then does, and freeing a connection reads nothing in the slab, so the two compose in either order. They may meet textually inTeardown, where the pool drain sits above the slab's release, and inDecRef: keep both. Until reactor: a connection its handler still holds at the stop is never freed, and keeps its descriptor open if its peer closed first #606 lands, a Kestrel connection held at the stop never lets go, as itsDecRefis posted to the stopped reactor, which drops it (from reading, as reactor: a connection its handler still holds at the stop is never freed, and keeps its descriptor open if its peer closed first #606 says). Its reactor's slab then stays allocated, where main freed it under a reader that could still use it. reactor: a connection its handler still holds at the stop is never freed, and keeps its descriptor open if its peer closed first #606's fix of that release frees it.For the reviewer.
DecRef's once-per-life gate, and adds no list or sweep.DecRefto be the handler's release of a connection the reactor accepted, which is its contract. ADecRefon a connection the reactor never handed out would give up a hold nobody took. The tests that build aTcpConnectionwith its public constructor never call it.