Skip to content

reactor: a stop leaves the shared receive slab to the last handler still holding a connection, so a read after it gets the bytes queued before it, not freed memory (#605) - #620

Closed
MDA2AV wants to merge 1 commit into
mainfrom
fix-605

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Fixes #605.
Touches core: src/ioxide/Reactor/Reactor.cs, src/ioxide/Reactor/Reactor.Runner.cs, src/ioxide/Reactor/Loop/Reactor.Loop.SharedRing.cs, src/ioxide/Reactor/Transport/Tcp/Reactor.Tcp.cs and src/ioxide/Connection/Tcp/TcpConnection.cs, because the shared receive slab, the teardown that frees it, the accept and the handler's release are all the reactor's own, and no satellite can tell when the last reader of that slab is done.

What was wrong. In the default shared receive mode every item a connection queues, and every slice a reader takes from that queue, points into the reactor's one receive slab, and Teardown freed that slab unconditionally. The stop empties the queue only of a connection whose handler has already let go (Clear in ReturnReleasedTcpConnections). A connection its handler still held kept what was queued. That is one still in the table at the stop, and also one its client had closed before the stop, which CloseFromRecv takes out of the table, so the stop never sees it. A read after the stop serves that queue before it reports the close, so ReadAsync with TryGetItem, TcpConnectionPipeReader (and kTLS RX on top of it) and TcpConnectionStream handed out slices of freed memory. So did every slice a reader took before the stop and still held.

Reproduced at main with a scratch program (a 1 MiB slab, which glibc maps on its own and unmaps when freed): reading 1000 bytes queued before the stop faulted the process with AccessViolationException for a connection open at the stop, for one its client closed before the stop, and for a TcpConnectionPipeReader sequence held across the stop, 3 of 3 runs each. Without reading the bytes, each read completed at once with all 1000, from inside the slab the reactor had freed. Incremental mode read them intact, as the issue says.

What changes. The slab has holders, counted in one field, _bufSlabHolders: the teardown, and every handler that has not let go yet. Each accept adds its handler's hold. The handler's DecRef gives it up, once per life, behind the gate that already makes DecRef once per life. The teardown gives up its own hold where it used to free the slab. Whichever lets go last frees it, on its own thread (ReleaseBufSlab). The kernel stopped using the slab when the teardown unregistered the ring, which it still does before the ring closes. While the reactor runs, the teardown's hold keeps the count above zero.

A stop whose handlers all let go during it frees the slab where it did. A handler parked in a read is one: the stop wakes it on the reactor thread and it lets go there, before the teardown's own release (a scratch check found the slab already freed when the join returned, 3 of 3). The buffer ring's control area is still freed at the stop, since nothing queued points into it. Incremental mode counts its handlers too, rather than branch on the mode, and has no shared slab to free.

No reader changes. This is the issue's second option, which also covers what a reader took before the stop. The first, emptying the queue at the stop, can't be done from the teardown, since the queue's one consumer may be reading it on another thread.

What a caller notices.

  • After a stop, a read of a connection its handler still holds gets the bytes queued before the stop, intact, and then the close, as incremental mode already did. Before, it got slices of freed memory.
  • The shared slab (RecvSlots x RecvBufferSize, 128 MiB of address space by default) stays allocated after a stop until the last handler still holding a connection lets go, and is freed on that handler's thread. A handler that never lets go keeps it, as the stop already keeps every held connection's write slab, and in incremental mode its buffer ring and slab.
  • Two atomic operations per connection, one at accept and one at DecRef. None per request.

Tests (E2E, StopParkedTests):

  1. a read from another thread after the stop gets the bytes queued before it, intact, from receive memory the stop kept, for a connection open at the stop;
  2. the same, for a connection its client closed before the stop;
  3. control: in incremental mode the same read gets them intact;
  4. what a TcpConnectionPipeReader took before the stop and still holds stays intact, in receive memory the stop kept;
  5. the receive slab is freed once the handlers the stop woke let go;
  6. the receive slab outlives the stop until the last of two handlers still holding a connection lets go.

Each one checks that the slab is still allocated, and that a slice lies inside it, before it reads a byte, so on main they fail with a message rather than a fault. Taking out each part of the fix fails its own tests:

Without Fails
the whole fix (main's sources) 1, 2 and 4: the receive slab the read points into was freed at the stop, under a connection its handler still holds; 6: the receive slab was freed at the stop, under two connections their handlers still hold
the teardown's release, main's unconditional free in its place (the handlers' holds kept) the same four, with the same messages
the handler's release in DecRef 5: the receive slab outlived a stop whose handlers have all let go; 6: the receive slab outlived the last handler to let go
the teardown's release, with nothing in its place 5 and 6, with the same two messages

3 and 5 pass on main. With the fix all pass.

Suites. On the head, built and run pinned to the safe CPUs: Unit 263/0, E2E 736/0 (730 before, six added), Tls 269/0 with the tls module loaded, so the kTLS tests ran and nothing was skipped, Chaos 105/0, Http 104/0 and File 13/0. Pg and Redis weren't run: they need their servers, and they drive ring client ops, not accepted connections.

Overlaps. None of the open PRs #594 to #604 touches these files, and git merge-tree --write-tree origin/fix-<n> HEAD exits 0 for each. Two of them work around this bug in their own readers, and either merge order works:

The parallel issues in the same teardown:

For the reviewer.

  • A new count, where existing state would not do. A connection's refcount says who holds that one connection, but the stop can't sum them, since a connection its client closed has left the table. Counting buffer ids out of the ring would need the kernel's ring head, and every return after the stop. The count is touched once at accept and once at the handler's release, reuses DecRef's once-per-life gate, and adds no list or sweep.
  • The count trusts DecRef to be the handler's release of a connection the reactor accepted, which is its contract. A DecRef on a connection the reactor never handed out would give up a hold nobody took. The tests that build a TcpConnection with its public constructor never call it.
  • After a stop the slab is freed on the thread of the last handler to let go, which can be a thread-pool thread.
  • No benchmark: two atomic operations per connection, against an accept and a close, are far below this box's noise floor, and nothing per request changed.

…ill holding a connection, so a read after it gets the bytes queued before it, not freed memory (#605)

In the default shared receive mode what a connection queues, and what a reader takes from that queue, points into the reactor's one receive slab, and Teardown freed it unconditionally. A connection its handler still held kept its queue across the stop: one still in the table, and one its client had closed, which had left the table. A read after the stop served that queue, and a reader still held what it took before the stop, so both handed out slices of freed memory, and reading them faulted the process.

The slab now has holders: the teardown and every handler that has not let go. Each accept adds its handler's hold, the handler's DecRef gives it up once per life, and the teardown gives up its own where it freed the slab. The last to let go frees it, on its own thread; the kernel stopped using it when the teardown unregistered the ring. A stop whose handlers all let go during it, as one parked in a read does, frees it where it did. Incremental mode counts its handlers too and has no shared slab.

Tests in StopParkedTests (E2E): a read after the stop of bytes queued before it, for a connection open at the stop and one its client closed before it, and what a pipe reader took before the stop, each get intact bytes from a slab still allocated; the slab is freed once the handlers the stop woke let go, and outlives the stop until the last of two held connections' handlers lets go. An incremental-mode control reads intact bytes on main too.
@MDA2AV

MDA2AV commented Nov 14, 2026

Copy link
Copy Markdown
Owner Author

Closed in favour of #634: a stop that drains, then frees everything in one place, instead of a fix per after-stop path.

@MDA2AV MDA2AV closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reactor: a read after the stop hands out bytes queued before it, as slices of the shared receive slab the stop freed

1 participant