Skip to content

reactor: a connection its handler still holds at the stop is freed when it lets go, and its socket closed if its peer closed first (#606) - #623

Closed
MDA2AV wants to merge 1 commit into
mainfrom
fix-606
Closed

MDA2AV wants to merge 1 commit into
mainfrom
fix-606

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Part of #606.
Touches core: src/ioxide/Reactor/Transport/Tcp/Reactor.Tcp.cs, src/ioxide/Reactor/Loop/Reactor.Drainers.cs, src/ioxide/Reactor/Loop/Reactor.Loop.Incremental.cs, src/ioxide/Reactor/Reactor.Runner.cs, src/ioxide/Connection/Tcp/TcpConnection.cs, because what keeps such a connection is the reactor's own ref and its recycle queue, which only the core can let go of or drain.

What was wrong

The stop keeps the memory of a connection its handler still holds, since that handler may still write to it. Nothing freed it once the handler let go:

  • Still in the table at the stop. The stop closed its socket and dropped it from the table but kept the reactor's ref, so the handler's DecRef took the count from 2 to 1, and only 0 recycles. Its write slab stayed allocated, and in incremental mode its buffer ring and slab too.
  • Already let go of by the reactor (its peer closed first, a recv error, a receive-queue overflow). CloseFromRecv had taken it out of the table and released the reactor's ref, so the stop never saw it. The handler's DecRef from another thread reached 0 and queued it on _recycleQ, which only the loop drains: its memory leaked, and its socket stayed open in CLOSE_WAIT, so its peer never saw the server's side close.
  • Let go of just before the stop. A release queued once the loop's last pass was past its recycle drain stayed queued the same way.

Reproduced at main 19aec5c with a scratch program: a connection held in each of the three ways, a stop, then the handler's release. 6 of 6 leaked (both receive modes, three cases each): the write slab, and in incremental mode the ring, were never freed, and where the peer had closed first the server's descriptor still named the same socket afterwards and the client never read the close.

What changes

  • CloseAcceptedTcpSockets sets ClientFd to -1 on each connection whose socket it closes: by the time its handler lets go, the number may be another socket's (Releasing an fd number mid-run kills an unrelated live connection #242).
  • ReturnReleasedTcpConnections releases the reactor's ref on a connection a handler still holds, so that handler's DecRef reaches 0. One whose send the kernel may still hold, a flush the stop could not end or a zero-copy one the stop dropped from the table before this, keeps its ref and its slab, as before.
  • EnqueueRecycle from another thread checks _postsClosed after it queues, as ScheduleOnReactor does, and on a stopped reactor drains the queue on that thread. The teardown drains it as well, right after it sets _postsClosed, so a release queued before the stop is not stranded. DrainStoppedRecycles closes the descriptor unless ClientFd is -1, then calls Dispose; it touches no ring and returns no buffer.
  • TcpConnection.Dispose frees the kept overflow segments of segmented writes. Every caller ran Clear first, which frees them; this one does not.
  • Incremental mode: before the ring is closed, UnregisterHeldConnectionBufRings unregisters every per-connection ring still registered (each gid not on the free stack), those of connections whose peer closed first included, so memory a later release frees is no longer the kernel's.

The reactor-thread recycle is unchanged; the one from another thread adds a volatile read after its wake.

Behaviour a caller would notice

  • After a stop, TcpConnection.ClientFd is -1 on every connection whose socket the stop closed, as its XML doc now says. Code that used the number after the stop reached a closed descriptor or another socket; it now gets EBADF.
  • A connection its handler held through the stop is freed at that handler's DecRef, on its thread. Code that touches a connection's buffers after its DecRef, which a running reactor would already have recycled under it, now touches freed memory after a stop, where it used to touch leaked memory.
  • A connection whose peer closed first, held through the stop, has its socket closed at its handler's DecRef: the peer gets the FIN, or a reset for what was never read.

Tests

E2E, Core/StopParkedTests.cs (each in both receive modes) and Core/TcpTimeoutTests.cs:

test fails with, when its part is reverted
stop: a connection its handler let go of has its memory freed, one a handler still holds keeps it until it lets go (extended: the held one is freed once it lets go) the stop's release of the reactor's ref, or the stopped drain in EnqueueRecycle: the write slab of the connection its handler let go of after the stop outlived that release
stop: a connection whose peer closed first, let go of after the stop, has its socket closed and its memory freed (new; its handler stages 64 KiB it never flushes) the stopped drain in EnqueueRecycle: the write slab outlived its handler's release. The segment free in Dispose, shared ring: 2 overflow segments outlived its handler's release. The close in DrainStoppedRecycles: the server's socket, whose peer closed first, is still open after its handler let go
stop: a connection let go of just before the stop, its recycle still queued, has its socket closed and its memory freed (new; the handler lets go and the stop is asked for while the reactor runs a post) the teardown's drain: the write slab outlived its handler's release. The segment free in Dispose, shared ring: 2 overflow segments outlived its handler's release. The close in DrainStoppedRecycles: the same still open as above
tcp/exit: a handler that lets go after its reactor stopped leaves a reused fd alone and tcp/half-close: a ShutdownWrite after its reactor stopped leaves a reused fd alone (existing; their helper's comment updated) ClientFd -1: expected [ok], got [the unrelated socket failed: OperationAborted], the release closing the number another socket had taken

With the whole core change reverted, the six new and extended tests fail as in the table. The ring sweep has no test that fails without it; see below.

Suites

Pinned to CPUs 0-3,6-7,14-31, one suite at a time, on this box's 6.14 kernel, so the incremental variants run too. Unit and E2E ran on the build before two comments were rewrapped, the rest on this commit:

  • Unit: 263 passed, 0 failed, 0 skipped.
  • E2E: 734 passed, 0 failed, 0 skipped: main's 730 and the four new tests.
  • Tls: 269 passed, 0 failed, 0 skipped.
  • Chaos: 105 passed, 0 failed, 0 skipped.
  • Http: 104 passed, 0 failed, 0 skipped.
  • File: 13 passed, 0 failed, 0 skipped.

StopParkedTests alone, 15 rounds: 225 passed, 0 failed. In a 3-round run of it with TcpTimeoutTests and ReactorSetupTeardownTests, round 3 failed reactor: a connection recycled into a full pool has its write slab freed right there with 0 of the two recycled connections had their slab freed with the pool full: the standalone race #590 describes and #598 fixes, on a path this does not change (a recycle on a running reactor). It passed in the full E2E run and the other two rounds.

Overlaps

No shared files with #594 to #604, and git merge-tree --write-tree origin/fix-<n> HEAD exits 0 for each.

For the reviewer

  • The ring sweep. A held connection still in the table has its multishot recv armed when the stop closes its socket (the request keeps the socket), so what its peer sends next can land in its ring until the ring's deferred teardown cancels the request, and that ring is now freed when the handler lets go. So the teardown unregisters it first, the rule it already applies to the shared ring ("unregistering is synchronous, closing is not"). No test fails without it: what it prevents is the kernel writing into freed memory, which no test sees deterministically. It costs one array of MaxConnections + 2 bools and one register call per ring still registered, at the stop only.
  • A handler that lets go between the stop's HandlerReleased check and its release makes the stop's release the last: the connection is recycled right there, while the ring is open, with ClientFd -1, so that recycle's close fails with EBADF.
  • The free does not call Clear, which bumps the reactor's generation counter (reactor-thread state), and returns no receive buffer: the shared ring is gone by then, and an incremental ring goes with its connection.
  • No wrk run: the reactor-thread recycle path is untouched, and the rest runs at the stop.

…en it lets go, and its socket closed if its peer closed first (#606)

The stop kept a held connection's memory, as its handler may still write to it, but nothing freed it
once that handler let go. One still in the table kept the reactor's ref, so the release took the count
to 1. One the reactor had already let go of (peer EOF, a recv error, a receive-queue overflow) reached
0 and went on the recycle queue, which only the loop drains, so its descriptor stayed open as well.

Now the stop lets go of the reactor's ref on a held connection it drops from the table, and marks the
socket it closed for it with ClientFd -1. A recycle on a stopped reactor, or one still queued when the
loop ended, is done by the thread that let go, or by the teardown's last step: the descriptor is closed
unless the stop closed that number, which may be another socket's by then, and the memory freed,
overflow segments included. A connection whose send the kernel may still hold keeps its slab, as before.

In incremental mode the stop unregisters every connection ring still registered while the ring is
open: those the table can't reach, whose peer closed first, included.
@MDA2AV

MDA2AV commented Nov 14, 2026

Copy link
Copy Markdown
Owner Author

Closed in favour of #634: a stop that drains, then frees everything in one place, instead of a fix per after-stop path.

@MDA2AV MDA2AV closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant