Repository navigation
Closed
Conversation
…en it lets go, and its socket closed if its peer closed first (#606) The stop kept a held connection's memory, as its handler may still write to it, but nothing freed it once that handler let go. One still in the table kept the reactor's ref, so the release took the count to 1. One the reactor had already let go of (peer EOF, a recv error, a receive-queue overflow) reached 0 and went on the recycle queue, which only the loop drains, so its descriptor stayed open as well. Now the stop lets go of the reactor's ref on a held connection it drops from the table, and marks the socket it closed for it with ClientFd -1. A recycle on a stopped reactor, or one still queued when the loop ended, is done by the thread that let go, or by the teardown's last step: the descriptor is closed unless the stop closed that number, which may be another socket's by then, and the memory freed, overflow segments included. A connection whose send the kernel may still hold keeps its slab, as before. In incremental mode the stop unregisters every connection ring still registered while the ring is open: those the table can't reach, whose peer closed first, included.
This was referenced Nov 14, 2026
Owner
Author
|
Closed in favour of #634: a stop that drains, then frees everything in one place, instead of a fix per after-stop path. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #606.
Touches core:
src/ioxide/Reactor/Transport/Tcp/Reactor.Tcp.cs,src/ioxide/Reactor/Loop/Reactor.Drainers.cs,src/ioxide/Reactor/Loop/Reactor.Loop.Incremental.cs,src/ioxide/Reactor/Reactor.Runner.cs,src/ioxide/Connection/Tcp/TcpConnection.cs, because what keeps such a connection is the reactor's own ref and its recycle queue, which only the core can let go of or drain.What was wrong
The stop keeps the memory of a connection its handler still holds, since that handler may still write to it. Nothing freed it once the handler let go:
DecReftook the count from 2 to 1, and only 0 recycles. Its write slab stayed allocated, and in incremental mode its buffer ring and slab too.CloseFromRecvhad taken it out of the table and released the reactor's ref, so the stop never saw it. The handler'sDecReffrom another thread reached 0 and queued it on_recycleQ, which only the loop drains: its memory leaked, and its socket stayed open in CLOSE_WAIT, so its peer never saw the server's side close.Reproduced at main 19aec5c with a scratch program: a connection held in each of the three ways, a stop, then the handler's release. 6 of 6 leaked (both receive modes, three cases each): the write slab, and in incremental mode the ring, were never freed, and where the peer had closed first the server's descriptor still named the same socket afterwards and the client never read the close.
What changes
CloseAcceptedTcpSocketssetsClientFdto -1 on each connection whose socket it closes: by the time its handler lets go, the number may be another socket's (Releasing an fd number mid-run kills an unrelated live connection #242).ReturnReleasedTcpConnectionsreleases the reactor's ref on a connection a handler still holds, so that handler'sDecRefreaches 0. One whose send the kernel may still hold, a flush the stop could not end or a zero-copy one the stop dropped from the table before this, keeps its ref and its slab, as before.EnqueueRecyclefrom another thread checks_postsClosedafter it queues, asScheduleOnReactordoes, and on a stopped reactor drains the queue on that thread. The teardown drains it as well, right after it sets_postsClosed, so a release queued before the stop is not stranded.DrainStoppedRecyclescloses the descriptor unlessClientFdis -1, then callsDispose; it touches no ring and returns no buffer.TcpConnection.Disposefrees the kept overflow segments of segmented writes. Every caller ranClearfirst, which frees them; this one does not.UnregisterHeldConnectionBufRingsunregisters every per-connection ring still registered (each gid not on the free stack), those of connections whose peer closed first included, so memory a later release frees is no longer the kernel's.The reactor-thread recycle is unchanged; the one from another thread adds a volatile read after its wake.
Behaviour a caller would notice
TcpConnection.ClientFdis -1 on every connection whose socket the stop closed, as its XML doc now says. Code that used the number after the stop reached a closed descriptor or another socket; it now gets EBADF.DecRef, on its thread. Code that touches a connection's buffers after itsDecRef, which a running reactor would already have recycled under it, now touches freed memory after a stop, where it used to touch leaked memory.DecRef: the peer gets the FIN, or a reset for what was never read.Tests
E2E,
Core/StopParkedTests.cs(each in both receive modes) andCore/TcpTimeoutTests.cs:stop: a connection its handler let go of has its memory freed, one a handler still holds keeps it until it lets go(extended: the held one is freed once it lets go)EnqueueRecycle:the write slab of the connection its handler let go of after the stop outlived that releasestop: a connection whose peer closed first, let go of after the stop, has its socket closed and its memory freed(new; its handler stages 64 KiB it never flushes)EnqueueRecycle:the write slab outlived its handler's release. The segment free inDispose, shared ring:2 overflow segments outlived its handler's release. The close inDrainStoppedRecycles:the server's socket, whose peer closed first, is still open after its handler let gostop: a connection let go of just before the stop, its recycle still queued, has its socket closed and its memory freed(new; the handler lets go and the stop is asked for while the reactor runs a post)the write slab outlived its handler's release. The segment free inDispose, shared ring:2 overflow segments outlived its handler's release. The close inDrainStoppedRecycles: the samestill openas abovetcp/exit: a handler that lets go after its reactor stopped leaves a reused fd aloneandtcp/half-close: a ShutdownWrite after its reactor stopped leaves a reused fd alone(existing; their helper's comment updated)ClientFd-1:expected [ok], got [the unrelated socket failed: OperationAborted], the release closing the number another socket had takenWith the whole core change reverted, the six new and extended tests fail as in the table. The ring sweep has no test that fails without it; see below.
Suites
Pinned to CPUs 0-3,6-7,14-31, one suite at a time, on this box's 6.14 kernel, so the incremental variants run too. Unit and E2E ran on the build before two comments were rewrapped, the rest on this commit:
StopParkedTestsalone, 15 rounds: 225 passed, 0 failed. In a 3-round run of it withTcpTimeoutTestsandReactorSetupTeardownTests, round 3 failedreactor: a connection recycled into a full pool has its write slab freed right therewith0 of the two recycled connections had their slab freed with the pool full: the standalone race #590 describes and #598 fixes, on a path this does not change (a recycle on a running reactor). It passed in the full E2E run and the other two rounds.Overlaps
No shared files with #594 to #604, and
git merge-tree --write-tree origin/fix-<n> HEADexits 0 for each.ScheduleOnReactor, which a stopped reactor drops, so after a stop thatDecRefnever runs and the connection is never freed. On main a connection live at the stop does not even get to that post, as its dispose never finishes (kestrel: a stopped reactor drops the pipe continuations ReactorPipeScheduler posts, so its connections' pumps and DisposeAsync wait forever #583); with kestrel: a connection whose reactor stops ends in Kestrel and finishes its dispose, instead of waiting forever on pipe continuations the reactor dropped (#583) #600, one whose app outlives the host's shutdown gets there. Posting the release the way kestrel: a connection whose reactor stops ends in Kestrel and finishes its dispose, instead of waiting forever on pipe continuations the reactor dropped (#583) #600's scheduler posts its drain, aReactorSwitchpost that a stopped reactor runs on the thread pool, would let it free the connection, but it can only be reached and tested on top of kestrel: a connection whose reactor stops ends in Kestrel and finishes its dispose, instead of waiting forever on pipe continuations the reactor dropped (#583) #600, so it is left for after that.DrainStoppedRecycles.EnqueueRecycleis unchanged. Until reactor: the reactor-thread checks compare managed thread ids, which the runtime reuses, so after a stop a new thread can pass for the reactor's #608, a release after the stop from a thread that passes for the reactor's takesRecycleagainst the torn-down ring: a connection whose peer closed first could already get there, and now one held at the stop can too. With reactor: the reactor-thread checks compare managed thread ids, which the runtime reuses, so after a stop a new thread can pass for the reactor's #608'st_currenttest, cleared at the end of the teardown, such a release takes the queued path and the_postsClosedcheck here frees it. The two change different lines.For the reviewer
MaxConnections + 2bools and one register call per ring still registered, at the stop only.HandlerReleasedcheck and its release makes the stop's release the last: the connection is recycled right there, while the ring is open, withClientFd-1, so that recycle'sclosefails with EBADF.Clear, which bumps the reactor's generation counter (reactor-thread state), and returns no receive buffer: the shared ring is gone by then, and an incremental ring goes with its connection.