Skip to content

reactor: a buffer return that finds its queue full after the stop is dropped, rather than spinning forever on a queue nothing drains (#607) - #619

Closed
MDA2AV wants to merge 1 commit into
mainfrom
fix-607
Closed

MDA2AV wants to merge 1 commit into
mainfrom
fix-607

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Fixes #607.
Touches core: src/ioxide/Reactor/Loop/Reactor.Drainers.cs and src/ioxide/Reactor/Loop/Reactor.Loop.Incremental.cs, because the return queues, their spin and the flag the stop sets are the reactor's own: every recv buffer a handler gives back from another thread goes through EnqueueReturnQ or EnqueueReturnQIncremental, and no satellite module sees either queue.

What was wrong

Off the reactor, EnqueueReturnQ spins until _returnQ takes the buffer id, and EnqueueReturnQIncremental does the same on _returnQInc. Only the loop drains those queues. After the stop every return from another thread kept its entry, and once all 16384 (shared ring) or 65536 (incremental) were taken, the next return spun forever and held its thread, typically a thread-pool worker.

Reproduced at main 19aec5c with a scratch program: a real reactor whose handlers hold recv buffers across the stop and give them back from the thread pool afterwards.

  • Shared ring, RecvSlots 32768 (the most the reactor accepts; at the default 4096 this queue cannot fill, since each id is out at most once), one connection holding 17000 buffers: 16384 of the 17000 returns went through, and the next was still spinning 5 s later.
  • Incremental, 32768 one-byte buffers per connection, three connections holding 22000 slices each: 65536 of 66000 went through. This queue takes an entry per slice rather than per buffer, so it can fill at the default sizes too, given enough connections holding slices.
  • The same through the public entry points after a real stop: 16384 of 16385, and 65536 of 65537.

What changes

A return that finds its queue full gives up once _postsClosed is set. The end of Teardown sets it, and from then on nothing drains the queue and no ring is left to take the buffer. The test sits inside the spin:

  • A return that finds room runs exactly what it ran before. In the JIT's tier-1 code for EnqueueReturnQ (x64, Dynamic PGO) the two versions are identical up to the full-queue branch, which gains one byte compare and a jump (257 to 268 bytes).
  • A return that found the queue full while the reactor was running, or still stopping, waits for the drain as before, and leaves when the teardown ends instead of never.

Returns after the stop that still find room take an entry, as before. Nothing reads them, and the queue is a managed array that goes with the reactor.

Behaviour a caller would notice

Only after the stop: a buffer return from another thread that finds the queue full now returns, where it never returned. While the reactor runs nothing changes, and a full queue still makes a return wait for the loop's drain.

Tests

Each fails when its half of the fix is reverted, and passes with it. Reverting the shared half fails exactly the two shared-ring tests; reverting the incremental half fails exactly the two incremental ones.

  • Unit, StoppedReturnTests: "reactor: a buffer return that finds its queue full waits while the reactor runs, and leaves when it stops (shared ring / incremental)". The real returns, on a reactor that never runs: the test fills the queue, starts one more return, checks it is still waiting 200 ms later, then sets the flag the end of Teardown sets and requires that return to finish within 3 s. Reverted: a return waiting on the full queue still spins after the stop, where nothing will drain it. It also catches the two near misses the E2E test cannot see: dropping a return on any full queue fails with a return that found the queue full while the reactor runs was dropped instead of waiting for room, and testing the flag only before the spin fails with the first message.
  • E2E, StopParkedTests: "stop: handlers holding more buffers than the return queue takes give them all back from another thread after the stop (shared ring / incremental)". The repro as a test, with one-byte buffers so each byte sent is a buffer, or a slice, of its own. Reverted: 16384 of the 17000 buffers went back after the stop, and the next return still spins on a queue nothing drains (shared ring) and 65536 of the 66000 buffers went back after the stop, and the next return still spins on a queue nothing drains (incremental). The incremental one skips below kernel 6.12, as the other incremental stop test does.

Suites

Built and run pinned to the safe CPUs, one at a time:

  • Unit: 265 passed, 0 failed, 0 skipped, the two new tests among them.
  • E2E: 732 passed, 0 failed, 0 skipped, the two new tests among them.
  • Tls: 269 passed, 0 failed, 0 skipped.
  • Chaos: 105 passed, 0 failed, 0 skipped.
  • The four new tests and the rest of StopParkedTests, run 10 more times with the box at load average 12: no failure.

CI passed on x64 and arm64, the four new tests among them on both. The first arm64 attempt failed one E2E test, "h2 off-reactor (nghttp2): many streams flushing from the thread pool at once all resume on the reactor", right after [conn] recv queue overflow; closing connection.: the client's frames filled the connection's 64-entry receive queue before the frame loop read them, which in the shared receive mode closes the connection. That happens while the reactor runs, where this change does nothing, and the rerun of that job passed.

Overlaps

For the reviewer

  • The flag is _postsClosed, the one point after which nothing drains the queue, which the stopped posts and ops already use. _stopRequested comes earlier, while the loop may still be on its last pass.
  • A dropped return puts its buffer back nowhere, and there is nowhere left: the same teardown closed the io_uring every buffer ring was registered with, and freed the shared one.

…dropped, rather than spinning forever on a queue nothing drains (#607)

Off the reactor, EnqueueReturnQ and EnqueueReturnQIncremental spin until their queue takes the buffer,
and only the loop drains those queues. After the stop every return from another thread kept its entry,
so once the 16384 entries of the shared ring's queue, or the 65536 of the incremental one, were taken,
the next return spun forever and held its thread: a handler that held buffers across the stop and gave
them back from the thread pool never finished.

A return that finds its queue full now gives up once the reactor's posts are closed, at the end of
Teardown, when nothing will drain the queue and no ring is left to take the buffer. The test sits
inside the spin, so a return that finds room runs the same code as before.

StoppedReturnTests (Unit): a return that finds the queue full waits while the reactor runs, and leaves
when the stop lands, in both modes. StopParkedTests (E2E): handlers holding more buffers than the
queue takes, 17000 on the shared ring and 3 x 22000 slices incremental, give them all back from the
thread pool after the stop.
@MDA2AV

MDA2AV commented Nov 14, 2026

Copy link
Copy Markdown
Owner Author

Closed in favour of #634: a stop that drains, then frees everything in one place, instead of a fix per after-stop path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reactor: after the stop, buffer returns from other threads fill a queue nothing drains, and the one that finds it full spins forever

1 participant