Skip to content

reactor: a ring op submitted from a completion the stop gives fails as one handed over after the stop does, rather than never completing (#609) - #621

Closed
MDA2AV wants to merge 1 commit into
mainfrom
fix-609
Closed

MDA2AV wants to merge 1 commit into
mainfrom
fix-609

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Fixes #609.
Touches core: src/ioxide/Reactor/Reactor.RingHost.cs, because the op is stranded by the reactor's own stop, in its op-slot table and its SQ, and only the reactor can refuse it there.

What was wrong

At the stop, CompleteParkedOps cancels each parked client op, frees its slot and then completes it with -ECANCELED, inline on the reactor thread. A waiter that submits a ring op again from that completion is on the reactor thread, so the submit takes a slot at once: the one just freed, from the top of the free stack, which the loop has already passed. Its SQE stays in an SQ that nothing submits again, nothing completes it, and the ring is then disposed. Whatever awaits it waits forever.

ClientTransport gets there for a name with more than one address: a connect the stop cancels fails, and the connect to the next address is submitted inline from that failure. The pool's open never ends, and neither do the requests queued on it. A handler that retries a ring op inline (a RingTimer wait in a loop, a reconnect) gets the same.

Reproduced at origin/main with a scratch program, each case with its op parked when another thread stops the reactor: a RingTimer waited on again, a RingSocket connected again, and ClientTransport with a silent first address and a refusing second one. All three were still waiting 3 s after the reactor had stopped and been joined. With this change all three end with ObjectDisposedException.

What changes

src/ioxide/Reactor/Reactor.RingHost.cs only.

  • CompleteParkedOps empties the free stack before it completes anything, and gives no freed slot back to it. The table goes with the ring, so nothing needs those slots again.
  • AllocOpSlot's grow branch throws ObjectDisposedException once the stop has emptied the stack, as HandOff already does for an op handed over from another thread after the stop.

No new field, and nothing added to a submit's path: the test is in the grow branch, which a submit takes only when no slot is free. A failed final submit is handled as before: when the SQ can't be emptied, CompleteParkedOps doesn't run and parked ops stay parked.

What a caller would notice

  • A ring op submitted on the reactor thread from a completion the stop gives (through RingSocket, RingTimer, RingFile, the pg and http clients, or IRingHost directly) now throws ObjectDisposedException ("Cannot access a disposed object. Object name: 'ioxide.Reactor'."), as one handed over from another thread after the stop does. On main it was accepted and never completed.
  • ClientTransport, after the stop cancels its connect to a name's first address, now fails with that exception rather than waiting forever. A pool's open then fails, and the requests queued on it end at the stop with "The reactor has stopped.", as they already did for a name with one address. The pool logs one [httpclient] connect to ... failed line for it.
  • A handler that doesn't catch it ends with the reactor's "connection handler faulted" line at the stop, where it hung silently before.

Tests

  • E2E, StopParkedTests: stop: a RingTimer wait a handler starts again from the one the stop ended fails rather than waiting forever. A handler parks in a RingTimer wait and the test stops the reactor. Once the stop ends that wait, the handler waits again, inline. It must end within 3 s, with ObjectDisposedException.
  • Http, HttpClientConnectTests: httpclient h1: a connect its reactor's stop cancels ends with the stop, rather than going on to the next address and waiting forever. ClientTransport.ConnectAsync to 127.0.0.3, whose full accept queue drops the SYN, then to 127.0.0.2, which refuses at once if it is ever dialed. The connect is checked to be parked, the reactor is stopped and joined, and the connect must end within 3 s with ObjectDisposedException.

Reverted:

  • The whole change (main's Reactor.RingHost.cs): the E2E test fails with the handler is still parked in the wait it started again, and the Http test with the connect is still waiting 3 s after its reactor stopped. The other 11 stop tests and the other 104 Http tests pass.
  • Only the emptying (main's CompleteParkedOps, the check kept): the E2E test fails the same way, as the second wait takes the slot just freed.
  • Only the check (the stack emptied, nothing thrown): the E2E test fails the same way, as the grow branch doubles the table and hands out a new slot.

Suites

Built and run pinned to CPUs 0-3, 6-7 and 14-31:

  • Unit: 263 passed, 0 failed
  • E2E: 730 passed, 1 failed. The failure is quic/fleet: a handshake under way when another reactor binds the port completes: server on :32896 failed to start: tcp bind failed on port 32896, a port another process held while the fleet started: StartQuicSharded doesn't move to another port on a collision as StartConfigured does, and other sessions run servers on this box. It fails before any stop. Rerun alone 3 times: 3 passed.
  • Http: 105 passed, 0 failed (main's 104 plus the one above), run again after the reverts with the same result
  • Tls: 269 passed, 0 failed
  • File: 13 passed, 0 failed
  • Chaos: 105 passed, 0 failed
  • Pg: 7 passed, 19 skipped, and Redis: 6 skipped, 1 pending (the SE.Redis one already there), with no servers here

Overlaps

For the reviewer

  • An inline retry loop that ignores the error result ends with the throw. One that catches every exception and submits again at once, with nothing else to await, now spins inside the stop on the reactor thread, where on main it parked on its second op. The same loop off the reactor already spins after a stop, as HandOff throws there too. Completing such an op later on the thread pool, as DrainStoppedOps does for a handoff that raced the stop, would move that spin off the reactor, but would also resume the handler off it, once the stop has passed its connection by, which then keeps its memory.
  • The throw comes out of the waiter's own submit, inside its continuation, so an async method gets it as its own exception. A custom IRingCompletion whose Complete submits directly, not through an awaiter, would throw out of CompleteParkedOps and end the teardown early. On main any Complete that throws does the same.
  • From reading, not tested: nothing can enter the ring from these completions any more either. On main, a resubmit that grew the op table or found the SQ full submitted what was queued, which could include the SQE of an op the stop had already completed as cancelled because its cancel found it still in the SQ.
  • ObjectDisposedException.ThrowIf(..., this) matches HandOff, so the message is the generic one, where ReactorSwitch says "The reactor has stopped."

…s one handed over after the stop does, rather than never completing (#609)

At the stop, CompleteParkedOps freed each parked client op's slot and then completed it with -ECANCELED, inline on the reactor thread. A waiter that submitted a ring op again from that completion took a slot at once: the one just freed, from the top of the free stack, which the loop had already passed. Its SQE stayed in an SQ that nothing submitted again, nothing completed it, and the ring was then disposed, so whatever awaited it waited forever. ClientTransport got there for a name with more than one address: a connect the stop cancelled was followed inline by one to the next address, which never ended, and neither did the pool's open or the requests queued on it. A handler that retries a ring op inline got the same.

CompleteParkedOps now empties the free stack before it completes anything and gives no slot back to it, so such a submit reaches AllocOpSlot's grow branch, which throws ObjectDisposedException once the stop has emptied the stack, as HandOff does for an op handed over from another thread after the stop. The test sits where only a submit to a full table goes, so no submit pays for it. The throw also ends an inline retry loop that ignores an error result. A failed final submit is handled as before: CompleteParkedOps does not run and parked ops stay parked.

Tests: in E2E's StopParkedTests, a handler that waits on its RingTimer again once the stop ended its first wait gets ObjectDisposedException. In Http's HttpClientConnectTests, a ClientTransport connect to a first address the stop cancels ends with ObjectDisposedException, instead of going on to a connect to the next address that never ends. On main both are still waiting 3 s after the stop, and without either half of the change the E2E test still fails.
@MDA2AV

MDA2AV commented Nov 14, 2026

Copy link
Copy Markdown
Owner Author

Closed in favour of #634: a stop that drains, then frees everything in one place, instead of a fix per after-stop path.

@MDA2AV MDA2AV closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reactor: a ring op submitted from a completion the stop cancels is never completed, so its waiters wait forever

1 participant