Repository navigation
Closed
Conversation
…s one handed over after the stop does, rather than never completing (#609) At the stop, CompleteParkedOps freed each parked client op's slot and then completed it with -ECANCELED, inline on the reactor thread. A waiter that submitted a ring op again from that completion took a slot at once: the one just freed, from the top of the free stack, which the loop had already passed. Its SQE stayed in an SQ that nothing submitted again, nothing completed it, and the ring was then disposed, so whatever awaited it waited forever. ClientTransport got there for a name with more than one address: a connect the stop cancelled was followed inline by one to the next address, which never ended, and neither did the pool's open or the requests queued on it. A handler that retries a ring op inline got the same. CompleteParkedOps now empties the free stack before it completes anything and gives no slot back to it, so such a submit reaches AllocOpSlot's grow branch, which throws ObjectDisposedException once the stop has emptied the stack, as HandOff does for an op handed over from another thread after the stop. The test sits where only a submit to a full table goes, so no submit pays for it. The throw also ends an inline retry loop that ignores an error result. A failed final submit is handled as before: CompleteParkedOps does not run and parked ops stay parked. Tests: in E2E's StopParkedTests, a handler that waits on its RingTimer again once the stop ended its first wait gets ObjectDisposedException. In Http's HttpClientConnectTests, a ClientTransport connect to a first address the stop cancels ends with ObjectDisposedException, instead of going on to a connect to the next address that never ends. On main both are still waiting 3 s after the stop, and without either half of the change the E2E test still fails.
This was referenced Nov 14, 2026
Owner
Author
|
Closed in favour of #634: a stop that drains, then frees everything in one place, instead of a fix per after-stop path. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #609.
Touches core:
src/ioxide/Reactor/Reactor.RingHost.cs, because the op is stranded by the reactor's own stop, in its op-slot table and its SQ, and only the reactor can refuse it there.What was wrong
At the stop,
CompleteParkedOpscancels each parked client op, frees its slot and then completes it with-ECANCELED, inline on the reactor thread. A waiter that submits a ring op again from that completion is on the reactor thread, so the submit takes a slot at once: the one just freed, from the top of the free stack, which the loop has already passed. Its SQE stays in an SQ that nothing submits again, nothing completes it, and the ring is then disposed. Whatever awaits it waits forever.ClientTransportgets there for a name with more than one address: a connect the stop cancels fails, and the connect to the next address is submitted inline from that failure. The pool's open never ends, and neither do the requests queued on it. A handler that retries a ring op inline (aRingTimerwait in a loop, a reconnect) gets the same.Reproduced at origin/main with a scratch program, each case with its op parked when another thread stops the reactor: a
RingTimerwaited on again, aRingSocketconnected again, andClientTransportwith a silent first address and a refusing second one. All three were still waiting 3 s after the reactor had stopped and been joined. With this change all three end withObjectDisposedException.What changes
src/ioxide/Reactor/Reactor.RingHost.csonly.CompleteParkedOpsempties the free stack before it completes anything, and gives no freed slot back to it. The table goes with the ring, so nothing needs those slots again.AllocOpSlot's grow branch throwsObjectDisposedExceptiononce the stop has emptied the stack, asHandOffalready does for an op handed over from another thread after the stop.No new field, and nothing added to a submit's path: the test is in the grow branch, which a submit takes only when no slot is free. A failed final submit is handled as before: when the SQ can't be emptied,
CompleteParkedOpsdoesn't run and parked ops stay parked.What a caller would notice
RingSocket,RingTimer,RingFile, the pg and http clients, orIRingHostdirectly) now throwsObjectDisposedException("Cannot access a disposed object. Object name: 'ioxide.Reactor'."), as one handed over from another thread after the stop does. On main it was accepted and never completed.ClientTransport, after the stop cancels its connect to a name's first address, now fails with that exception rather than waiting forever. A pool's open then fails, and the requests queued on it end at the stop with "The reactor has stopped.", as they already did for a name with one address. The pool logs one[httpclient] connect to ... failedline for it.Tests
StopParkedTests:stop: a RingTimer wait a handler starts again from the one the stop ended fails rather than waiting forever. A handler parks in aRingTimerwait and the test stops the reactor. Once the stop ends that wait, the handler waits again, inline. It must end within 3 s, withObjectDisposedException.HttpClientConnectTests:httpclient h1: a connect its reactor's stop cancels ends with the stop, rather than going on to the next address and waiting forever.ClientTransport.ConnectAsyncto 127.0.0.3, whose full accept queue drops the SYN, then to 127.0.0.2, which refuses at once if it is ever dialed. The connect is checked to be parked, the reactor is stopped and joined, and the connect must end within 3 s withObjectDisposedException.Reverted:
Reactor.RingHost.cs): the E2E test fails withthe handler is still parked in the wait it started again, and the Http test withthe connect is still waiting 3 s after its reactor stopped. The other 11 stop tests and the other 104 Http tests pass.CompleteParkedOps, the check kept): the E2E test fails the same way, as the second wait takes the slot just freed.Suites
Built and run pinned to CPUs 0-3, 6-7 and 14-31:
quic/fleet: a handshake under way when another reactor binds the port completes: server on :32896 failed to start: tcp bind failed on port 32896, a port another process held while the fleet started:StartQuicShardeddoesn't move to another port on a collision asStartConfigureddoes, and other sessions run servers on this box. It fails before any stop. Rerun alone 3 times: 3 passed.Overlaps
Reactor.RingHost.csorHttpClientConnectTests.cs.git merge-tree --write-treewith this branch is clean for each of nghttp2: a request says whether a body follows its HEADERS, so a host need not guess from content-length (#581) #594 to http2: an answer waiting on the client's credit has its unread upload credited, so a client that uploads everything first gets it (#588) #604 and for reactor: a buffer return that finds its queue full after the stop is dropped, rather than spinning forever on a queue nothing drains (#607) #619.StopParkedTests.cstoo, elsewhere in the file: it merges cleanly.CompleteParkedOps, the retry then submits nothing, and its waiters resume through a yield the stop fails. A variant of httpclient: a request waiting out a pool's reconnect backoff ends when its reactor stops, rather than waiting forever (#584) #596 that went on to connect after the cancelled wait is where this was found; with this change that connect throws instead of waiting.SubmitClientOpandSubmitTimeout, in this file but in other hunks. After a stop whose final submit succeeded, a submit that passes for the reactor's already throws here; reactor: the reactor-thread checks compare managed thread ids, which the runtime reuses, so after a stop a new thread can pass for the reactor's #608 covers the rest.For the reviewer
HandOffthrows there too. Completing such an op later on the thread pool, asDrainStoppedOpsdoes for a handoff that raced the stop, would move that spin off the reactor, but would also resume the handler off it, once the stop has passed its connection by, which then keeps its memory.asyncmethod gets it as its own exception. A customIRingCompletionwhoseCompletesubmits directly, not through an awaiter, would throw out ofCompleteParkedOpsand end the teardown early. On main anyCompletethat throws does the same.ObjectDisposedException.ThrowIf(..., this)matchesHandOff, so the message is the generic one, whereReactorSwitchsays "The reactor has stopped."