Repository navigation
Conversation
…are out, so a request already in is served on the reactor (#614) A RunAsync begun on the thread pool knows no reactor and learns it from the first read that completes there. Before #593 the loop awaited its SETTINGS flush, which a TCP pipe completes on the reactor, so that first read was made from the reactor. #593 started the preface flush detached, as each pass's flush now is, so the first read was made from the pool, and completed there at once when the client's preface and request were already in: a client need not wait for the server's preface (RFC 9113 3.4). That pass then ran on the pool with no reactor known, so the handler's writes, flushes and body reads ran inline on its own pool thread while the SETTINGS flush resumed on the reactor. Stage, the queue and the pump are reactor-only, so frames were torn: the client's FRAME_SIZE_ERROR and PROTOCOL_ERROR, and echoes that came back with other bytes. A RunAsync begun off the reactor awaits its SETTINGS flush again, as the nghttp2 twin does. Begun on the reactor, the loop still reads on behind that flush. OffReactorH2FixTests: the two "h2 RunAsync begun on the thread pool" tests also check that the connection made no read before its SETTINGS flush completed. With the fix reverted the http2 one fails in 10 of 10 rounds, "the connection read only once its SETTINGS were out (1 reads made before that flush completed)"; the nghttp2 twin passes either way.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #614.
What was wrong. A RunAsync begun on the thread pool knows no reactor:
Reactor.Currentis null there, and the loop learns its reactor from the first read that completes on it. Before #593 the loop awaited its SETTINGS flush, which a TCP pipe completes on the reactor, so its first read was made from the reactor. #593 starts that flush detached, as it does each pass's flush, so the first read was made from the pool. A client need not wait for the server's preface (RFC 9113 3.4): when the client's preface and request were already in, that read completed at once on the pool, and the pass ran there with no reactor known.OffReactorwas then false, so the handler'sWriteHeaders, flushes and body reads ran inline on its own pool thread while the SETTINGS flush resumed on the reactor.Stage, the queue and the pump are reactor-only and share_flushing,_queuedand the pipe writer unlocked, so frames were torn. The issue's hypothesis is right.Reproduced at main 19aec5c, not only read, with a scratch copy of the late-start test that also records where the handler was dispatched (
bench/.work/repro614in the worktree, built and run pinned):What changes.
Http2Connection.RunCoreAsync: begun off the reactor (_reactornull), the loop awaits its SETTINGS flush (FlushAsync), as it did before #593 and asNghttp2Connectiondoes, so its first read is made where that flush completed: over a TCP pipe, the reactor. Begun on the reactor, it still starts that flush and reads on behind it (FlushPassAsync). With the fix, the same scratch: the test as it is, 1,000 of 1,000 rounds dispatched on the reactor on each stack, none failed; the request held, 999 of 1,000 for http2 (the one left is under "What is left").Behaviour a caller would notice. A RunAsync begun off a reactor thread, which includes one over a pipe with no reactor at all, makes its first read only once its SETTINGS flush has completed, as before #593. Over a transport that completes a flush only once its peer reads, that first read waits for it. Begun on the reactor, nothing changes.
Tests. OffReactorH2FixTests: the two "h2 RunAsync begun on the thread pool" tests (http2, nghttp2) also assert that the connection made no read before its SETTINGS flush completed; the test's transport counts the reads made before its first flush is done. With the fix reverted, the http2 one fails in 10 of 10 rounds of the file:
the connection read only once its SETTINGS were out (1 reads made before that flush completed), or 2 when the harness's listen probe read early too. The nghttp2 twin passes either way. With the fix, 80 of 80 over the same 10 rounds. The check is on that order rather than on where the handler was dispatched: a dispatch can still land off the reactor in the window below, so a test on it would be flaky.Suites (Release, pinned to the safe CPUs): Unit 263 passed, 0 failed; E2E 730 passed, 0 failed; Chaos 105 passed, 0 failed; Tls 269 passed, 0 failed.
What is left. Learning the reactor from the thread a completion runs on cannot be made certain from an
IDuplexPipe. When the reactor finishes the SETTINGS send before the pool thread has awaited it, the flush is seen complete on the pool, and a first read that completes at once there runs the pass off the reactor as before. Two ways in showed up: the eventfd write that hands the reactor the flush wakes it, and it can preempt a pool thread on its CPU; and a process's first such connection, most likely because its pool thread is still compiling the await path. The nghttp2 twin has had this window all along. With the request held: http2 with this fix, 1 of 1,000 rounds, the first one; nghttp2 at main, 19 of 1,000, 15 of them with the pool thread on the reactor's CPU. That run had the pool thread there in 58 rounds against 1 in the http2 run, so the rate follows the box's load. With the test's own timing it was 0 of 1,000 on each stack: after a lost race the first read usually still parks, and completes on the reactor. Closing it needs the connection told its reactor before its first pass, for example by theTcpConnectionconstructor taking the connection's reactor (internal to the core today) and anIDuplexPipeoverload taking one, or by a way for a pipe to name its reactor. Each is an API decision and would cover nghttp2 as well, so it is not in this PR.Overlaps. #603 (fix-592) and #604 (fix-588) change
Http2Connection.csinDispatchBuffered,CompleteBufferedAsyncandRetireStream; this changes the start ofRunCoreAsynconly.git merge-tree --write-tree origin/fix-<n> HEADis clean (exit 0) against fix-588, fix-592 and every other open PR branch, fix-581 to fix-592, so either order merges as is. No open PR touchesOffReactorH2FixTests.cs. Of the issues in progress, #615 points atHttp2Connection.csnearRetireStream, away from this change.For the reviewer.
StrictClientnow take it for the preface; all 263 pass unchanged.