Skip to content

nghttp2: the connection's receive window opens to 2^31-1 behind the SETTINGS, so a connection's request bodies in flight are its streams times their window (#612) - #632

Closed
MDA2AV wants to merge 1 commit into
mainfrom
fix-612

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Fixes #612.

What was wrong

RFC 9113 6.9.2: SETTINGS_INITIAL_WINDOW_SIZE sizes stream windows only. The connection's receive window starts at 65,535 and grows only by WINDOW_UPDATE on stream 0, and a sender stays within both windows (6.9.1).

The shim advertised 1000 streams of a 1 MiB window, but ih2_new never raised the connection's window. nghttp2 gave stream 0 back only what had arrived (buffered bodies, by its automatic WINDOW_UPDATE) or what handlers had read (streamed bodies, through ih2_consume). So a connection had at most 64 KiB of request body in flight across all its streams, which bounds uploads by round trips, and a handler that did not read held the whole of it: every other upload on the connection waited until that handler read or returned.

Reproduced at main 19aec5c before any change, with a scratch program over the strict test transport and main's library (sha256 5db0c664...):

[preface] SETTINGS{0x2=0 0x3=1000 0x4=1048576}
[shared] after 65535 B unread on stream 1:
[shared] then 16384 B on stream 3: GOAWAY(error 0x3); stream 3's handler read 0 B
[stalled] flushes in flight: 1; behind it stream 3's handler read 0 of 16384 B

The preface has no WINDOW_UPDATE on stream 0. 16 KiB that fits stream 3's own window still overran the connection's, a GOAWAY with FLOW_CONTROL_ERROR. And the frame loop read nothing while a flush was stalled, which is what made the window part unsafe on its own (below). The Chaos test that holds a handler pinned the stall at exactly 65,535.

What changes

What #593 did for ioxide.http2, here. All in src/protocols/ioxide.nghttp2: nothing in src/ioxide changes.

  • Shim. ih2_new opens the connection's window with nghttp2_session_set_local_window_size(session, NGHTTP2_FLAG_NONE, 0, NGHTTP2_MAX_WINDOW_SIZE) right after nghttp2_submit_settings. nghttp2 sends its queued SETTINGS first, so the preface is SETTINGS and then WINDOW_UPDATE(0, 2,147,418,112), in one write. nghttp2 tops the window up itself once half of it is consumed, in both modes: as DATA arrives, with its automatic WINDOW_UPDATE, and through ih2_consume without it. ih2_server_new takes max_concurrent_streams and initial_window_size for its SETTINGS. ih2_client_new, which nothing calls, keeps its 1000 streams of 1 MiB and gets the open window too.
  • ABI counter. The nghttp2 shim had none to bump, so it gains one the way ngtcp2's works: IH2_ABI 1 with its history line, an ih2_abi() export, and Nghttp2.RequireAbi() in the Nghttp2Connection constructor. Against main's library the fixed binding now throws InvalidOperationException: ioxide.nghttp2: the native library reports ABI 0, this binding needs 1. A stale libioxide_nghttp2.so is being loaded - rebuild it with scripts/build-nghttp2-native.sh., where it would otherwise have read the two new arguments out of registers nobody set.
  • Options. Nghttp2Options.InitialWindowSize (256 KiB) and MaxConcurrentStreams (100), Http2Options' defaults, with docs saying what nghttp2 does with them. nghttp2 already enforced stream windows; nothing changes there.
  • Frame loop. RunCoreAsync awaited each pass's flush before it read again. With the window open, an upload behind a download the peer is not reading fills the transport's receive queue, and the delivery past its 64 entries closes the connection (TcpOptions.RecvQueueEntries). The loop now starts the pass's drain and reads on. What later passes submit stays in nghttp2's queue and leaves on that drain's next turn, through the _drainAgain it already had.
    • What can queue there is bounded by nghttp2 itself: ih2_read fails the session past 5000 frames besides DATA, or past 1000 queued SETTINGS and PING acknowledgements (NGHTTP2_DEFAULT_MAX_OUTBOUND_QUEUE_SIZE, NGHTTP2_DEFAULT_MAX_OBQ_FLOOD_ITEM). That stands where ioxide.http2 has its 1 MiB. DATA is bounded by the peer's windows, and a streamed writer by its 64 KiB unframed.
    • A drain nobody awaits (FlushEgressAsync(detached: true)) reports a transport fault through IsBroken instead of throwing. Any drain that ends with the connection broken cancels the loop's pending read, and the loop stops without feeding what that read returned. A drain that faults off the reactor ends on it, as http2's pump does.
    • A drain fires the body wakes once it is done, so a stream it closed by sending RST_STREAM, nghttp2's own resets among them, ends its reader while the loop is parked on a read. Before, only the loop fired them, after a flush it had waited for.
    • At the end of input, or once the session is done, the loop waits for the drain in flight before the teardown, so what it queued, a GOAWAY among it, goes out first. Not once the connection is broken: nothing more can go out.
    • The SETTINGS flush is still waited for. Until the peer has it, it can send at most 65,535 bytes of DATA, so waiting risks no overflow; the first pass then finds no drain in flight; and a RunAsync begun off the reactor first reads from the reactor, where the TCP pipe completes that flush. tests/http2: "h2 RunAsync begun on the thread pool (http2)" fails now and then, once with FRAME_SIZE_ERROR from the server #614 suspects ioxide.http2 lost that by detaching its own.
  • Docs. Nghttp2BodyReader no longer says a handler that never reads holds the connection window down for every other stream. Nothing in docs/ needed changing: the nghttp2 pages say every byte is credited once, which still holds.

Behaviour a caller would notice

  • The server's preface carries WINDOW_UPDATE(0, 2,147,418,112) behind its SETTINGS, which takes the connection's window from 65,535 to 2^31-1.
  • Defaults move to 100 concurrent streams, from 1000, and a 256 KiB stream window, from 1 MiB: a connection's request bodies in flight are now 100 times 256 KiB, 25 MiB, where they were 64 KiB.
  • A handler that does not read holds back only its own upload.
  • A DATA frame past a stream's window still ends the connection with GOAWAY(FLOW_CONTROL_ERROR): that is nghttp2's answer, and RFC 9113 6.9 allows it as well as a stream error. At main the 65,535 connection window always ran out first, with the same GOAWAY. ioxide.http2 resets just the stream, so the two stacks differ there, as they did.
  • A streamed writer whose flush finds a drain in flight returns at once and its bytes leave with that drain, as they always did behind another stream's drain. With the loop reading on, a pass can now run while its previous pass's drain is in flight, so small chunks flushed in it can share a DATA frame.
  • Added: Nghttp2Options.InitialWindowSize, Nghttp2Options.MaxConcurrentStreams. Removed: none. The internal ih2_server_new takes two more arguments (ABI 1).

Native libraries

runtimes/linux-x64 and runtimes/linux-arm64 are rebuilt with scripts/build-nghttp2-native.sh at its pinned nghttp2 ref (500edc92, 1.70.90) and its default WORK:

  • x64 on the host: GCC 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1), binutils 2.42, cmake 3.28.3.
  • arm64 with RID=linux-arm64 in a throwaway ubuntu:24.04 container: gcc-13-aarch64-linux-gnu 13.3.0-6ubuntu2~24.04.1cross1, binutils-aarch64-linux-gnu 2.42-4ubuntu2.10, cmake 3.28.3-1build7, git 2.43.0.
  • Both toolchains first rebuilt main's shim byte for byte, so the libraries differ from main's by this change alone. A second build of this shim with each is identical to the first.
sha256 main this PR
linux-x64 5db0c664b48293584f36f7e485b7671230303a3a757b98822fd68168912b46e9 7ece43419cb29eab09b6730a1e10e94b0f62423f74bb15caffc167ab15e1c5d1
linux-arm64 68a99fa8cf516d5c063e8c6fe4abfa804a96518117001dff22ad15cd0ad89d45 c241bb01f450577688e5d92b2b33a16cf5bfcdc84f56e08470b53b64dc069041
  • In both, ih2_new, ih2_server_new and ih2_client_new grow and ih2_abi is new. Every other symbol keeps its size and moves with the code behind them.
  • Both still need nothing past libc (and, on arm64, the loader), keep GLIBC 2.17 as their highest version, export main's 15 ih2_ symbols plus ih2_abi, and built without a warning. The shim also compiles clean with -Wall -Wextra.

Tests

New, Nghttp2ReceiveWindowTests (Unit), over the strict test transport:

  • nghttp2 flow: the server's preface is its SETTINGS, then a WINDOW_UPDATE that opens the connection's window to 2^31-1, in the same write
  • nghttp2: by default the server advertises 100 concurrent streams, each with a 256 KiB window, and Nghttp2Options sets both
  • nghttp2 flow: concurrent uploads are each held to their own stream window, and a byte past one ends the connection with FLOW_CONTROL_ERROR. Three handlers that do not read each have exactly one 256 KiB window in flight. Two are let go and take 512 KiB each. A byte past the third's window is nghttp2's GOAWAY(FLOW_CONTROL_ERROR), and that handler's read is cut off.
  • nghttp2 flow: every byte comes back once, on nghttp2's top-up half way down the connection's window, whether it was read, dropped, reset or sent after its answer. With the stream windows at 2^31-1: bodies read, dropped unread when their handler answered, dropped by the peer's reset, and sent after the answer, some padded; then stream 1 in full frames up to 1,073,741,823 bytes. No top-up comes before the last byte, and then exactly one, for every byte charged. About a second of the suite.
  • nghttp2 flow: the frame loop goes on reading while a flush is stalled. Behind a flush the peer does not take, a 32 KiB upload is read, and a PING is answered once the flush is out.
  • nghttp2 flow: a stream nghttp2 resets behind a stalled flush ends its handler's read once that flush's drain sends the reset. A body past its content-length, read behind the stall, and nothing more from the peer.
  • nghttp2 flow: a request read behind a stalled flush as the input ends is answered before the connection is torn down

New, Nghttp2ChaosTests (Chaos), over TCP with the harness's 4 KiB receive buffers, the twin of #593's http2 test:

  • nghttp2: a stream window of upload sent while the server's flush is stalled behind a download the client is not reading arrives whole

Changed, because they counted on the 65,535 connection window:

  • Nghttp2StreamedRequestTests (Chaos):
    • a handler that has not read holds its own upload to one stream window, while another on the connection runs past it, was ...holds its peer to one window... with the stall asserted at 65,535. It now stalls at the 262,144-byte stream window, and meanwhile a 4 MiB upload on stream 3 of the same connection completes.
    • The two a peer RST_STREAM (...) mid-body tests and a handler parked on its body when the connection goes away took the connection's credit as the sign that the handler had read 32 KiB. With the window open there is none to wait for, so they wait for the stream's, at a 65,535-byte stream window.
    • uploads a handler never reads ... crossed the connection window with five 40 KiB uploads. Each now sends 100 KiB past a 65,535-byte stream window.
    • a padded upload ... runs at a 65,535-byte stream window, so padding never credited would still stall it on its own.
    • a body left unread when its stream is reset ... and a handler that fails mid-upload ... no longer name the connection's credit, which they can no longer see. The Unit top-up test counts it.
  • Nghttp2ChaosTests: a streamed body arrives as separate DATA frames waits for the server's SETTINGS ACK before its request. A chunk flushed while another drain is in flight leaves with that drain, and with the loop reading on, the drain of the pass that carried the client's SETTINGS can still be in flight when the request's pass runs: the four chunks then share one frame. It failed 2 of 100 runs alone that way (0 of 100 on main), and 0 of 400 with the wait.
  • OffReactorH2FixTests (E2E): the two nghttp2 uploads run at a 65,535-byte stream window through Nghttp2Options, as their http2 twins already do through Http2Options.
  • StrictClient gains EndInput().

Each part reverted with the rest in place, Unit and Chaos run whole:

Reverted Fails
The shim's nghttp2_session_set_local_window_size (x64 library rebuilt without it) Unit, 3: the preface test, the preface was [type 4 on 0], not SETTINGS and then a WINDOW_UPDATE on stream 0; the concurrent test, unread, the uploads had [65535, 0, 0] bytes in flight, not a whole 262144-byte window each; the top-up test, the connection ended with a GOAWAY 1098839 bytes in. Chaos, 2: the stalled download, the server did not open its connection window; the held handler, the server took 65535 bytes before its handler read any, not one 262144-byte stream window.
Nghttp2Options' defaults back to 1000 and 1 MiB The SETTINGS test, the SETTINGS advertised 1000 streams of 1048576 bytes, not 100 of 262144, and the concurrent test, [524288, 524288, 524288] in flight.
Setup passing 100 and 256 KiB in place of the options The SETTINGS test, the SETTINGS advertised 100 streams of 262144 bytes, not 7 of 100000, and the top-up test, whose 2^31-1 stream window is lost: the connection ended with a GOAWAY 1098839 bytes in.
The loop awaiting each pass's flush Unit: behind the stalled flush the handler read 0 of the 32768 bytes sent after it. Chaos: the download stopped at 33554432 of 33554432 bytes, and the upload's answer never came, the receive queue having overflowed.
The wait for the drain in flight at the end the connection was torn down with an answer still queued behind its flush
CancelPendingRead at the end of a drain that found the connection broken The existing nghttp2: a pass's flush that reports the peer closed ends the connection, without waiting for its read: the connection is still waiting on a read
FireBodyWakes at the end of a drain expected [was cut off], got [still reading]
The SETTINGS flush detached, as a pass's is Chaos nghttp2 streamed response: an endless response and a short one on one connection both get through: 100 of 100 runs alone fail with the short response never ended, and the feed sent 12730174 bytes. 0 of 400 with the await, 0 of 100 on main.
This binding over main's library Every connection: InvalidOperationException: ioxide.nghttp2: the native library reports ABI 0, this binding needs 1. A stale libioxide_nghttp2.so is being loaded - rebuild it with scripts/build-nghttp2-native.sh.

The new tests use the new options, so they do not build against main; the scratch program above is main's side.

Suites

With the fix, pinned to the safe CPUs, on the commit pushed here:

  • Unit: 270 passed, 0 failed, 0 skipped (the 263 on main and the 7 new)
  • Chaos: 106 passed, 0 failed, 0 skipped (the 105 on main and the 1 new)
  • E2E: 730 passed, 0 failed, 0 skipped

CI passed on both runners, the arm64 one with this PR's arm64 library: Unit 270, E2E 723 with 7 skipped, Http 104, Pg 26, Redis 6 with 1 pending, Tls 269, File 13, Chaos 106, none failed.

Only Unit, Chaos and E2E reference ioxide.nghttp2. The two Chaos cases whose outcome depends on timing, a streamed body arrives as separate DATA frames and an endless response and a short one, were also run alone 100 times each on the final code, and the new stalled-download test 40 times: no failures.

h2load against Playground/Http2/Nghttp2Buffered, 2 reactors on CPUs 14-15 and h2load on 16-23, 10 s after 2 s of warm-up, main and this branch alternated twice (the box had other sessions on it):

main this PR
GET, -c 64 -m 16 1,335,142 and 1,323,813 req/s 1,365,090 and 1,323,830 req/s
256 KiB POST, -c 16 -m 8 16,813 and 17,186 req/s 20,692 and 21,743 req/s

GETs are within noise. Uploads are about 25% faster even over loopback, where a round trip costs almost nothing, and the client receives a third of the bytes per request it did, the difference being WINDOW_UPDATEs.

Overlaps

git merge-tree --write-tree against every open PR branch:

For the reviewer

  • The ABI counter is new to this shim. It is one P/Invoke per connection, in the constructor, so a stale library fails where the connection is built and the host's handler logs it.
  • What bounds the frames queued behind a stalled flush is nghttp2's limits, which fail the session (ih2_read returns NGHTTP2_ERR_FLOODED) where ioxide.http2 pauses reading past 1 MiB. A peer that reads its responses never gets near them: per stream it can have the server queue one HEADERS, one WINDOW_UPDATE at a time and one RST_STREAM, besides its acknowledgements. nghttp2_session_get_outbound_queue_size would give a managed pause instead, if that is preferred.
  • A session nghttp2 ends itself, with a GOAWAY for a connection error, while another drain is in flight is noticed by the loop only at its next read, as on main when that drain was a handler's: ih2_is_dead is checked after a pass, and over TCP a cancelled read does not wake a parked one. A peer that then goes silent is held until ReadTimeoutMs, like any silent peer.
  • The end-of-input wait is for the drain in flight, which loops until nothing more is queued, so a streamed writer still writing keeps it going, as it kept the loop's own drain going on main. It is skipped once the connection is broken, and a stalled flush is bounded by SendTimeoutMs.
  • The loop's IsBroken check after a read has no test of its own: what it stops, a pass fed to a connection already known broken, shows nothing a test can see beyond handlers run for no one.

…ETTINGS, so a connection's request bodies in flight are its streams times their window (#612)

SETTINGS size stream windows only (RFC 9113 6.9.2), and the shim never raised the connection's, so it stayed at 65,535: a connection had at most 64 KiB of request body in flight across all its streams, and one handler that did not read held all of it.

ih2_new now opens it with nghttp2_session_set_local_window_size right behind the SETTINGS, so the preface is SETTINGS and then WINDOW_UPDATE(0, 2,147,418,112), and nghttp2 tops it up once half of it is consumed. ih2_server_new takes the SETTINGS' MAX_CONCURRENT_STREAMS and INITIAL_WINDOW_SIZE, which Nghttp2Options now carries with ioxide.http2's defaults: 100 streams of 256 KiB, from 1000 of 1 MiB. The shim gains ih2_abi (ABI 1), and a connection refuses a library of another revision.

RunCoreAsync no longer waits for a pass's flush before it reads again: with the window open, an upload behind a download the peer is not reading would overflow the transport's receive queue and close the connection. What later passes submit leaves with the drain in flight, and nghttp2 bounds what can queue there. A drain nobody awaits reports a fault through IsBroken and ends the loop's read, every drain fires the body wakes of the streams it closed, and at the end of input the loop waits for the drain in flight before the teardown. The SETTINGS flush is still waited for.

Both libioxide_nghttp2.so are rebuilt with scripts/build-nghttp2-native.sh at its pinned ref.

Tests: Nghttp2ReceiveWindowTests (Unit) for the preface, the SETTINGS, the stream windows, the connection's top-up after 1 GiB, and the frame loop behind a stalled flush; a Chaos test for an upload behind a stalled download over TCP. The Chaos tests that counted on the 65,535 connection window now hold uploads to their stream's, and the off-reactor E2E nghttp2 uploads run at 65,535 as their http2 twins do.
@MDA2AV

MDA2AV commented Nov 15, 2026

Copy link
Copy Markdown
Owner Author

Closed: opening nghttp2's connection window needs the frame loop to read on behind a flush, and that let a long streamed response hold back a short one, since nghttp2 sends non-incremental streams strictly in order. The 65,535-byte connection window is conformant and only slower for uploads, so it stays as it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nghttp2: the connection's receive window stays at 65,535, so a connection has at most 64 KiB of request body in flight across its streams

1 participant