Skip to content

tls: a read from another thread that has to decrypt returns completed once the reactor has stopped, also one the stop found queued, instead of throwing (#585) - #597

Closed
MDA2AV wants to merge 1 commit into
mainfrom
fix-585
Closed

MDA2AV wants to merge 1 commit into
mainfrom
fix-585

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Fixes #585.

Called from another thread with nothing decrypted left to take, TlsDecryptingPipeReader.ReadAsync (BoringSSL's read half) awaits a ReactorSwitch so it can decrypt on the reactor. Once the reactor has stopped, the switch's continuation runs on the thread pool, where GetResult throws ObjectDisposedException. That happens to a read made after the stop, and to one whose switch was still queued when the stop came, since the stop's last drain hands it to the pool. Under kTLS RX, TlsConnectionDualPipe hands out TcpConnectionPipeReader behind TlsProloguePipeReader instead, and the same read returns IsCompleted: true at once, because the stop closed the connection. The handshake picks the half, not the application, so a caller that ends on a completed read got an exception, but only on BoringSSL connections. #580 made the two write halves agree in the same situation.

A read parked on the connection before the stop returned completed on both halves before this too, as the stop's close completes it.

What changes. In FillOnReactor, the cold NoInlining method an off-reactor read goes through:

  • The switch's ObjectDisposedException is caught, and the buffer ends the way a closed snapshot ends it, so the read completes with IsCompleted: true and whatever was decrypted but not consumed. Catching, rather than checking PostsClosed before the switch, also covers the read the stop found queued: it got past any such check while the reactor still ran, as tls: a flush from another thread returns completed once the reactor has stopped, also one the stop found queued, instead of waiting forever (#568) #580's flush did.
  • The ending is shared with Ingest through a one-line EndClosed(), so a receive-queue overflow before the stop still ends the read with the IOException it ends with on the reactor. The kTLS half reports that overflow the same way (TcpConnectionPipeReader.LostToOverflow, checked below). Ending cleanly there would report data the overflow dropped as the peer's clean end.
  • What is queued on the connection is not decrypted. In the default shared-buffer mode those items point into the reactor's receive buffers, and the stop freed them. Ciphertext that arrived before the stop is dropped, and the read reports the end of the stream with TlsSession.Closed still false, as for a peer that closed without close_notify.

No new state, nothing the reactor's path does changes, and no native change.

Why core. TlsDecryptingPipeReader lives in src/ioxide/Tls/Pipes, and the core's TlsConnectionDualPipe constructs it. The fix is in the reader's own off-reactor read, so there is no satellite it could live in. It is 19 lines added and 4 removed in that one file, the class remarks included.

What a caller notices. A read from another thread on a BoringSSL TLS pipe that has to decrypt, made once the reactor has stopped or queued when it stopped, returns a completed ReadResult instead of throwing ObjectDisposedException, or throws IOException if the receive queue overflowed before the stop. Nothing in this repo depends on the old exception. Http2Connection and Nghttp2Connection read their pipe and then switch to the reactor themselves, so off the reactor after a stop they end on their own switch either way, and Kestrel's HopDuplexPipe decrypts in its own pump. The change shows for direct users of TlsConnectionDualPipe that read off the reactor, which its remarks allow ("Usable from any thread").

Tests. In OffReactorFlushTests (Tls), beside the stopped-reactor flush tests:

  1. a read from another thread that has to decrypt returns completed once the reactor has stopped. It replaces the test that asserted the ObjectDisposedException;
  2. control: under kTLS RX the same read returns completed, at once there. This is the answer the fix matches;
  3. a read from another thread still queued when the reactor stops returns completed. The handler holds the reactor, in the pass the client's byte resumed, while another thread reads, and then stops it, so the read's switch is still queued when the loop ends. The test also requires that the read never reached the connection (the _pending the stop's close sets is still set), so a read the reactor ran before its stop, which the stop's close completes too, can't pass for it;
  4. a read from another thread once the reactor has stopped reports a receive-queue overflow before the stop as an IOException, on the existing overflow test's setup (64-byte buffers, a 16-entry queue);
  5. a read from another thread once the reactor has stopped returns completed with nothing decrypted when 1000 bytes of ciphertext were queued before the stop.

On main, 1, 3, 4 and 5 fail with ObjectDisposedException and the control passes. Taking out each part of the fix fails its own test:

Without Fails
the whole fix (main's TlsDecryptingPipeReader.cs) 1 and 5: the read after the stop: ObjectDisposedException; 3: expected [completed], got [ObjectDisposedException]; 4: expected [IOException], got [ObjectDisposedException]
the catch, with EndClosed() kept in Ingest the same four, the same messages
EndClosed() in the catch (a plain End(null)) 4: expected [IOException], got [completed]
leaving the queue alone (Fill() in the catch, decrypting as on the reactor) 5: the read after the stop: completed, with 1000 bytes, decrypted out of the freed receive buffers; 3: expected [completed], got [completed, by a read that reached the connection]

With the fix all pass. Looping the file's tests in a scratch runner, all 50 rounds passed with the fix (750 tests), and on main the four failed in each of 10 rounds. The same runner, with two throwaway tests, showed the kTLS RX answers this matches: an overflow before the stop gives IOException, 3 of 3.

Suites. On the head: Unit 263/0, E2E 730/0, Tls 273/0 (269 before: one test replaced, four added). The tls module is loaded on this box, so the kTLS tests ran and nothing was skipped. Chaos, Http, File, Pg and Redis weren't run, since this change doesn't reach them. No benchmark: the reactor's read path is unchanged, and only FillOnReactor, already an async method, gains an exception handler.

For the reviewer.

  • The dropped ciphertext reads as an end of stream, not an error. Calling it an error would need the reader to look at the connection's queue, or keep state to tell "dropped" from "nothing queued". The stream end with TlsSession.Closed false is what a peer that closes without close_notify gets too.
  • Under kTLS RX the halves still differ when bytes were queued before the stop. There the same read hands out what was queued, as slices into the receive buffers the stop freed: the throwaway test above saw completed at once, with 1000 bytes, 3 of 3, and those bytes can fault once read. The plain connection's reader does the same. That is the teardown's, which frees the shared buffers without draining the connections their handlers still hold, not this reader's, so it is left for its own issue, and this PR doesn't copy it.

… once the reactor has stopped, also one the stop found queued, instead of throwing (#585)

Called off the reactor with nothing decrypted left to take, TlsDecryptingPipeReader.ReadAsync (BoringSSL's read half) switches to the reactor to decrypt. Once the reactor has stopped, that switch resumes on the thread pool and throws ObjectDisposedException, for a read made after the stop and for one whose switch the stop found queued alike. The kTLS RX half, TcpConnectionPipeReader behind TlsProloguePipeReader, returns a completed read there instead, since the stop closed the connection. The handshake picks the half, not the application, so the two should agree.

FillOnReactor now catches the switch's ObjectDisposedException and ends the buffer as a closed snapshot does, through EndClosed(), which Ingest's closed path now shares: the read completes with IsCompleted and whatever was decrypted but not consumed, or, after a receive-queue overflow before the stop, fails with the IOException a read on the reactor and the kTLS half report. What is still queued on the connection is not decrypted: in the default shared-buffer mode it points into the receive buffers the stop freed.

Tests (Tls, OffReactorFlushTests): the test that asserted the ObjectDisposedException now asserts a completed read, and a kTLS RX control gives the same answer. A read whose switch is posted while the handler holds the reactor, in the pass it then stops, returns completed without having reached the connection. A read after the stop reports an overflow before it as an IOException, and returns completed with nothing decrypted when ciphertext was queued before it. On main the four BoringSSL tests fail with ObjectDisposedException and the control passes.
@MDA2AV

MDA2AV commented Nov 15, 2026

Copy link
Copy Markdown
Owner Author

Closed: with the stop draining first (#634, PR #636), a read during the drain behaves as on a live reactor, and once the reactor is torn down every call into it fails fast with ObjectDisposedException, which is the intended answer.

@MDA2AV MDA2AV closed this Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do not merge Kept open for reference; not to be merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tls: after its reactor stops, a read from another thread throws over BoringSSL but returns a closed read under kTLS RX

1 participant