Skip to content

ngtcp2: a handshake the TLS stack refuses closes with its alert as CRYPTO_ERROR, not PROTOCOL_VIOLATION (#638) - #650

Open
MDA2AV wants to merge 2 commits into
mainfrom
fix-638
Open

MDA2AV wants to merge 2 commits into
mainfrom
fix-638

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Fixes #638.

Stacked on #646: it targets main so GitHub links and closes the issue, so until #646 merges its diff also shows those commits. Merge in order.

Every engine-side close went through iq_conn_close_liberr, which calls
ngtcp2_ccerr_set_liberr. That maps NGTCP2_ERR_CRYPTO to PROTOCOL_VIOLATION,
as it maps any library error it has no case for, while the alert the TLS
stack raised sits in the conn. RFC 9001 4.8 has a handshake failure sent
as CRYPTO_ERROR, 0x100 + the alert, which is also what ngtcp2's own
examples do. Now, for NGTCP2_ERR_CRYPTO (either sign: the managed side
passes some codes negated) with an alert set, the close takes
ngtcp2_ccerr_set_tls_alert(ngtcp2_conn_get_tls_alert2(conn)); every other
error keeps set_liberr. libioxide_ngtcp2.so rebuilt for linux-x64 and
linux-arm64.

Tests: QuicSniHostileTests' refusals assert the close is a CRYPTO_ERROR,
and decode_error (0x132) for two host names, where the shim chooses the
alert; QuicTransportParameterTests asserts missing_extension (0x16d) for
the #639 hellos and adds no_application_protocol (0x178) for a hello
offering only h2. Hello takes an ALPN. With #639's library all seven of
these fail with "closed with 0xa"; with this one all eleven pass.

Suites (linux-x64): Unit 306/0/0, E2E 800/0/0, Chaos 105/0/0. The arm64 library is built (4-core cross build in ubuntu:24.04, GLIBC_2.38, same 35 iq_ exports), not run here; CI's arm64 leg runs it.

What a caller notices. A peer whose handshake is refused now learns why: h3spec's KeyUpdate (Handshake and 1-RTT), no_application_protocol and EndOfEarlyData cases, and with #646 the two missing_extension ones, close with the alert they expect. No managed API changes.

…th missing_extension instead of aborting the process (#639)

RFC 9001 8.2 makes a ClientHello without the quic_transport_parameters
extension a missing_extension alert. ngtcp2's picotls glue passes the
extension on only when it is there and returns 0 when it is not, so the
handshake went on without the client's transport parameters. A server
learns its negotiated version from them, and ngtcp2 checks that they
arrived only once the handshake completes; the first 1-RTT packet the
server wrote before then failed assert(conn->negotiated_version), which
ngtcp2 keeps in Release builds, and aborted the process.

The shim now wraps the collected_extensions callback that
ngtcp2_crypto_picotls_configure_server_session installs: without the
extension among those collected, which includes a hello carrying only the
draft codepoint 0xffa5, it returns PTLS_ALERT_MISSING_EXTENSION, and the
connection closes. libioxide_ngtcp2.so rebuilt for linux-x64 and
linux-arm64 at the pinned refs.

QuicTransportParameterTests: a hand-built ClientHello without the
extension, and one with only 0xffa5, are refused, and the same server then
answers a well-formed one. With the previous libioxide_ngtcp2.so the first
of them aborted the test host at ngtcp2_conn.c:3633.
…YPTO_ERROR, not PROTOCOL_VIOLATION (#638)

Every engine-side close went through iq_conn_close_liberr, which calls
ngtcp2_ccerr_set_liberr. That maps NGTCP2_ERR_CRYPTO to PROTOCOL_VIOLATION,
as it maps any library error it has no case for, while the alert the TLS
stack raised sits in the conn. RFC 9001 4.8 has a handshake failure sent
as CRYPTO_ERROR, 0x100 + the alert, which is also what ngtcp2's own
examples do. Now, for NGTCP2_ERR_CRYPTO (either sign: the managed side
passes some codes negated) with an alert set, the close takes
ngtcp2_ccerr_set_tls_alert(ngtcp2_conn_get_tls_alert2(conn)); every other
error keeps set_liberr. libioxide_ngtcp2.so rebuilt for linux-x64 and
linux-arm64.

Tests: QuicSniHostileTests' refusals assert the close is a CRYPTO_ERROR,
and decode_error (0x132) for two host names, where the shim chooses the
alert; QuicTransportParameterTests asserts missing_extension (0x16d) for
the #639 hellos and adds no_application_protocol (0x178) for a hello
offering only h2. Hello takes an ALPN. With #639's library all seven of
these fail with "closed with 0xa"; with this one all eleven pass.
@MDA2AV
MDA2AV changed the base branch from fix-639 to main November 17, 2026 19:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ngtcp2: a handshake that fails in TLS closes with PROTOCOL_VIOLATION instead of its alert, so the peer cannot tell why

1 participant