Skip to content

Ktor compliance and resilience tests - #1824

Open
bjhham wants to merge 5 commits into
MDA2AV:mainfrom
ktorio:compliance_resilience_tests
Open

bjhham wants to merge 5 commits into
MDA2AV:mainfrom
ktorio:compliance_resilience_tests

Conversation

@bjhham

@bjhham bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Description

Add compliance and resilience to the Ktor frameworks. They both seem pretty ok on my pi though with wildly different arguments, so we'll see how it goes on the big server.

Also, increased the blockable I/O dispatcher threads and reduced worker threads in the tuned one.


PR Commands — comment on this PR to trigger (requires collaborator approval):

Command Description
/benchmark -f <framework> Run every test the framework subscribes to
/benchmark -f <framework> -t <test> Run one test only
/benchmark -f <framework> -t <t1>,<t2>,... Run several tests in one job: one approval, one results commit, untouched tests keep their numbers
/benchmark -f <framework> --save Run and save results (updates the leaderboard on merge)
/benchmark -f <framework> -t <test> --save Run one test and save results
/benchmark -f <framework> --compare <other> Measure the deltas against another framework instead of this one
/benchmark-multiple -f <fw1>,<fw2>,... Benchmark several frameworks in one run — takes -t and --save too; saved results land in a single commit
/benchmark-multiple --save No -f needed: benchmark and save every framework the PR touches
/benchmark-test -t <test> Benchmark all enabled frameworks subscribed to <test> and save the results
/spec -f <framework> Run the protocol-compliance suites the framework lists in spec in its meta.json
/spec -f <framework> --save Run them and save the scores to site/data/spec/ (lands with the merge, like saved results)

For /benchmark, always specify -f <framework>; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory — one table per framework on multi runs. A new benchmark comment while a run is in flight queues behind it (one deep) instead of cancelling it. For multi-framework PRs (dependency bumps, same-language refactors) prefer /benchmark-multiple, which runs everything in a single job and commits all saved results together, so no run overwrites another. --compare works on single-framework runs only.

A /benchmark without -t also runs the compliance suites, and its --save saves their scores with the results. /spec runs the suites alone; without -f it runs them for every framework the PR touches. See Compliance.

What the deltas are measured against. By default, this framework's own results published on main - answering "did this change help?". When you are tuning a variant or a successor entry, --compare re-bases them on another entry instead:

/benchmark -f genhttp --compare genhttp-kestrel

The reply states which baseline it used, and profiles the other framework does not run show n/a rather than a delta.


Run benchmarks locally

You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.

./scripts/validate.sh <framework>
./scripts/benchmark-lite.sh <framework> baseline
./scripts/benchmark-lite.sh --load-threads 4 <framework>

Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk) are built as self-contained Docker images on first run.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Heads up! This PR modifies the following frameworks:

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

/spec -f ktor --save

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

/spec -f ktor-tuned --save

@github-actions

Copy link
Copy Markdown
Contributor

👋 Spec request received. A collaborator will review and approve the run.

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

👋 Spec request received. A collaborator will review and approve the run.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ This run's results were not saved: the push to ktorio/HttpArena was refused, most likely a fork that does not allow edits from maintainers. Nothing was committed, and the board keeps the numbers it has. Save them with a new run.

@github-actions

Copy link
Copy Markdown
Contributor

Spec Results

Framework: ktor

Compliance

Suite Tool Score Failed
H/1.1 (h1) http11probe d50d818 125 / 159 34
H/2 (h2) h2spec 2.6.0 129 / 146 (tls 130/146, h2c 129/146) 17
H/3 (h3) h3spec v0.1.15 58 / 81 23
H/1.1: 34 failed
Test Expected Got
RFC9112-5.1-OBS-FOLD Obs-fold (line folding) in headers should be rejected (RFC 9112 §5.1, MUST) 400 200, connection left open
RFC9112-7.1-MISSING-HOST Request without Host header must be rejected with 400 (RFC 9112 §3.2, MUST) 400 200, connection left open
RFC9112-2.3-INVALID-VERSION Invalid HTTP version must be rejected (RFC 9112 §2.3, SHOULD) 400/505, close, or timeout = warn 200, connection left open
COMP-HOST-WITH-USERINFO Host header with userinfo (user@host) must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-HOST-WITH-PATH Host header with path component must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-ASTERISK-WITH-GET Asterisk-form (*) request-target with GET must be rejected (RFC 9112 §3.2.4, MUST) 400, close, or timeout = warn 404, connection left open
COMP-OPTIONS-STAR OPTIONS * is the only valid asterisk-form request (RFC 9112 §3.2.4, MUST) 2xx or 405; close/timeout = warn 404, connection left open
COMP-UNKNOWN-TE-501 Unknown Transfer-Encoding without CL should be rejected with 501 (RFC 9112 §6.1, MUST) 400/501 or close 200, connection left open
COMP-METHOD-CONNECT CONNECT to an origin server must be rejected (RFC 9110 §9.3.6, SHOULD) 400/405/501 or close 404, connection left open
COMP-HOST-EMPTY-VALUE Empty Host header value must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-REQUEST-LINE-TAB Tab as request-line delimiter — SHOULD reject but MAY parse on whitespace (RFC 9112 §3, SHOULD) 400 or 2xx; close/timeout = warn 404, connection left open
COMP-DATE-HEADER Origin server must include Date header in responses (RFC 9110 §6.6.1, MUST) 2xx with Date header 200, connection left open
COMP-VERSION-CASE HTTP version is case-sensitive — lowercase 'http' must be rejected (RFC 9112 §2.3, MUST) 400, close, or timeout = warn 404, connection left open
SMUG-TE-XCHUNKED Transfer-Encoding: xchunked must not be treated as chunked (RFC 9112 §6.1, MUST) 400/501 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-BARE-SEMICOLON Chunk size with bare semicolon and no extension name must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-INVALID-TOKEN Chunk extension with invalid token character must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-UNDERSCORE Chunk size with underscores (1_0) must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-TE-EMPTY-VALUE Transfer-Encoding with empty value must be rejected (RFC 9112 §6.1, MUST) 400 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-HEX-PREFIX Chunk size with 0x prefix must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SIZE-PLUS Chunk size with leading plus sign must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SIZE-TRAILING-OWS Chunk size with trailing whitespace before CRLF must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-LEADING-SP Chunk size with leading space must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-MISSING-TRAILING-CRLF Chunk data without trailing CRLF must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SPILL Chunk declares size 5 but sends 7 bytes — oversized chunk data must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-CTRL NUL byte in chunk extension must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-CR Bare CR (not CRLF) in chunk extension — some parsers treat CR alone as line ending (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-TE-IDENTITY Transfer-Encoding: identity (deprecated) with CL must be rejected (RFC 9112 §7, MUST) 400/501 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-NEGATIVE Negative chunk size must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-MULTIPLE-HOST-COMMA Host header with comma-separated values must be rejected (RFC 9110 §7.2, MUST) 400 or close 200, connection left open
SMUG-CHUNK-INVALID-SIZE-DESYNC Invalid chunk size (+0) with poison byte — detects chunk-size parser desync (RFC 9112 §7.1, MUST) 400, or close 200, timed out (Accepted invalid chunk-size request and kept connection open — follow-up GET returned no response)
MAL-NUL-IN-URL NUL byte in URL should be rejected (MUST) 400 or close 404, connection left open
MAL-NON-ASCII-URL Non-ASCII bytes (UTF-8 é) in URL must be rejected (MUST) 400 or close 404, connection left open
MAL-CHUNK-SIZE-OVERFLOW Chunk size with integer overflow must be rejected (MUST) 400 or close 200, connection left open
MAL-URL-OVERLONG-UTF8 Overlong UTF-8 encoding of / (0xC0 0xAF) in URL must be rejected (MUST) 400 or close 404, connection left open
H/2: 17 failed
Test Expected Got
generic/4/4 Sends a POST request with trailers HEADERS Frame (stream_id:1) Timeout
http2/3.5/2 Sends invalid connection preface GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed Error: unexpected EOF
http2/5.1/5 half closed (remote): Sends a DATA frame GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/5.1/6 half closed (remote): Sends a HEADERS frame GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/5.1/12 closed: Sends a HEADERS frame GOAWAY Frame (Error Code: STREAM_CLOSED) or Connection closed RST_STREAM Frame (length:4, flags:0x00, stream_id:1)
http2/5.1.1/2 Sends stream identifier that is numerically smaller than previous GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:5)
http2/6.1/2 Sends a DATA frame on the stream that is not in "open" or "half-closed (local)" state GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/6.3/2 Sends a PRIORITY frame with a length other than 5 octets GOAWAY Frame (Error Code: FRAME_SIZE_ERROR) or RST_STREAM Frame (Error Code: FRAME_SIZE_ERROR) or Connection closed Timeout
http2/6.9/2 Sends a WINDOW_UPDATE frame with a flow control window increment of 0 on a stream GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/6.9.1/3 Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR) RST_STREAM Frame (length:4, flags:0x00, stream_id:1)
http2/8.1/1 Sends a second HEADERS frame without the END_STREAM flag GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.1/3 Sends a HEADERS frame that contains a pseudo-header field as trailers GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.3/2 Sends a HEADERS frame that omits ":method" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:1)
http2/8.1.2.3/3 Sends a HEADERS frame that omits ":scheme" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:1)
http2/8.1.2.3/4 Sends a HEADERS frame that omits ":path" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed HEADERS Frame (length:4, flags:0x05, stream_id:1)
http2/8.1.2.6/1 Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.6/2 Sends a HEADERS frame with the "content-length" header field which does not equal the sum of the multiple DATA frames payload length GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
H/3: 23 failed
Test Expected Got
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_source_connection_id is missing [Transport 7.3] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if original_destination_connection_id is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if preferred_address is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if retry_source_connection_id is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if stateless_reset_token is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if a parameter value is malformed [Transport 18] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_udp_payload_size < 1200 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if ack_delay_exponent > 20 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_ack_delay >= 2^14 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_bidi > 2^60 [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_uni > 2^60 [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send FRAME_ENCODING_ERROR if a frame is cut short [Transport 12.4] - predicate failed on expected exception: QUICException / TransportErrorIsReceived ProtocolViolation ""
QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Handshake are non-zero [Transport 17.2] - did not get expected exception: QUICException
QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Short are non-zero [Transport 17.3.1] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with keep-alive as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with proxy-connection as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with transfer-encoding as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with upgrade as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST send H3_FRAME_ERROR if a SETTINGS frame stops mid-parameter [HTTP/3 7.1] - did not get expected exception: QUICException
HTTP/3 servers treats a repeated setting identifier as an error whatever its value [HTTP/3 7.2.4] - did not get expected exception: QUICException
HTTP/3 servers MUST send H3_FRAME_UNEXPECTED if PUSH_PROMISE is received [HTTP/3 7.2.5] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 512) "Decompression of header block failed."
HTTP/3 servers MUST send H3_FRAME_ERROR if MAX_PUSH_ID has octets after its push ID [HTTP/3 7.1] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 261) "Unexpected frame type '0' received"
HTTP/3 servers MUST NOT buffer an over-long frame on a control stream [HTTP/3 7.1] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 262) "Received an invalid frame len 1073741824 for frame of type 7."
Full log
[info] framework: ktor (ktor, Kotlin)
[info] subscribed tests: baseline,async,latency-1m,latency-10k,latency-500k-8cpu,limited-conn,async,pipelined,json-comp,json-tls,8gbit,static-tls,async-db,baseline-h2,static-h2,baseline-h2c,json-h2c,baseline-h3,static-h3,echo-ws,echo-ws-pipeline,fortunes
[info] compliance suites: h1 h2 h3
[info] building image: httparena-ktor
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 750B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/eclipse-temurin:25-jre
#2 ...

#3 [internal] load metadata for docker.io/library/gradle:9.5.1-jdk25-corretto
#3 DONE 0.9s

#2 [internal] load metadata for docker.io/library/eclipse-temurin:25-jre
#2 DONE 1.1s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [build 1/7] FROM docker.io/library/gradle:9.5.1-jdk25-corretto@sha256:f5f9ca1fd04046f506e3fedb24d6f582caecacc6fb58ced2c2a901fffeba752a
#5 resolve docker.io/library/gradle:9.5.1-jdk25-corretto@sha256:f5f9ca1fd04046f506e3fedb24d6f582caecacc6fb58ced2c2a901fffeba752a 0.1s done
#5 DONE 0.1s

#6 [stage-1 1/3] FROM docker.io/library/eclipse-temurin:25-jre@sha256:945247f9d092703e0c6be92243ef3c51640229e23ab34d353c8dcdc73e6cd4e6
#6 resolve docker.io/library/eclipse-temurin:25-jre@sha256:945247f9d092703e0c6be92243ef3c51640229e23ab34d353c8dcdc73e6cd4e6 0.1s done
#6 DONE 0.1s

#7 [stage-1 2/3] WORKDIR /app
#7 CACHED

#8 [internal] load build context
#8 transferring context: 41.78kB done
#8 DONE 0.0s

#9 [build 2/7] WORKDIR /app
#9 CACHED

#10 [build 3/7] COPY build.gradle.kts settings.gradle.kts gradle.properties ./
#10 CACHED

#11 [build 4/7] COPY gradle/libs.versions.toml ./gradle/
#11 CACHED

#12 [build 5/7] RUN gradle dependencies --no-daemon -q 2>/dev/null || true
#12 CACHED

#13 [build 6/7] COPY src ./src
#13 DONE 0.9s

#14 [build 7/7] RUN gradle buildFatJar --no-daemon -q
#14 DONE 30.9s

#15 [stage-1 3/3] COPY --from=build /app/build/libs/ktor-httparena.jar .
#15 DONE 0.1s

#16 exporting to image
#16 exporting layers
#16 exporting layers 1.0s done
#16 exporting manifest sha256:6352e731facc92e134a397b5545d09640e7f02b73ed9a043fd5e8140b2803b47 0.0s done
#16 exporting config sha256:e7b0b1abf8a925d273004af94f283ac25152c08af5ad87b65b892c659971366a 0.0s done
#16 exporting attestation manifest sha256:c1f254ee645404b2a8f0cd580b977b820cfdb8dc418585c6907b0cf50f52f726 0.0s done
#16 exporting manifest list sha256:a70e298ec97734a2f63c5124bfed7e3d054c852c44120eec0c001dca1d41fd40
#16 exporting manifest list sha256:a70e298ec97734a2f63c5124bfed7e3d054c852c44120eec0c001dca1d41fd40 0.0s done
#16 naming to docker.io/library/httparena-ktor:latest done
#16 unpacking to docker.io/library/httparena-ktor:latest
#16 unpacking to docker.io/library/httparena-ktor:latest 0.2s done
#16 DONE 1.4s
[10:15:24] spec h1: ktor
[info] waiting for server...
[info] server ready
httparena-spec-ktor
httparena-spec-ktor
[10:19:17] spec h2: ktor
[info] waiting for server...
[info] server ready
httparena-spec-ktor
httparena-spec-ktor
[10:20:53] spec h3: ktor
[info] waiting for server...
[info] server ready
httparena-spec-ktor
httparena-spec-ktor
[spec] h1  http11probe d50d818: 125/159 (34 failed)
[spec] h2  h2spec 2.6.0: 129/146 (17 failed)
[spec] h3  h3spec v0.1.15: 58/81 (23 failed)
[info] wrote results/spec/ktor.json
[info] saved site/data/spec/ktor.json

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ This run's results were not saved: the push to ktorio/HttpArena was refused, most likely a fork that does not allow edits from maintainers. Nothing was committed, and the board keeps the numbers it has. Save them with a new run.

@github-actions

Copy link
Copy Markdown
Contributor

Spec Results

Framework: ktor-tuned

Compliance

Suite Tool Score Failed
H/1.1 (h1) http11probe d50d818 125 / 159 34
H/2 (h2) h2spec 2.6.0 130 / 146 (tls 130/146, h2c 132/146) 16
H/3 (h3) h3spec v0.1.15 58 / 81 23
H/1.1: 34 failed
Test Expected Got
RFC9112-5.1-OBS-FOLD Obs-fold (line folding) in headers should be rejected (RFC 9112 §5.1, MUST) 400 200, connection left open
RFC9112-7.1-MISSING-HOST Request without Host header must be rejected with 400 (RFC 9112 §3.2, MUST) 400 200, connection left open
RFC9112-2.3-INVALID-VERSION Invalid HTTP version must be rejected (RFC 9112 §2.3, SHOULD) 400/505, close, or timeout = warn 200, connection left open
COMP-HOST-WITH-USERINFO Host header with userinfo (user@host) must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-HOST-WITH-PATH Host header with path component must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-ASTERISK-WITH-GET Asterisk-form (*) request-target with GET must be rejected (RFC 9112 §3.2.4, MUST) 400, close, or timeout = warn 404, connection left open
COMP-OPTIONS-STAR OPTIONS * is the only valid asterisk-form request (RFC 9112 §3.2.4, MUST) 2xx or 405; close/timeout = warn 404, connection left open
COMP-UNKNOWN-TE-501 Unknown Transfer-Encoding without CL should be rejected with 501 (RFC 9112 §6.1, MUST) 400/501 or close 200, connection left open
COMP-METHOD-CONNECT CONNECT to an origin server must be rejected (RFC 9110 §9.3.6, SHOULD) 400/405/501 or close 404, connection left open
COMP-HOST-EMPTY-VALUE Empty Host header value must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-REQUEST-LINE-TAB Tab as request-line delimiter — SHOULD reject but MAY parse on whitespace (RFC 9112 §3, SHOULD) 400 or 2xx; close/timeout = warn 404, connection left open
COMP-DATE-HEADER Origin server must include Date header in responses (RFC 9110 §6.6.1, MUST) 2xx with Date header 200, connection left open
COMP-VERSION-CASE HTTP version is case-sensitive — lowercase 'http' must be rejected (RFC 9112 §2.3, MUST) 400, close, or timeout = warn 404, connection left open
SMUG-TE-XCHUNKED Transfer-Encoding: xchunked must not be treated as chunked (RFC 9112 §6.1, MUST) 400/501 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-BARE-SEMICOLON Chunk size with bare semicolon and no extension name must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-INVALID-TOKEN Chunk extension with invalid token character must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-UNDERSCORE Chunk size with underscores (1_0) must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-TE-EMPTY-VALUE Transfer-Encoding with empty value must be rejected (RFC 9112 §6.1, MUST) 400 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-HEX-PREFIX Chunk size with 0x prefix must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SIZE-PLUS Chunk size with leading plus sign must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SIZE-TRAILING-OWS Chunk size with trailing whitespace before CRLF must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-LEADING-SP Chunk size with leading space must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-MISSING-TRAILING-CRLF Chunk data without trailing CRLF must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SPILL Chunk declares size 5 but sends 7 bytes — oversized chunk data must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-CTRL NUL byte in chunk extension must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-CR Bare CR (not CRLF) in chunk extension — some parsers treat CR alone as line ending (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-TE-IDENTITY Transfer-Encoding: identity (deprecated) with CL must be rejected (RFC 9112 §7, MUST) 400/501 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-NEGATIVE Negative chunk size must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-MULTIPLE-HOST-COMMA Host header with comma-separated values must be rejected (RFC 9110 §7.2, MUST) 400 or close 200, connection left open
SMUG-CHUNK-INVALID-SIZE-DESYNC Invalid chunk size (+0) with poison byte — detects chunk-size parser desync (RFC 9112 §7.1, MUST) 400, or close 200, timed out (Accepted invalid chunk-size request and kept connection open — follow-up GET returned no response)
MAL-NUL-IN-URL NUL byte in URL should be rejected (MUST) 400 or close 404, connection left open
MAL-NON-ASCII-URL Non-ASCII bytes (UTF-8 é) in URL must be rejected (MUST) 400 or close 404, connection left open
MAL-CHUNK-SIZE-OVERFLOW Chunk size with integer overflow must be rejected (MUST) 400 or close 200, connection left open
MAL-URL-OVERLONG-UTF8 Overlong UTF-8 encoding of / (0xC0 0xAF) in URL must be rejected (MUST) 400 or close 404, connection left open
H/2: 16 failed
Test Expected Got
generic/4/4 Sends a POST request with trailers HEADERS Frame (stream_id:1) Timeout
http2/5.1/5 half closed (remote): Sends a DATA frame GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/5.1/6 half closed (remote): Sends a HEADERS frame GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/5.1/12 closed: Sends a HEADERS frame GOAWAY Frame (Error Code: STREAM_CLOSED) or Connection closed RST_STREAM Frame (length:4, flags:0x00, stream_id:1)
http2/5.1.1/2 Sends stream identifier that is numerically smaller than previous GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed RST_STREAM Frame (length:4, flags:0x00, stream_id:3)
http2/6.1/2 Sends a DATA frame on the stream that is not in "open" or "half-closed (local)" state GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/6.3/2 Sends a PRIORITY frame with a length other than 5 octets GOAWAY Frame (Error Code: FRAME_SIZE_ERROR) or RST_STREAM Frame (Error Code: FRAME_SIZE_ERROR) or Connection closed Timeout
http2/6.9/2 Sends a WINDOW_UPDATE frame with a flow control window increment of 0 on a stream GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/6.9.1/3 Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR) RST_STREAM Frame (length:4, flags:0x00, stream_id:1)
http2/8.1/1 Sends a second HEADERS frame without the END_STREAM flag GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.1/3 Sends a HEADERS frame that contains a pseudo-header field as trailers GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.3/2 Sends a HEADERS frame that omits ":method" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:1)
http2/8.1.2.3/3 Sends a HEADERS frame that omits ":scheme" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:1)
http2/8.1.2.3/4 Sends a HEADERS frame that omits ":path" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed HEADERS Frame (length:4, flags:0x05, stream_id:1)
http2/8.1.2.6/1 Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.6/2 Sends a HEADERS frame with the "content-length" header field which does not equal the sum of the multiple DATA frames payload length GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
H/3: 23 failed
Test Expected Got
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_source_connection_id is missing [Transport 7.3] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if original_destination_connection_id is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if preferred_address is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if retry_source_connection_id is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if stateless_reset_token is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if a parameter value is malformed [Transport 18] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_udp_payload_size < 1200 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if ack_delay_exponent > 20 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_ack_delay >= 2^14 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_bidi > 2^60 [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_uni > 2^60 [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send FRAME_ENCODING_ERROR if a frame is cut short [Transport 12.4] - predicate failed on expected exception: QUICException / TransportErrorIsReceived ProtocolViolation ""
QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Handshake are non-zero [Transport 17.2] - did not get expected exception: QUICException
QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Short are non-zero [Transport 17.3.1] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with keep-alive as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with proxy-connection as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with transfer-encoding as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with upgrade as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST send H3_FRAME_ERROR if a SETTINGS frame stops mid-parameter [HTTP/3 7.1] - did not get expected exception: QUICException
HTTP/3 servers treats a repeated setting identifier as an error whatever its value [HTTP/3 7.2.4] - did not get expected exception: QUICException
HTTP/3 servers MUST send H3_FRAME_UNEXPECTED if PUSH_PROMISE is received [HTTP/3 7.2.5] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 512) "Decompression of header block failed."
HTTP/3 servers MUST send H3_FRAME_ERROR if MAX_PUSH_ID has octets after its push ID [HTTP/3 7.1] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 261) "Unexpected frame type '0' received"
HTTP/3 servers MUST NOT buffer an over-long frame on a control stream [HTTP/3 7.1] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 262) "Received an invalid frame len 1073741824 for frame of type 7."
Full log
[info] framework: ktor-tuned (ktor-tuned, Kotlin)
[info] subscribed tests: baseline,async,latency-1m,latency-10k,latency-500k-8cpu,limited-conn,async,pipelined,json-comp,json-tls,8gbit,static-tls,async-db,baseline-h2,static-h2,baseline-h2c,json-h2c,baseline-h3,static-h3,echo-ws,echo-ws-pipeline
[info] compliance suites: h1 h2 h3
[info] building image: httparena-ktor-tuned
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 762B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/gradle:9.5.1-jdk25-corretto
#2 DONE 0.6s

#3 [internal] load metadata for docker.io/library/eclipse-temurin:25-jre
#3 DONE 0.8s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [internal] load build context
#5 DONE 0.0s

#6 [build 1/7] FROM docker.io/library/gradle:9.5.1-jdk25-corretto@sha256:f5f9ca1fd04046f506e3fedb24d6f582caecacc6fb58ced2c2a901fffeba752a
#6 resolve docker.io/library/gradle:9.5.1-jdk25-corretto@sha256:f5f9ca1fd04046f506e3fedb24d6f582caecacc6fb58ced2c2a901fffeba752a 0.1s done
#6 DONE 0.1s

#7 [stage-1 1/3] FROM docker.io/library/eclipse-temurin:25-jre@sha256:945247f9d092703e0c6be92243ef3c51640229e23ab34d353c8dcdc73e6cd4e6
#7 resolve docker.io/library/eclipse-temurin:25-jre@sha256:945247f9d092703e0c6be92243ef3c51640229e23ab34d353c8dcdc73e6cd4e6 0.1s done
#7 DONE 0.1s

#5 [internal] load build context
#5 transferring context: 43.84kB done
#5 DONE 0.0s

#8 [build 2/7] WORKDIR /app
#8 CACHED

#9 [build 3/7] COPY build.gradle.kts settings.gradle.kts gradle.properties ./
#9 CACHED

#10 [build 4/7] COPY gradle/libs.versions.toml ./gradle/
#10 CACHED

#11 [build 5/7] RUN gradle dependencies --no-daemon -q 2>/dev/null || true
#11 CACHED

#12 [build 6/7] COPY src ./src
#12 DONE 0.4s

#13 [build 7/7] RUN gradle buildFatJar --no-daemon -q
#13 DONE 33.1s

#14 [stage-1 2/3] WORKDIR /app
#14 CACHED

#15 [stage-1 3/3] COPY --from=build /app/build/libs/ktor-tuned-httparena.jar .
#15 DONE 0.1s

#16 exporting to image
#16 exporting layers
#16 exporting layers 1.1s done
#16 exporting manifest sha256:1638c1793eb515cde1ec2cae777761b8aa007dd6ca8cf979a2f696ebf2f1d5f5 0.0s done
#16 exporting config sha256:cf810049a7001bc11a7ecd471a04994384b1b0e1feb356c496ce4571ad0fba48 0.0s done
#16 exporting attestation manifest sha256:2e8217b604384adac26995865895f88381b31ecb71e5ab11565c62ba41494d07 0.1s done
#16 exporting manifest list sha256:43cfca1d95e6255788abd024fb4c6344cfc4b39915faec45b7866a5bed9051e3
#16 exporting manifest list sha256:43cfca1d95e6255788abd024fb4c6344cfc4b39915faec45b7866a5bed9051e3 0.0s done
#16 naming to docker.io/library/httparena-ktor-tuned:latest done
#16 unpacking to docker.io/library/httparena-ktor-tuned:latest
#16 unpacking to docker.io/library/httparena-ktor-tuned:latest 0.3s done
#16 DONE 1.5s
[10:27:37] spec h1: ktor-tuned
[info] waiting for server...
[info] server ready
httparena-spec-ktor-tuned
httparena-spec-ktor-tuned
[10:31:29] spec h2: ktor-tuned
[info] waiting for server...
[info] server ready
httparena-spec-ktor-tuned
httparena-spec-ktor-tuned
[10:32:56] spec h3: ktor-tuned
[info] waiting for server...
[info] server ready
httparena-spec-ktor-tuned
httparena-spec-ktor-tuned
[spec] h1  http11probe d50d818: 125/159 (34 failed)
[spec] h2  h2spec 2.6.0: 130/146 (16 failed)
[spec] h3  h3spec v0.1.15: 58/81 (23 failed)
[info] wrote results/spec/ktor-tuned.json
[info] saved site/data/spec/ktor-tuned.json

@MDA2AV

MDA2AV commented Nov 17, 2026

Copy link
Copy Markdown
Owner

/resilience -f ktor

@github-actions

Copy link
Copy Markdown
Contributor

👋 Resilience request received. A collaborator will review and approve the run.

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

/resilience -f ktor

@github-actions

Copy link
Copy Markdown
Contributor

👋 Resilience request received. A collaborator will review and approve the run.

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

If the JVM args prevent the killing then I can include them in the other JVM frameworks in a follow up.

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Looks like the args didn't help... Maybe I'll try -Xmx3g

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

/resilience -f ktor-tuned

@github-actions

Copy link
Copy Markdown
Contributor

👋 Resilience request received. A collaborator will review and approve the run.

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

I'll need to take these resilience tests offline to get them sorted. I'll disable them now for Ktor and see if I can figure out what's going on there.

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

/benchmark -f ktor-tuned

@github-actions

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@MDA2AV

MDA2AV commented Nov 17, 2026

Copy link
Copy Markdown
Owner

@bjhham should I remove the results from here?

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Yes, please. I'll get my hands on a server and investigate the memory issue before committing the results.

Repository owner deleted a comment from github-actions Bot Nov 17, 2026
Repository owner deleted a comment from github-actions Bot Nov 17, 2026
Repository owner deleted a comment from github-actions Bot Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: ktor-tuned | Test: all tests

Test Conn RPS Rate CPU/req p99 p99.9 CPU Mem Δ RPS Δ Mem
baseline 4096 1,517,695 — — — — 6466.9% 57.0GiB +0.7% +866.1%
pipelined 4096 2,571,379 — — — — 6199.5% 73.0GiB -3.9% +432.8%
limited-conn 4096 690,917 — — — — 5086.1% 5.4GiB +14.6% +10.2%
json-comp 4096 252,600 — — — — 6317.6% 9.9GiB +1.2% +80.0%
json-comp 16384 243,457 — — — — 6177.4% 21.9GiB ~0% +130.5%
json-tls 4096 870,591 — — — — 6197.6% 13.3GiB -2.1% +107.8%
8gbit 512 49,283 0.9857 74.03us 175.0us 8162.0us 340.6% 2.6GiB -0.2% +62.5%
static-tls 1024 264,496 — — — — 6293.3% 9.4GiB -0.1% +154.1%
async-db 1024 46,778 — — — — 2012.9% 2.6GiB -8.6% -7.1%
baseline-h2 256 1,003,101 — — — — 6309.9% 94.5GiB -10.6% +668.3%
baseline-h2 1024 937,376 — — — — 6003.1% 121.0GiB -11.3% +361.8%
static-h2 256 238,913 — — — — 6133.3% 16.0GiB +0.5% +105.1%
static-h2 1024 193,068 — — — — 5660.3% 23.4GiB -11.7% +91.8%
baseline-h2c 256 1,047,455 — — — — 6330.1% 94.4GiB -10.7% +584.1%
baseline-h2c 1024 958,379 — — — — 6162.2% 121.3GiB -13.9% +372.0%
baseline-h2c 4096 468,490 — — — — 5084.8% 126.3GiB -29.5% +303.5%
json-h2c 1024 605,625 — — — — 6278.7% 105.8GiB -17.0% +804.3%
json-h2c 4096 485,003 — — — — 5799.9% 123.7GiB -29.1% +372.1%
baseline-h3 64 379,884 — — — — 4265.1% 6.2GiB +0.4% +26.5%
static-h3 64 134,896 — — — — 5542.6% 6.7GiB +65.0% +116.1%
echo-ws 512 1,351,472 — — — — 6417.6% 2.5GiB +24.3% -7.4%
echo-ws 4096 1,358,649 — — — — 6688.1% 10.5GiB +15.1% +288.9%
echo-ws 16384 945,258 — — — — 6281.8% 9.8GiB +9.6% +157.9%
echo-ws-pipeline 512 15,450,260 — — — — 6508.8% 17.5GiB +31.1% +264.6%
echo-ws-pipeline 4096 14,660,754 — — — — 6614.9% 26.8GiB +12.9% +482.6%
echo-ws-pipeline 16384 12,364,008 — — — — 6483.1% 76.9GiB +4.0% +580.5%
latency-1m 1024 997,673 0.9977 46.45us 1430.0us 6858.0us 5030.5% 12.4GiB ~0% +134.0%
latency-10k 1024 9,982 0.9982 48.87us 141.0us 2747.0us 46.7% 599MiB ~0% -61.0%
latency-500k-8cpu 1024 265,410 0.5308 ⚠️ 30.02us 9244672.0us 9334784.0us 778.4% 5.1GiB -4.1% +112.5%
async 32000 1,136,668 — — — — 6213.7% 65.4GiB -10.1% +445.0%

Composite score, main against this run, in the board's default view:

Family main this run Δ
HTTP/1.1 4052.8 (37 of 109) 4086.7 (33 of 109) +33.9
HTTP/2 739.3 (24 of 41) 631.5 (26 of 41) -107.8
HTTP/3 357.3 (13 of 20) 541.2 (11 of 20) +183.9
WebSocket 485.3 (31 of 39) 564.7 (31 of 39) +79.4

The results on main were measured in another session, so a few points up or down can be the machine and not the change.

Compliance

Suite Tool Score Failed
H/1.1 (h1) http11probe d50d818 125 / 159 34
H/2 (h2) h2spec 2.6.0 132 / 146 (tls 132/146, h2c 132/146) 14
H/3 (h3) h3spec v0.1.15 58 / 81 23
H/1.1: 34 failed
Test Expected Got
RFC9112-5.1-OBS-FOLD Obs-fold (line folding) in headers should be rejected (RFC 9112 §5.1, MUST) 400 200, connection left open
RFC9112-7.1-MISSING-HOST Request without Host header must be rejected with 400 (RFC 9112 §3.2, MUST) 400 200, connection left open
RFC9112-2.3-INVALID-VERSION Invalid HTTP version must be rejected (RFC 9112 §2.3, SHOULD) 400/505, close, or timeout = warn 200, connection left open
COMP-HOST-WITH-USERINFO Host header with userinfo (user@host) must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-HOST-WITH-PATH Host header with path component must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-ASTERISK-WITH-GET Asterisk-form (*) request-target with GET must be rejected (RFC 9112 §3.2.4, MUST) 400, close, or timeout = warn 404, connection left open
COMP-OPTIONS-STAR OPTIONS * is the only valid asterisk-form request (RFC 9112 §3.2.4, MUST) 2xx or 405; close/timeout = warn 404, connection left open
COMP-UNKNOWN-TE-501 Unknown Transfer-Encoding without CL should be rejected with 501 (RFC 9112 §6.1, MUST) 400/501 or close 200, connection left open
COMP-METHOD-CONNECT CONNECT to an origin server must be rejected (RFC 9110 §9.3.6, SHOULD) 400/405/501 or close 404, connection left open
COMP-HOST-EMPTY-VALUE Empty Host header value must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-REQUEST-LINE-TAB Tab as request-line delimiter — SHOULD reject but MAY parse on whitespace (RFC 9112 §3, SHOULD) 400 or 2xx; close/timeout = warn 404, connection left open
COMP-DATE-HEADER Origin server must include Date header in responses (RFC 9110 §6.6.1, MUST) 2xx with Date header 200, connection left open
COMP-VERSION-CASE HTTP version is case-sensitive — lowercase 'http' must be rejected (RFC 9112 §2.3, MUST) 400, close, or timeout = warn 404, connection left open
SMUG-TE-XCHUNKED Transfer-Encoding: xchunked must not be treated as chunked (RFC 9112 §6.1, MUST) 400/501 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-BARE-SEMICOLON Chunk size with bare semicolon and no extension name must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-INVALID-TOKEN Chunk extension with invalid token character must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-UNDERSCORE Chunk size with underscores (1_0) must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-TE-EMPTY-VALUE Transfer-Encoding with empty value must be rejected (RFC 9112 §6.1, MUST) 400 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-HEX-PREFIX Chunk size with 0x prefix must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SIZE-PLUS Chunk size with leading plus sign must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SIZE-TRAILING-OWS Chunk size with trailing whitespace before CRLF must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-LEADING-SP Chunk size with leading space must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-MISSING-TRAILING-CRLF Chunk data without trailing CRLF must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SPILL Chunk declares size 5 but sends 7 bytes — oversized chunk data must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-CTRL NUL byte in chunk extension must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-CR Bare CR (not CRLF) in chunk extension — some parsers treat CR alone as line ending (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-TE-IDENTITY Transfer-Encoding: identity (deprecated) with CL must be rejected (RFC 9112 §7, MUST) 400/501 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-NEGATIVE Negative chunk size must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-MULTIPLE-HOST-COMMA Host header with comma-separated values must be rejected (RFC 9110 §7.2, MUST) 400 or close 200, connection left open
SMUG-CHUNK-INVALID-SIZE-DESYNC Invalid chunk size (+0) with poison byte — detects chunk-size parser desync (RFC 9112 §7.1, MUST) 400, or close 200, timed out (Accepted invalid chunk-size request and kept connection open — follow-up GET returned no response)
MAL-NUL-IN-URL NUL byte in URL should be rejected (MUST) 400 or close 404, connection left open
MAL-NON-ASCII-URL Non-ASCII bytes (UTF-8 é) in URL must be rejected (MUST) 400 or close 404, connection left open
MAL-CHUNK-SIZE-OVERFLOW Chunk size with integer overflow must be rejected (MUST) 400 or close 200, connection left open
MAL-URL-OVERLONG-UTF8 Overlong UTF-8 encoding of / (0xC0 0xAF) in URL must be rejected (MUST) 400 or close 404, connection left open
H/2: 14 failed
Test Expected Got
generic/4/4 Sends a POST request with trailers HEADERS Frame (stream_id:1) Timeout
http2/5.1/5 half closed (remote): Sends a DATA frame GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/5.1/12 closed: Sends a HEADERS frame GOAWAY Frame (Error Code: STREAM_CLOSED) or Connection closed RST_STREAM Frame (length:4, flags:0x00, stream_id:1)
http2/5.1.1/2 Sends stream identifier that is numerically smaller than previous GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:5)
http2/6.1/2 Sends a DATA frame on the stream that is not in "open" or "half-closed (local)" state GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/6.9/2 Sends a WINDOW_UPDATE frame with a flow control window increment of 0 on a stream GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/6.9.1/3 Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR) RST_STREAM Frame (length:4, flags:0x00, stream_id:1)
http2/8.1/1 Sends a second HEADERS frame without the END_STREAM flag GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.1/3 Sends a HEADERS frame that contains a pseudo-header field as trailers GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.3/2 Sends a HEADERS frame that omits ":method" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:1)
http2/8.1.2.3/3 Sends a HEADERS frame that omits ":scheme" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:1)
http2/8.1.2.3/4 Sends a HEADERS frame that omits ":path" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed HEADERS Frame (length:4, flags:0x05, stream_id:1)
http2/8.1.2.6/1 Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.6/2 Sends a HEADERS frame with the "content-length" header field which does not equal the sum of the multiple DATA frames payload length GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
H/3: 23 failed
Test Expected Got
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_source_connection_id is missing [Transport 7.3] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if original_destination_connection_id is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if preferred_address is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if retry_source_connection_id is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if stateless_reset_token is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if a parameter value is malformed [Transport 18] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_udp_payload_size < 1200 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if ack_delay_exponent > 20 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_ack_delay >= 2^14 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_bidi > 2^60 [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_uni > 2^60 [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send FRAME_ENCODING_ERROR if a frame is cut short [Transport 12.4] - predicate failed on expected exception: QUICException / TransportErrorIsReceived ProtocolViolation ""
QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Handshake are non-zero [Transport 17.2] - did not get expected exception: QUICException
QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Short are non-zero [Transport 17.3.1] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with keep-alive as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with proxy-connection as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with transfer-encoding as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with upgrade as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST send H3_FRAME_ERROR if a SETTINGS frame stops mid-parameter [HTTP/3 7.1] - did not get expected exception: QUICException
HTTP/3 servers treats a repeated setting identifier as an error whatever its value [HTTP/3 7.2.4] - did not get expected exception: QUICException
HTTP/3 servers MUST send H3_FRAME_UNEXPECTED if PUSH_PROMISE is received [HTTP/3 7.2.5] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 512) "Decompression of header block failed."
HTTP/3 servers MUST send H3_FRAME_ERROR if MAX_PUSH_ID has octets after its push ID [HTTP/3 7.1] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 261) "Unexpected frame type '0' received"
HTTP/3 servers MUST NOT buffer an over-long frame on a control stream [HTTP/3 7.1] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 262) "Received an invalid frame len 1073741824 for frame of type 7."

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Looks like reducing the worker threads causes some of the benchmarks to decline and the static h3 one must have been running out of IO threads. I'll have some optimizations for the h2 pipeline in an upcoming release and I'll get back to ya on these resilience tests.

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

/benchmark-multiple --save

@github-actions

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@MDA2AV

MDA2AV commented Nov 17, 2026

Copy link
Copy Markdown
Owner

Looks like reducing the worker threads causes some of the benchmarks to decline and the static h3 one must have been running out of IO threads. I'll have some optimizations for the h2 pipeline in an upcoming release and I'll get back to ya on these resilience tests.

there seems to be an abnormal memory usage in the latest run too 120GB+ in some tests

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

there seems to be an abnormal memory usage in the latest run too 120GB+ in some tests

It's likely just the VM arg to increase it from 1/4 of the host's memory to 1/2, then the garbage collector just lets the garbage pile up.

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

If you want to monitor the actual memory consumption for JVM applications, there's some extra tooling that tracks the "old generation" of allocated memory for things that linger on past GC runs.

@MDA2AV

MDA2AV commented Nov 17, 2026

Copy link
Copy Markdown
Owner

It's likely just the VM arg to increase it from 1/4 of the host's memory to 1/2, then the garbage collector just lets the garbage pile up.

Isnt there a way to just let it be the minimum between like 10GB and 1/2 of the available? For me this is fine but the memory may be misleading to people visiting the site

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Isnt there a way to just let it be the minimum between like 10GB and 1/2 of the available? For me this is fine but the memory may be misleading to people visiting the site

There's no ootb args for this unfortunately. Without measuring old gen it'll just occupy whatever space it's given and appear wasteful in the reports.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ This run's results were not saved: the push to ktorio/HttpArena was refused, most likely a fork that does not allow edits from maintainers. Nothing was committed, and the board keeps the numbers it has. Save them with a new run.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Results

Frameworks: 2 | Test: all tests

✅ ktor

Test Conn RPS Rate CPU/req p99 p99.9 CPU Mem Δ RPS Δ Mem
baseline 4096 975,505 — — — — 5650.1% 13.3GiB -7.7% +155.8%
pipelined 4096 2,145,020 — — — — 5369.0% 74.0GiB -3.6% +781.0%
limited-conn 4096 805,555 — — — — 5486.1% 22.9GiB -1.5% +131.3%
json-comp 4096 215,698 — — — — 6337.7% 10.9GiB ~0% +14.7%
json-comp 16384 171,972 — — — — 5842.4% 75.7GiB -18.6% +582.0%
json-tls 4096 514,171 — — — — 4819.2% 11.4GiB +6.8% +107.3%
8gbit 512 49,353 0.9871 96.02us 193.0us 7142.0us 456.6% 2.5GiB ~0% +66.7%
static-tls 1024 256,796 — — — — 6194.1% 4.8GiB -3.2% +23.1%
async-db 1024 52,342 — — — — 2374.4% 5.3GiB +11.9% +96.3%
fortunes 1024 58,940 — — — — 5153.4% 23.0GiB -4.6% +115.0%
baseline-h2 256 776,122 — — — — 6154.3% 82.1GiB -5.8% +474.1%
baseline-h2 1024 749,725 — — — — 6038.9% 117.0GiB +2.1% +375.6%
static-h2 256 222,008 — — — — 6070.3% 16.6GiB -1.3% +176.7%
static-h2 1024 142,862 — — — — 5635.6% 94.8GiB -30.8% +549.3%
baseline-h2c 256 809,735 — — — — 6225.3% 75.7GiB -5.3% +429.4%
baseline-h2c 1024 767,545 — — — — 5880.2% 117.9GiB +2.2% +367.9%
baseline-h2c 4096 417,222 — — — — 5269.4% 127.2GiB -8.3% +313.0%
json-h2c 1024 463,871 — — — — 5559.6% 91.2GiB +3.6% +629.6%
json-h2c 4096 322,811 — — — — 4974.6% 123.9GiB -19.5% +369.3%
baseline-h3 64 330,233 — — — — 3991.1% 5.8GiB -2.3% +18.4%
static-h3 64 132,075 — — — — 6030.0% 6.1GiB +35.0% +27.1%
echo-ws 512 1,029,165 — — — — 3392.3% 2.5GiB -3.1% +56.2%
echo-ws 4096 963,948 — — — — 3425.3% 5.2GiB -1.3% +147.6%
echo-ws 16384 753,029 — — — — 3361.3% 9.3GiB +12.4% +126.8%
echo-ws-pipeline 512 10,829,316 — — — — 3333.2% 5.0GiB -6.2% -43.8%
echo-ws-pipeline 4096 9,720,275 — — — — 3550.8% 15.4GiB -12.1% +366.7%
echo-ws-pipeline 16384 7,107,081 — — — — 3595.1% 62.5GiB -21.8% +453.1%
latency-1m 1024 997,449 0.9974 51.30us 3115.0us 17352.0us 5614.7% 11.7GiB ~0% +125.0%
latency-10k 1024 9,981 0.9981 69.42us 156.0us 3113.0us 66.9% 592MiB ~0% +23.1%
latency-500k-8cpu 1024 221,373 0.4427 ⚠️ 31.08us 10981376.0us 11104256.0us 676.5% 2.3GiB -1.8% +76.9%
async 32000 850,051 — — — — 5424.7% 24.6GiB -7.6% +176.4%

Composite score, main against this run, in the board's default view:

Family main this run Δ
HTTP/1.1 3264.7 (44 of 109) 3263.7 (44 of 109) -1.1
HTTP/2 548.8 (30 of 41) 496.4 (30 of 41) -52.4
HTTP/3 404.8 (12 of 20) 521.6 (12 of 20) +116.8
WebSocket 419.8 (33 of 39) 398.2 (34 of 39) -21.6

The results on main were measured in another session, so a few points up or down can be the machine and not the change.

Compliance

Suite Tool Score Failed
H/1.1 (h1) http11probe d50d818 125 / 159 34
H/2 (h2) h2spec 2.6.0 129 / 146 (tls 130/146, h2c 129/146) 17
H/3 (h3) h3spec v0.1.15 58 / 81 23
H/1.1: 34 failed
Test Expected Got
RFC9112-5.1-OBS-FOLD Obs-fold (line folding) in headers should be rejected (RFC 9112 §5.1, MUST) 400 200, connection left open
RFC9112-7.1-MISSING-HOST Request without Host header must be rejected with 400 (RFC 9112 §3.2, MUST) 400 200, connection left open
RFC9112-2.3-INVALID-VERSION Invalid HTTP version must be rejected (RFC 9112 §2.3, SHOULD) 400/505, close, or timeout = warn 200, connection left open
COMP-HOST-WITH-USERINFO Host header with userinfo (user@host) must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-HOST-WITH-PATH Host header with path component must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-ASTERISK-WITH-GET Asterisk-form (*) request-target with GET must be rejected (RFC 9112 §3.2.4, MUST) 400, close, or timeout = warn 404, connection left open
COMP-OPTIONS-STAR OPTIONS * is the only valid asterisk-form request (RFC 9112 §3.2.4, MUST) 2xx or 405; close/timeout = warn 404, connection left open
COMP-UNKNOWN-TE-501 Unknown Transfer-Encoding without CL should be rejected with 501 (RFC 9112 §6.1, MUST) 400/501 or close 200, connection left open
COMP-METHOD-CONNECT CONNECT to an origin server must be rejected (RFC 9110 §9.3.6, SHOULD) 400/405/501 or close 404, connection left open
COMP-HOST-EMPTY-VALUE Empty Host header value must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-REQUEST-LINE-TAB Tab as request-line delimiter — SHOULD reject but MAY parse on whitespace (RFC 9112 §3, SHOULD) 400 or 2xx; close/timeout = warn 404, connection left open
COMP-DATE-HEADER Origin server must include Date header in responses (RFC 9110 §6.6.1, MUST) 2xx with Date header 200, connection left open
COMP-VERSION-CASE HTTP version is case-sensitive — lowercase 'http' must be rejected (RFC 9112 §2.3, MUST) 400, close, or timeout = warn 404, connection left open
SMUG-TE-XCHUNKED Transfer-Encoding: xchunked must not be treated as chunked (RFC 9112 §6.1, MUST) 400/501 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-BARE-SEMICOLON Chunk size with bare semicolon and no extension name must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-INVALID-TOKEN Chunk extension with invalid token character must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-UNDERSCORE Chunk size with underscores (1_0) must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-TE-EMPTY-VALUE Transfer-Encoding with empty value must be rejected (RFC 9112 §6.1, MUST) 400 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-HEX-PREFIX Chunk size with 0x prefix must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SIZE-PLUS Chunk size with leading plus sign must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SIZE-TRAILING-OWS Chunk size with trailing whitespace before CRLF must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-LEADING-SP Chunk size with leading space must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-MISSING-TRAILING-CRLF Chunk data without trailing CRLF must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SPILL Chunk declares size 5 but sends 7 bytes — oversized chunk data must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-CTRL NUL byte in chunk extension must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-CR Bare CR (not CRLF) in chunk extension — some parsers treat CR alone as line ending (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-TE-IDENTITY Transfer-Encoding: identity (deprecated) with CL must be rejected (RFC 9112 §7, MUST) 400/501 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-NEGATIVE Negative chunk size must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-MULTIPLE-HOST-COMMA Host header with comma-separated values must be rejected (RFC 9110 §7.2, MUST) 400 or close 200, connection left open
SMUG-CHUNK-INVALID-SIZE-DESYNC Invalid chunk size (+0) with poison byte — detects chunk-size parser desync (RFC 9112 §7.1, MUST) 400, or close 200, timed out (Accepted invalid chunk-size request and kept connection open — follow-up GET returned no response)
MAL-NUL-IN-URL NUL byte in URL should be rejected (MUST) 400 or close 404, connection left open
MAL-NON-ASCII-URL Non-ASCII bytes (UTF-8 é) in URL must be rejected (MUST) 400 or close 404, connection left open
MAL-CHUNK-SIZE-OVERFLOW Chunk size with integer overflow must be rejected (MUST) 400 or close 200, connection left open
MAL-URL-OVERLONG-UTF8 Overlong UTF-8 encoding of / (0xC0 0xAF) in URL must be rejected (MUST) 400 or close 404, connection left open
H/2: 17 failed
Test Expected Got
generic/4/4 Sends a POST request with trailers HEADERS Frame (stream_id:1) Timeout
http2/3.5/2 Sends invalid connection preface GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed Error: unexpected EOF
http2/5.1/5 half closed (remote): Sends a DATA frame GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/5.1/6 half closed (remote): Sends a HEADERS frame GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/5.1/12 closed: Sends a HEADERS frame GOAWAY Frame (Error Code: STREAM_CLOSED) or Connection closed RST_STREAM Frame (length:4, flags:0x00, stream_id:1)
http2/5.1.1/2 Sends stream identifier that is numerically smaller than previous GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:5)
http2/6.1/2 Sends a DATA frame on the stream that is not in "open" or "half-closed (local)" state GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/6.3/2 Sends a PRIORITY frame with a length other than 5 octets GOAWAY Frame (Error Code: FRAME_SIZE_ERROR) or RST_STREAM Frame (Error Code: FRAME_SIZE_ERROR) or Connection closed Timeout
http2/6.9/2 Sends a WINDOW_UPDATE frame with a flow control window increment of 0 on a stream GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/6.9.1/3 Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR) RST_STREAM Frame (length:4, flags:0x00, stream_id:1)
http2/8.1/1 Sends a second HEADERS frame without the END_STREAM flag GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.1/3 Sends a HEADERS frame that contains a pseudo-header field as trailers GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.3/2 Sends a HEADERS frame that omits ":method" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:1)
http2/8.1.2.3/3 Sends a HEADERS frame that omits ":scheme" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:1)
http2/8.1.2.3/4 Sends a HEADERS frame that omits ":path" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed HEADERS Frame (length:4, flags:0x05, stream_id:1)
http2/8.1.2.6/1 Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.6/2 Sends a HEADERS frame with the "content-length" header field which does not equal the sum of the multiple DATA frames payload length GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
H/3: 23 failed
Test Expected Got
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_source_connection_id is missing [Transport 7.3] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if original_destination_connection_id is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if preferred_address is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if retry_source_connection_id is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if stateless_reset_token is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if a parameter value is malformed [Transport 18] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_udp_payload_size < 1200 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if ack_delay_exponent > 20 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_ack_delay >= 2^14 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_bidi > 2^60 [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_uni > 2^60 [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send FRAME_ENCODING_ERROR if a frame is cut short [Transport 12.4] - predicate failed on expected exception: QUICException / TransportErrorIsReceived ProtocolViolation ""
QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Handshake are non-zero [Transport 17.2] - did not get expected exception: QUICException
QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Short are non-zero [Transport 17.3.1] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with keep-alive as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with proxy-connection as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with transfer-encoding as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with upgrade as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST send H3_FRAME_ERROR if a SETTINGS frame stops mid-parameter [HTTP/3 7.1] - did not get expected exception: QUICException
HTTP/3 servers treats a repeated setting identifier as an error whatever its value [HTTP/3 7.2.4] - did not get expected exception: QUICException
HTTP/3 servers MUST send H3_FRAME_UNEXPECTED if PUSH_PROMISE is received [HTTP/3 7.2.5] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 512) "Decompression of header block failed."
HTTP/3 servers MUST send H3_FRAME_ERROR if MAX_PUSH_ID has octets after its push ID [HTTP/3 7.1] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 261) "Unexpected frame type '0' received"
HTTP/3 servers MUST NOT buffer an over-long frame on a control stream [HTTP/3 7.1] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 262) "Received an invalid frame len 1073741824 for frame of type 7."

✅ ktor-tuned

Test Conn RPS Rate CPU/req p99 p99.9 CPU Mem Δ RPS Δ Mem
baseline 4096 1,522,542 — — — — 6298.8% 24.4GiB +1.1% +313.6%
pipelined 4096 2,303,001 — — — — 6332.1% 69.1GiB -14.0% +404.4%
limited-conn 4096 700,363 — — — — 5039.8% 5.2GiB +16.2% +6.1%
json-comp 4096 257,072 — — — — 6487.7% 10.4GiB +3.0% +89.1%
json-comp 16384 244,300 — — — — 6108.3% 24.2GiB +0.3% +154.7%
json-tls 4096 876,409 — — — — 6244.0% 13.1GiB -1.4% +104.7%
8gbit 512 49,418 0.9884 74.40us 175.0us 4146.0us 355.0% 2.5GiB ~0% +56.2%
static-tls 1024 251,821 — — — — 6391.0% 9.3GiB -4.9% +151.4%
async-db 1024 45,582 — — — — 1978.9% 2.6GiB -11.0% -7.1%
baseline-h2 256 1,016,779 — — — — 6237.3% 105.7GiB -9.4% +759.3%
baseline-h2 1024 942,509 — — — — 6053.1% 121.1GiB -10.8% +362.2%
static-h2 256 240,300 — — — — 6158.0% 17.4GiB +1.1% +123.1%
static-h2 1024 155,594 — — — — 6161.0% 18.1GiB -28.8% +48.4%
baseline-h2c 256 1,043,320 — — — — 6360.6% 94.8GiB -11.0% +587.0%
baseline-h2c 1024 1,000,256 — — — — 6282.2% 117.8GiB -10.1% +358.4%
baseline-h2c 4096 466,502 — — — — 5140.5% 126.2GiB -29.8% +303.2%
json-h2c 1024 674,042 — — — — 6385.4% 95.5GiB -7.6% +716.2%
json-h2c 4096 543,178 — — — — 5887.6% 123.8GiB -20.6% +372.5%
baseline-h3 64 383,724 — — — — 4267.1% 6.6GiB +1.4% +34.7%
static-h3 64 137,218 — — — — 5849.4% 5.3GiB +67.9% +71.0%
echo-ws 512 1,366,943 — — — — 6399.0% 2.5GiB +25.8% -7.4%
echo-ws 4096 1,389,371 — — — — 6702.0% 10.9GiB +17.7% +303.7%
echo-ws 16384 1,063,467 — — — — 6662.9% 13.2GiB +23.3% +247.4%
echo-ws-pipeline 512 15,247,799 — — — — 6563.2% 9.3GiB +29.4% +93.8%
echo-ws-pipeline 4096 14,631,527 — — — — 6620.1% 11.6GiB +12.7% +152.2%
echo-ws-pipeline 16384 12,215,964 — — — — 6440.5% 60.6GiB +2.8% +436.3%
latency-1m 1024 997,375 0.9974 47.17us 1658.0us 12732.0us 5053.2% 12.5GiB ~0% +135.8%
latency-10k 1024 9,981 0.9981 47.53us 137.0us 2753.0us 46.0% 609MiB ~0% -60.4%
latency-500k-8cpu 1024 266,492 0.5330 ⚠️ 29.91us 9203712.0us 9293824.0us 781.5% 2.3GiB -3.7% -4.2%
async 32000 1,154,794 — — — — 6234.9% 65.8GiB -8.6% +448.3%

Composite score, main against this run, in the board's default view:

Family main this run Δ
HTTP/1.1 4052.8 (37 of 109) 4109.3 (31 of 109) +56.5
HTTP/2 739.3 (24 of 41) 641.1 (26 of 41) -98.2
HTTP/3 357.3 (13 of 20) 550.0 (11 of 20) +192.7
WebSocket 485.3 (31 of 39) 576.1 (31 of 39) +90.8

The results on main were measured in another session, so a few points up or down can be the machine and not the change.

Compliance

Suite Tool Score Failed
H/1.1 (h1) http11probe d50d818 125 / 159 34
H/2 (h2) h2spec 2.6.0 130 / 146 (tls 132/146, h2c 130/146) 16
H/3 (h3) h3spec v0.1.15 58 / 81 23
H/1.1: 34 failed
Test Expected Got
RFC9112-5.1-OBS-FOLD Obs-fold (line folding) in headers should be rejected (RFC 9112 §5.1, MUST) 400 200, connection left open
RFC9112-7.1-MISSING-HOST Request without Host header must be rejected with 400 (RFC 9112 §3.2, MUST) 400 200, connection left open
RFC9112-2.3-INVALID-VERSION Invalid HTTP version must be rejected (RFC 9112 §2.3, SHOULD) 400/505, close, or timeout = warn 200, connection left open
COMP-HOST-WITH-USERINFO Host header with userinfo (user@host) must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-HOST-WITH-PATH Host header with path component must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-ASTERISK-WITH-GET Asterisk-form (*) request-target with GET must be rejected (RFC 9112 §3.2.4, MUST) 400, close, or timeout = warn 404, connection left open
COMP-OPTIONS-STAR OPTIONS * is the only valid asterisk-form request (RFC 9112 §3.2.4, MUST) 2xx or 405; close/timeout = warn 404, connection left open
COMP-UNKNOWN-TE-501 Unknown Transfer-Encoding without CL should be rejected with 501 (RFC 9112 §6.1, MUST) 400/501 or close 200, connection left open
COMP-METHOD-CONNECT CONNECT to an origin server must be rejected (RFC 9110 §9.3.6, SHOULD) 400/405/501 or close 404, connection left open
COMP-HOST-EMPTY-VALUE Empty Host header value must be rejected (RFC 9112 §3.2, MUST) 400 or close 200, connection left open
COMP-REQUEST-LINE-TAB Tab as request-line delimiter — SHOULD reject but MAY parse on whitespace (RFC 9112 §3, SHOULD) 400 or 2xx; close/timeout = warn 404, connection left open
COMP-DATE-HEADER Origin server must include Date header in responses (RFC 9110 §6.6.1, MUST) 2xx with Date header 200, connection left open
COMP-VERSION-CASE HTTP version is case-sensitive — lowercase 'http' must be rejected (RFC 9112 §2.3, MUST) 400, close, or timeout = warn 404, connection left open
SMUG-TE-XCHUNKED Transfer-Encoding: xchunked must not be treated as chunked (RFC 9112 §6.1, MUST) 400/501 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-BARE-SEMICOLON Chunk size with bare semicolon and no extension name must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-INVALID-TOKEN Chunk extension with invalid token character must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-UNDERSCORE Chunk size with underscores (1_0) must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-TE-EMPTY-VALUE Transfer-Encoding with empty value must be rejected (RFC 9112 §6.1, MUST) 400 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-HEX-PREFIX Chunk size with 0x prefix must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SIZE-PLUS Chunk size with leading plus sign must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SIZE-TRAILING-OWS Chunk size with trailing whitespace before CRLF must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-LEADING-SP Chunk size with leading space must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-MISSING-TRAILING-CRLF Chunk data without trailing CRLF must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-SPILL Chunk declares size 5 but sends 7 bytes — oversized chunk data must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-CTRL NUL byte in chunk extension must be rejected (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-CHUNK-EXT-CR Bare CR (not CRLF) in chunk extension — some parsers treat CR alone as line ending (RFC 9112 §7.1.1, MUST) 400 or close 200, connection left open
SMUG-TE-IDENTITY Transfer-Encoding: identity (deprecated) with CL must be rejected (RFC 9112 §7, MUST) 400/501 or close 200, connection left open (Used CL (ignored TE variant))
SMUG-CHUNK-NEGATIVE Negative chunk size must be rejected (RFC 9112 §7.1, MUST) 400 or close 200, connection left open
SMUG-MULTIPLE-HOST-COMMA Host header with comma-separated values must be rejected (RFC 9110 §7.2, MUST) 400 or close 200, connection left open
SMUG-CHUNK-INVALID-SIZE-DESYNC Invalid chunk size (+0) with poison byte — detects chunk-size parser desync (RFC 9112 §7.1, MUST) 400, or close 200, timed out (Accepted invalid chunk-size request and kept connection open — follow-up GET returned no response)
MAL-NUL-IN-URL NUL byte in URL should be rejected (MUST) 400 or close 404, connection left open
MAL-NON-ASCII-URL Non-ASCII bytes (UTF-8 é) in URL must be rejected (MUST) 400 or close 404, connection left open
MAL-CHUNK-SIZE-OVERFLOW Chunk size with integer overflow must be rejected (MUST) 400 or close 200, connection left open
MAL-URL-OVERLONG-UTF8 Overlong UTF-8 encoding of / (0xC0 0xAF) in URL must be rejected (MUST) 400 or close 404, connection left open
H/2: 16 failed
Test Expected Got
generic/4/4 Sends a POST request with trailers HEADERS Frame (stream_id:1) Timeout
http2/3.5/2 Sends invalid connection preface GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed Error: unexpected EOF
http2/5.1/5 half closed (remote): Sends a DATA frame GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/5.1/6 half closed (remote): Sends a HEADERS frame GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/5.1/12 closed: Sends a HEADERS frame GOAWAY Frame (Error Code: STREAM_CLOSED) or Connection closed RST_STREAM Frame (length:4, flags:0x00, stream_id:1)
http2/5.1.1/2 Sends stream identifier that is numerically smaller than previous GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:5)
http2/6.1/2 Sends a DATA frame on the stream that is not in "open" or "half-closed (local)" state GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed Timeout
http2/6.9/2 Sends a WINDOW_UPDATE frame with a flow control window increment of 0 on a stream GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/6.9.1/3 Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR) RST_STREAM Frame (length:4, flags:0x00, stream_id:1)
http2/8.1/1 Sends a second HEADERS frame without the END_STREAM flag GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.1/3 Sends a HEADERS frame that contains a pseudo-header field as trailers GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.3/2 Sends a HEADERS frame that omits ":method" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:1)
http2/8.1.2.3/3 Sends a HEADERS frame that omits ":scheme" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed DATA Frame (length:7, flags:0x01, stream_id:1)
http2/8.1.2.3/4 Sends a HEADERS frame that omits ":path" pseudo-header field GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed HEADERS Frame (length:4, flags:0x05, stream_id:1)
http2/8.1.2.6/1 Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
http2/8.1.2.6/2 Sends a HEADERS frame with the "content-length" header field which does not equal the sum of the multiple DATA frames payload length GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed Timeout
H/3: 23 failed
Test Expected Got
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_source_connection_id is missing [Transport 7.3] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if original_destination_connection_id is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if preferred_address is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if retry_source_connection_id is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if stateless_reset_token is received [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if a parameter value is malformed [Transport 18] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_udp_payload_size < 1200 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if ack_delay_exponent > 20 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_ack_delay >= 2^14 [Transport 7.4 and 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_bidi > 2^60 [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_uni > 2^60 [Transport 18.2] - did not get expected exception: QUICException
QUIC servers MUST send FRAME_ENCODING_ERROR if a frame is cut short [Transport 12.4] - predicate failed on expected exception: QUICException / TransportErrorIsReceived ProtocolViolation ""
QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Handshake are non-zero [Transport 17.2] - did not get expected exception: QUICException
QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Short are non-zero [Transport 17.3.1] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with keep-alive as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with proxy-connection as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with transfer-encoding as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST treat a request with upgrade as malformed [HTTP/3 4.2] - did not get expected exception: QUICException
HTTP/3 servers MUST send H3_FRAME_ERROR if a SETTINGS frame stops mid-parameter [HTTP/3 7.1] - did not get expected exception: QUICException
HTTP/3 servers treats a repeated setting identifier as an error whatever its value [HTTP/3 7.2.4] - did not get expected exception: QUICException
HTTP/3 servers MUST send H3_FRAME_UNEXPECTED if PUSH_PROMISE is received [HTTP/3 7.2.5] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 512) "Decompression of header block failed."
HTTP/3 servers MUST send H3_FRAME_ERROR if MAX_PUSH_ID has octets after its push ID [HTTP/3 7.1] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 261) "Unexpected frame type '0' received"
HTTP/3 servers MUST NOT buffer an over-long frame on a control stream [HTTP/3 7.1] - predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 262) "Received an invalid frame len 1073741824 for frame of type 7."

@bjhham

bjhham commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

lgtm, merge away 👍

I'll follow up later about these resilience tests

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants