Repository navigation
Conversation
|
👋 Heads up! This PR modifies the following frameworks:
|
|
/spec -f ktor --save |
|
/spec -f ktor-tuned --save |
|
👋 Spec request received. A collaborator will review and approve the run. |
1 similar comment
|
👋 Spec request received. A collaborator will review and approve the run. |
|
|
Spec ResultsFramework: Compliance
H/1.1: 34 failed
H/2: 17 failed
H/3: 23 failed
Full log |
|
|
Spec ResultsFramework: Compliance
H/1.1: 34 failed
H/2: 16 failed
H/3: 23 failed
Full log |
|
/resilience -f ktor |
|
👋 Resilience request received. A collaborator will review and approve the run. |
|
/resilience -f ktor |
|
👋 Resilience request received. A collaborator will review and approve the run. |
|
If the JVM args prevent the killing then I can include them in the other JVM frameworks in a follow up. |
|
Looks like the args didn't help... Maybe I'll try -Xmx3g |
|
/resilience -f ktor-tuned |
|
👋 Resilience request received. A collaborator will review and approve the run. |
|
I'll need to take these resilience tests offline to get them sorted. I'll disable them now for Ktor and see if I can figure out what's going on there. |
|
/benchmark -f ktor-tuned |
|
👋 Benchmark request received. A collaborator will review and approve the run. |
|
@bjhham should I remove the results from here? |
|
Yes, please. I'll get my hands on a server and investigate the memory issue before committing the results. |
Benchmark ResultsFramework:
Composite score, main against this run, in the board's default view:
The results on main were measured in another session, so a few points up or down can be the machine and not the change. Compliance
H/1.1: 34 failed
H/2: 14 failed
H/3: 23 failed
|
|
Looks like reducing the worker threads causes some of the benchmarks to decline and the static h3 one must have been running out of IO threads. I'll have some optimizations for the h2 pipeline in an upcoming release and I'll get back to ya on these resilience tests. |
|
/benchmark-multiple --save |
|
👋 Benchmark request received. A collaborator will review and approve the run. |
there seems to be an abnormal memory usage in the latest run too 120GB+ in some tests |
It's likely just the VM arg to increase it from 1/4 of the host's memory to 1/2, then the garbage collector just lets the garbage pile up. |
|
If you want to monitor the actual memory consumption for JVM applications, there's some extra tooling that tracks the "old generation" of allocated memory for things that linger on past GC runs. |
Isnt there a way to just let it be the minimum between like 10GB and 1/2 of the available? For me this is fine but the memory may be misleading to people visiting the site |
There's no ootb args for this unfortunately. Without measuring old gen it'll just occupy whatever space it's given and appear wasteful in the reports. |
|
|
Benchmark ResultsFrameworks: 2 | Test: ✅
|
| Test | Conn | RPS | Rate | CPU/req | p99 | p99.9 | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|---|---|---|---|
| baseline | 4096 | 975,505 | — | — | — | — | 5650.1% | 13.3GiB | -7.7% | +155.8% |
| pipelined | 4096 | 2,145,020 | — | — | — | — | 5369.0% | 74.0GiB | -3.6% | +781.0% |
| limited-conn | 4096 | 805,555 | — | — | — | — | 5486.1% | 22.9GiB | -1.5% | +131.3% |
| json-comp | 4096 | 215,698 | — | — | — | — | 6337.7% | 10.9GiB | ~0% | +14.7% |
| json-comp | 16384 | 171,972 | — | — | — | — | 5842.4% | 75.7GiB | -18.6% | +582.0% |
| json-tls | 4096 | 514,171 | — | — | — | — | 4819.2% | 11.4GiB | +6.8% | +107.3% |
| 8gbit | 512 | 49,353 | 0.9871 | 96.02us | 193.0us | 7142.0us | 456.6% | 2.5GiB | ~0% | +66.7% |
| static-tls | 1024 | 256,796 | — | — | — | — | 6194.1% | 4.8GiB | -3.2% | +23.1% |
| async-db | 1024 | 52,342 | — | — | — | — | 2374.4% | 5.3GiB | +11.9% | +96.3% |
| fortunes | 1024 | 58,940 | — | — | — | — | 5153.4% | 23.0GiB | -4.6% | +115.0% |
| baseline-h2 | 256 | 776,122 | — | — | — | — | 6154.3% | 82.1GiB | -5.8% | +474.1% |
| baseline-h2 | 1024 | 749,725 | — | — | — | — | 6038.9% | 117.0GiB | +2.1% | +375.6% |
| static-h2 | 256 | 222,008 | — | — | — | — | 6070.3% | 16.6GiB | -1.3% | +176.7% |
| static-h2 | 1024 | 142,862 | — | — | — | — | 5635.6% | 94.8GiB | -30.8% | +549.3% |
| baseline-h2c | 256 | 809,735 | — | — | — | — | 6225.3% | 75.7GiB | -5.3% | +429.4% |
| baseline-h2c | 1024 | 767,545 | — | — | — | — | 5880.2% | 117.9GiB | +2.2% | +367.9% |
| baseline-h2c | 4096 | 417,222 | — | — | — | — | 5269.4% | 127.2GiB | -8.3% | +313.0% |
| json-h2c | 1024 | 463,871 | — | — | — | — | 5559.6% | 91.2GiB | +3.6% | +629.6% |
| json-h2c | 4096 | 322,811 | — | — | — | — | 4974.6% | 123.9GiB | -19.5% | +369.3% |
| baseline-h3 | 64 | 330,233 | — | — | — | — | 3991.1% | 5.8GiB | -2.3% | +18.4% |
| static-h3 | 64 | 132,075 | — | — | — | — | 6030.0% | 6.1GiB | +35.0% | +27.1% |
| echo-ws | 512 | 1,029,165 | — | — | — | — | 3392.3% | 2.5GiB | -3.1% | +56.2% |
| echo-ws | 4096 | 963,948 | — | — | — | — | 3425.3% | 5.2GiB | -1.3% | +147.6% |
| echo-ws | 16384 | 753,029 | — | — | — | — | 3361.3% | 9.3GiB | +12.4% | +126.8% |
| echo-ws-pipeline | 512 | 10,829,316 | — | — | — | — | 3333.2% | 5.0GiB | -6.2% | -43.8% |
| echo-ws-pipeline | 4096 | 9,720,275 | — | — | — | — | 3550.8% | 15.4GiB | -12.1% | +366.7% |
| echo-ws-pipeline | 16384 | 7,107,081 | — | — | — | — | 3595.1% | 62.5GiB | -21.8% | +453.1% |
| latency-1m | 1024 | 997,449 | 0.9974 | 51.30us | 3115.0us | 17352.0us | 5614.7% | 11.7GiB | ~0% | +125.0% |
| latency-10k | 1024 | 9,981 | 0.9981 | 69.42us | 156.0us | 3113.0us | 66.9% | 592MiB | ~0% | +23.1% |
| latency-500k-8cpu | 1024 | 221,373 | 0.4427 |
31.08us | 10981376.0us | 11104256.0us | 676.5% | 2.3GiB | -1.8% | +76.9% |
| async | 32000 | 850,051 | — | — | — | — | 5424.7% | 24.6GiB | -7.6% | +176.4% |
Composite score, main against this run, in the board's default view:
| Family | main | this run | Δ |
|---|---|---|---|
| HTTP/1.1 | 3264.7 (44 of 109) | 3263.7 (44 of 109) | -1.1 |
| HTTP/2 | 548.8 (30 of 41) | 496.4 (30 of 41) | -52.4 |
| HTTP/3 | 404.8 (12 of 20) | 521.6 (12 of 20) | +116.8 |
| WebSocket | 419.8 (33 of 39) | 398.2 (34 of 39) | -21.6 |
The results on main were measured in another session, so a few points up or down can be the machine and not the change.
Compliance
| Suite | Tool | Score | Failed |
|---|---|---|---|
H/1.1 (h1) |
http11probe d50d818 | 125 / 159 | 34 |
H/2 (h2) |
h2spec 2.6.0 | 129 / 146 (tls 130/146, h2c 129/146) | 17 |
H/3 (h3) |
h3spec v0.1.15 | 58 / 81 | 23 |
H/1.1: 34 failed
| Test | Expected | Got |
|---|---|---|
RFC9112-5.1-OBS-FOLD Obs-fold (line folding) in headers should be rejected (RFC 9112 §5.1, MUST) |
400 | 200, connection left open |
RFC9112-7.1-MISSING-HOST Request without Host header must be rejected with 400 (RFC 9112 §3.2, MUST) |
400 | 200, connection left open |
RFC9112-2.3-INVALID-VERSION Invalid HTTP version must be rejected (RFC 9112 §2.3, SHOULD) |
400/505, close, or timeout = warn | 200, connection left open |
COMP-HOST-WITH-USERINFO Host header with userinfo (user@host) must be rejected (RFC 9112 §3.2, MUST) |
400 or close | 200, connection left open |
COMP-HOST-WITH-PATH Host header with path component must be rejected (RFC 9112 §3.2, MUST) |
400 or close | 200, connection left open |
COMP-ASTERISK-WITH-GET Asterisk-form (*) request-target with GET must be rejected (RFC 9112 §3.2.4, MUST) |
400, close, or timeout = warn | 404, connection left open |
COMP-OPTIONS-STAR OPTIONS * is the only valid asterisk-form request (RFC 9112 §3.2.4, MUST) |
2xx or 405; close/timeout = warn | 404, connection left open |
COMP-UNKNOWN-TE-501 Unknown Transfer-Encoding without CL should be rejected with 501 (RFC 9112 §6.1, MUST) |
400/501 or close | 200, connection left open |
COMP-METHOD-CONNECT CONNECT to an origin server must be rejected (RFC 9110 §9.3.6, SHOULD) |
400/405/501 or close | 404, connection left open |
COMP-HOST-EMPTY-VALUE Empty Host header value must be rejected (RFC 9112 §3.2, MUST) |
400 or close | 200, connection left open |
COMP-REQUEST-LINE-TAB Tab as request-line delimiter — SHOULD reject but MAY parse on whitespace (RFC 9112 §3, SHOULD) |
400 or 2xx; close/timeout = warn | 404, connection left open |
COMP-DATE-HEADER Origin server must include Date header in responses (RFC 9110 §6.6.1, MUST) |
2xx with Date header | 200, connection left open |
COMP-VERSION-CASE HTTP version is case-sensitive — lowercase 'http' must be rejected (RFC 9112 §2.3, MUST) |
400, close, or timeout = warn | 404, connection left open |
SMUG-TE-XCHUNKED Transfer-Encoding: xchunked must not be treated as chunked (RFC 9112 §6.1, MUST) |
400/501 or close | 200, connection left open (Used CL (ignored TE variant)) |
SMUG-CHUNK-BARE-SEMICOLON Chunk size with bare semicolon and no extension name must be rejected (RFC 9112 §7.1.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-EXT-INVALID-TOKEN Chunk extension with invalid token character must be rejected (RFC 9112 §7.1.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-UNDERSCORE Chunk size with underscores (1_0) must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-TE-EMPTY-VALUE Transfer-Encoding with empty value must be rejected (RFC 9112 §6.1, MUST) |
400 or close | 200, connection left open (Used CL (ignored TE variant)) |
SMUG-CHUNK-HEX-PREFIX Chunk size with 0x prefix must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-SIZE-PLUS Chunk size with leading plus sign must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-SIZE-TRAILING-OWS Chunk size with trailing whitespace before CRLF must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-LEADING-SP Chunk size with leading space must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-MISSING-TRAILING-CRLF Chunk data without trailing CRLF must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-SPILL Chunk declares size 5 but sends 7 bytes — oversized chunk data must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-EXT-CTRL NUL byte in chunk extension must be rejected (RFC 9112 §7.1.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-EXT-CR Bare CR (not CRLF) in chunk extension — some parsers treat CR alone as line ending (RFC 9112 §7.1.1, MUST) |
400 or close | 200, connection left open |
SMUG-TE-IDENTITY Transfer-Encoding: identity (deprecated) with CL must be rejected (RFC 9112 §7, MUST) |
400/501 or close | 200, connection left open (Used CL (ignored TE variant)) |
SMUG-CHUNK-NEGATIVE Negative chunk size must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-MULTIPLE-HOST-COMMA Host header with comma-separated values must be rejected (RFC 9110 §7.2, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-INVALID-SIZE-DESYNC Invalid chunk size (+0) with poison byte — detects chunk-size parser desync (RFC 9112 §7.1, MUST) |
400, or close | 200, timed out (Accepted invalid chunk-size request and kept connection open — follow-up GET returned no response) |
MAL-NUL-IN-URL NUL byte in URL should be rejected (MUST) |
400 or close | 404, connection left open |
MAL-NON-ASCII-URL Non-ASCII bytes (UTF-8 é) in URL must be rejected (MUST) |
400 or close | 404, connection left open |
MAL-CHUNK-SIZE-OVERFLOW Chunk size with integer overflow must be rejected (MUST) |
400 or close | 200, connection left open |
MAL-URL-OVERLONG-UTF8 Overlong UTF-8 encoding of / (0xC0 0xAF) in URL must be rejected (MUST) |
400 or close | 404, connection left open |
H/2: 17 failed
| Test | Expected | Got |
|---|---|---|
generic/4/4 Sends a POST request with trailers |
HEADERS Frame (stream_id:1) | Timeout |
http2/3.5/2 Sends invalid connection preface |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Error: unexpected EOF |
http2/5.1/5 half closed (remote): Sends a DATA frame |
GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed | Timeout |
http2/5.1/6 half closed (remote): Sends a HEADERS frame |
GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed | Timeout |
http2/5.1/12 closed: Sends a HEADERS frame |
GOAWAY Frame (Error Code: STREAM_CLOSED) or Connection closed | RST_STREAM Frame (length:4, flags:0x00, stream_id:1) |
http2/5.1.1/2 Sends stream identifier that is numerically smaller than previous |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed | DATA Frame (length:7, flags:0x01, stream_id:5) |
http2/6.1/2 Sends a DATA frame on the stream that is not in "open" or "half-closed (local)" state |
GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed | Timeout |
http2/6.3/2 Sends a PRIORITY frame with a length other than 5 octets |
GOAWAY Frame (Error Code: FRAME_SIZE_ERROR) or RST_STREAM Frame (Error Code: FRAME_SIZE_ERROR) or Connection closed | Timeout |
http2/6.9/2 Sends a WINDOW_UPDATE frame with a flow control window increment of 0 on a stream |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Timeout |
http2/6.9.1/3 Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream |
RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR) | RST_STREAM Frame (length:4, flags:0x00, stream_id:1) |
http2/8.1/1 Sends a second HEADERS frame without the END_STREAM flag |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Timeout |
http2/8.1.2.1/3 Sends a HEADERS frame that contains a pseudo-header field as trailers |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Timeout |
http2/8.1.2.3/2 Sends a HEADERS frame that omits ":method" pseudo-header field |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | DATA Frame (length:7, flags:0x01, stream_id:1) |
http2/8.1.2.3/3 Sends a HEADERS frame that omits ":scheme" pseudo-header field |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | DATA Frame (length:7, flags:0x01, stream_id:1) |
http2/8.1.2.3/4 Sends a HEADERS frame that omits ":path" pseudo-header field |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | HEADERS Frame (length:4, flags:0x05, stream_id:1) |
http2/8.1.2.6/1 Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Timeout |
http2/8.1.2.6/2 Sends a HEADERS frame with the "content-length" header field which does not equal the sum of the multiple DATA frames payload length |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Timeout |
H/3: 23 failed
| Test | Expected | Got |
|---|---|---|
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_source_connection_id is missing [Transport 7.3] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if original_destination_connection_id is received [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if preferred_address is received [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if retry_source_connection_id is received [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if stateless_reset_token is received [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if a parameter value is malformed [Transport 18] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_udp_payload_size < 1200 [Transport 7.4 and 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if ack_delay_exponent > 20 [Transport 7.4 and 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_ack_delay >= 2^14 [Transport 7.4 and 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_bidi > 2^60 [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_uni > 2^60 [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send FRAME_ENCODING_ERROR if a frame is cut short [Transport 12.4] | - | predicate failed on expected exception: QUICException / TransportErrorIsReceived ProtocolViolation "" |
| QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Handshake are non-zero [Transport 17.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Short are non-zero [Transport 17.3.1] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST treat a request with keep-alive as malformed [HTTP/3 4.2] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST treat a request with proxy-connection as malformed [HTTP/3 4.2] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST treat a request with transfer-encoding as malformed [HTTP/3 4.2] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST treat a request with upgrade as malformed [HTTP/3 4.2] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST send H3_FRAME_ERROR if a SETTINGS frame stops mid-parameter [HTTP/3 7.1] | - | did not get expected exception: QUICException |
| HTTP/3 servers treats a repeated setting identifier as an error whatever its value [HTTP/3 7.2.4] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST send H3_FRAME_UNEXPECTED if PUSH_PROMISE is received [HTTP/3 7.2.5] | - | predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 512) "Decompression of header block failed." |
| HTTP/3 servers MUST send H3_FRAME_ERROR if MAX_PUSH_ID has octets after its push ID [HTTP/3 7.1] | - | predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 261) "Unexpected frame type '0' received" |
| HTTP/3 servers MUST NOT buffer an over-long frame on a control stream [HTTP/3 7.1] | - | predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 262) "Received an invalid frame len 1073741824 for frame of type 7." |
✅ ktor-tuned
| Test | Conn | RPS | Rate | CPU/req | p99 | p99.9 | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|---|---|---|---|
| baseline | 4096 | 1,522,542 | — | — | — | — | 6298.8% | 24.4GiB | +1.1% | +313.6% |
| pipelined | 4096 | 2,303,001 | — | — | — | — | 6332.1% | 69.1GiB | -14.0% | +404.4% |
| limited-conn | 4096 | 700,363 | — | — | — | — | 5039.8% | 5.2GiB | +16.2% | +6.1% |
| json-comp | 4096 | 257,072 | — | — | — | — | 6487.7% | 10.4GiB | +3.0% | +89.1% |
| json-comp | 16384 | 244,300 | — | — | — | — | 6108.3% | 24.2GiB | +0.3% | +154.7% |
| json-tls | 4096 | 876,409 | — | — | — | — | 6244.0% | 13.1GiB | -1.4% | +104.7% |
| 8gbit | 512 | 49,418 | 0.9884 | 74.40us | 175.0us | 4146.0us | 355.0% | 2.5GiB | ~0% | +56.2% |
| static-tls | 1024 | 251,821 | — | — | — | — | 6391.0% | 9.3GiB | -4.9% | +151.4% |
| async-db | 1024 | 45,582 | — | — | — | — | 1978.9% | 2.6GiB | -11.0% | -7.1% |
| baseline-h2 | 256 | 1,016,779 | — | — | — | — | 6237.3% | 105.7GiB | -9.4% | +759.3% |
| baseline-h2 | 1024 | 942,509 | — | — | — | — | 6053.1% | 121.1GiB | -10.8% | +362.2% |
| static-h2 | 256 | 240,300 | — | — | — | — | 6158.0% | 17.4GiB | +1.1% | +123.1% |
| static-h2 | 1024 | 155,594 | — | — | — | — | 6161.0% | 18.1GiB | -28.8% | +48.4% |
| baseline-h2c | 256 | 1,043,320 | — | — | — | — | 6360.6% | 94.8GiB | -11.0% | +587.0% |
| baseline-h2c | 1024 | 1,000,256 | — | — | — | — | 6282.2% | 117.8GiB | -10.1% | +358.4% |
| baseline-h2c | 4096 | 466,502 | — | — | — | — | 5140.5% | 126.2GiB | -29.8% | +303.2% |
| json-h2c | 1024 | 674,042 | — | — | — | — | 6385.4% | 95.5GiB | -7.6% | +716.2% |
| json-h2c | 4096 | 543,178 | — | — | — | — | 5887.6% | 123.8GiB | -20.6% | +372.5% |
| baseline-h3 | 64 | 383,724 | — | — | — | — | 4267.1% | 6.6GiB | +1.4% | +34.7% |
| static-h3 | 64 | 137,218 | — | — | — | — | 5849.4% | 5.3GiB | +67.9% | +71.0% |
| echo-ws | 512 | 1,366,943 | — | — | — | — | 6399.0% | 2.5GiB | +25.8% | -7.4% |
| echo-ws | 4096 | 1,389,371 | — | — | — | — | 6702.0% | 10.9GiB | +17.7% | +303.7% |
| echo-ws | 16384 | 1,063,467 | — | — | — | — | 6662.9% | 13.2GiB | +23.3% | +247.4% |
| echo-ws-pipeline | 512 | 15,247,799 | — | — | — | — | 6563.2% | 9.3GiB | +29.4% | +93.8% |
| echo-ws-pipeline | 4096 | 14,631,527 | — | — | — | — | 6620.1% | 11.6GiB | +12.7% | +152.2% |
| echo-ws-pipeline | 16384 | 12,215,964 | — | — | — | — | 6440.5% | 60.6GiB | +2.8% | +436.3% |
| latency-1m | 1024 | 997,375 | 0.9974 | 47.17us | 1658.0us | 12732.0us | 5053.2% | 12.5GiB | ~0% | +135.8% |
| latency-10k | 1024 | 9,981 | 0.9981 | 47.53us | 137.0us | 2753.0us | 46.0% | 609MiB | ~0% | -60.4% |
| latency-500k-8cpu | 1024 | 266,492 | 0.5330 |
29.91us | 9203712.0us | 9293824.0us | 781.5% | 2.3GiB | -3.7% | -4.2% |
| async | 32000 | 1,154,794 | — | — | — | — | 6234.9% | 65.8GiB | -8.6% | +448.3% |
Composite score, main against this run, in the board's default view:
| Family | main | this run | Δ |
|---|---|---|---|
| HTTP/1.1 | 4052.8 (37 of 109) | 4109.3 (31 of 109) | +56.5 |
| HTTP/2 | 739.3 (24 of 41) | 641.1 (26 of 41) | -98.2 |
| HTTP/3 | 357.3 (13 of 20) | 550.0 (11 of 20) | +192.7 |
| WebSocket | 485.3 (31 of 39) | 576.1 (31 of 39) | +90.8 |
The results on main were measured in another session, so a few points up or down can be the machine and not the change.
Compliance
| Suite | Tool | Score | Failed |
|---|---|---|---|
H/1.1 (h1) |
http11probe d50d818 | 125 / 159 | 34 |
H/2 (h2) |
h2spec 2.6.0 | 130 / 146 (tls 132/146, h2c 130/146) | 16 |
H/3 (h3) |
h3spec v0.1.15 | 58 / 81 | 23 |
H/1.1: 34 failed
| Test | Expected | Got |
|---|---|---|
RFC9112-5.1-OBS-FOLD Obs-fold (line folding) in headers should be rejected (RFC 9112 §5.1, MUST) |
400 | 200, connection left open |
RFC9112-7.1-MISSING-HOST Request without Host header must be rejected with 400 (RFC 9112 §3.2, MUST) |
400 | 200, connection left open |
RFC9112-2.3-INVALID-VERSION Invalid HTTP version must be rejected (RFC 9112 §2.3, SHOULD) |
400/505, close, or timeout = warn | 200, connection left open |
COMP-HOST-WITH-USERINFO Host header with userinfo (user@host) must be rejected (RFC 9112 §3.2, MUST) |
400 or close | 200, connection left open |
COMP-HOST-WITH-PATH Host header with path component must be rejected (RFC 9112 §3.2, MUST) |
400 or close | 200, connection left open |
COMP-ASTERISK-WITH-GET Asterisk-form (*) request-target with GET must be rejected (RFC 9112 §3.2.4, MUST) |
400, close, or timeout = warn | 404, connection left open |
COMP-OPTIONS-STAR OPTIONS * is the only valid asterisk-form request (RFC 9112 §3.2.4, MUST) |
2xx or 405; close/timeout = warn | 404, connection left open |
COMP-UNKNOWN-TE-501 Unknown Transfer-Encoding without CL should be rejected with 501 (RFC 9112 §6.1, MUST) |
400/501 or close | 200, connection left open |
COMP-METHOD-CONNECT CONNECT to an origin server must be rejected (RFC 9110 §9.3.6, SHOULD) |
400/405/501 or close | 404, connection left open |
COMP-HOST-EMPTY-VALUE Empty Host header value must be rejected (RFC 9112 §3.2, MUST) |
400 or close | 200, connection left open |
COMP-REQUEST-LINE-TAB Tab as request-line delimiter — SHOULD reject but MAY parse on whitespace (RFC 9112 §3, SHOULD) |
400 or 2xx; close/timeout = warn | 404, connection left open |
COMP-DATE-HEADER Origin server must include Date header in responses (RFC 9110 §6.6.1, MUST) |
2xx with Date header | 200, connection left open |
COMP-VERSION-CASE HTTP version is case-sensitive — lowercase 'http' must be rejected (RFC 9112 §2.3, MUST) |
400, close, or timeout = warn | 404, connection left open |
SMUG-TE-XCHUNKED Transfer-Encoding: xchunked must not be treated as chunked (RFC 9112 §6.1, MUST) |
400/501 or close | 200, connection left open (Used CL (ignored TE variant)) |
SMUG-CHUNK-BARE-SEMICOLON Chunk size with bare semicolon and no extension name must be rejected (RFC 9112 §7.1.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-EXT-INVALID-TOKEN Chunk extension with invalid token character must be rejected (RFC 9112 §7.1.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-UNDERSCORE Chunk size with underscores (1_0) must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-TE-EMPTY-VALUE Transfer-Encoding with empty value must be rejected (RFC 9112 §6.1, MUST) |
400 or close | 200, connection left open (Used CL (ignored TE variant)) |
SMUG-CHUNK-HEX-PREFIX Chunk size with 0x prefix must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-SIZE-PLUS Chunk size with leading plus sign must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-SIZE-TRAILING-OWS Chunk size with trailing whitespace before CRLF must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-LEADING-SP Chunk size with leading space must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-MISSING-TRAILING-CRLF Chunk data without trailing CRLF must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-SPILL Chunk declares size 5 but sends 7 bytes — oversized chunk data must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-EXT-CTRL NUL byte in chunk extension must be rejected (RFC 9112 §7.1.1, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-EXT-CR Bare CR (not CRLF) in chunk extension — some parsers treat CR alone as line ending (RFC 9112 §7.1.1, MUST) |
400 or close | 200, connection left open |
SMUG-TE-IDENTITY Transfer-Encoding: identity (deprecated) with CL must be rejected (RFC 9112 §7, MUST) |
400/501 or close | 200, connection left open (Used CL (ignored TE variant)) |
SMUG-CHUNK-NEGATIVE Negative chunk size must be rejected (RFC 9112 §7.1, MUST) |
400 or close | 200, connection left open |
SMUG-MULTIPLE-HOST-COMMA Host header with comma-separated values must be rejected (RFC 9110 §7.2, MUST) |
400 or close | 200, connection left open |
SMUG-CHUNK-INVALID-SIZE-DESYNC Invalid chunk size (+0) with poison byte — detects chunk-size parser desync (RFC 9112 §7.1, MUST) |
400, or close | 200, timed out (Accepted invalid chunk-size request and kept connection open — follow-up GET returned no response) |
MAL-NUL-IN-URL NUL byte in URL should be rejected (MUST) |
400 or close | 404, connection left open |
MAL-NON-ASCII-URL Non-ASCII bytes (UTF-8 é) in URL must be rejected (MUST) |
400 or close | 404, connection left open |
MAL-CHUNK-SIZE-OVERFLOW Chunk size with integer overflow must be rejected (MUST) |
400 or close | 200, connection left open |
MAL-URL-OVERLONG-UTF8 Overlong UTF-8 encoding of / (0xC0 0xAF) in URL must be rejected (MUST) |
400 or close | 404, connection left open |
H/2: 16 failed
| Test | Expected | Got |
|---|---|---|
generic/4/4 Sends a POST request with trailers |
HEADERS Frame (stream_id:1) | Timeout |
http2/3.5/2 Sends invalid connection preface |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Error: unexpected EOF |
http2/5.1/5 half closed (remote): Sends a DATA frame |
GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed | Timeout |
http2/5.1/6 half closed (remote): Sends a HEADERS frame |
GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed | Timeout |
http2/5.1/12 closed: Sends a HEADERS frame |
GOAWAY Frame (Error Code: STREAM_CLOSED) or Connection closed | RST_STREAM Frame (length:4, flags:0x00, stream_id:1) |
http2/5.1.1/2 Sends stream identifier that is numerically smaller than previous |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or Connection closed | DATA Frame (length:7, flags:0x01, stream_id:5) |
http2/6.1/2 Sends a DATA frame on the stream that is not in "open" or "half-closed (local)" state |
GOAWAY Frame (Error Code: STREAM_CLOSED) or RST_STREAM Frame (Error Code: STREAM_CLOSED) or Connection closed | Timeout |
http2/6.9/2 Sends a WINDOW_UPDATE frame with a flow control window increment of 0 on a stream |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Timeout |
http2/6.9.1/3 Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream |
RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR) | RST_STREAM Frame (length:4, flags:0x00, stream_id:1) |
http2/8.1/1 Sends a second HEADERS frame without the END_STREAM flag |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Timeout |
http2/8.1.2.1/3 Sends a HEADERS frame that contains a pseudo-header field as trailers |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Timeout |
http2/8.1.2.3/2 Sends a HEADERS frame that omits ":method" pseudo-header field |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | DATA Frame (length:7, flags:0x01, stream_id:1) |
http2/8.1.2.3/3 Sends a HEADERS frame that omits ":scheme" pseudo-header field |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | DATA Frame (length:7, flags:0x01, stream_id:1) |
http2/8.1.2.3/4 Sends a HEADERS frame that omits ":path" pseudo-header field |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | HEADERS Frame (length:4, flags:0x05, stream_id:1) |
http2/8.1.2.6/1 Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Timeout |
http2/8.1.2.6/2 Sends a HEADERS frame with the "content-length" header field which does not equal the sum of the multiple DATA frames payload length |
GOAWAY Frame (Error Code: PROTOCOL_ERROR) or RST_STREAM Frame (Error Code: PROTOCOL_ERROR) or Connection closed | Timeout |
H/3: 23 failed
| Test | Expected | Got |
|---|---|---|
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_source_connection_id is missing [Transport 7.3] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if original_destination_connection_id is received [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if preferred_address is received [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if retry_source_connection_id is received [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if stateless_reset_token is received [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if a parameter value is malformed [Transport 18] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_udp_payload_size < 1200 [Transport 7.4 and 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if ack_delay_exponent > 20 [Transport 7.4 and 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if max_ack_delay >= 2^14 [Transport 7.4 and 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_bidi > 2^60 [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_max_streams_uni > 2^60 [Transport 18.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send FRAME_ENCODING_ERROR if a frame is cut short [Transport 12.4] | - | predicate failed on expected exception: QUICException / TransportErrorIsReceived ProtocolViolation "" |
| QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Handshake are non-zero [Transport 17.2] | - | did not get expected exception: QUICException |
| QUIC servers MUST send PROTOCOL_VIOLATION if reserved bits in Short are non-zero [Transport 17.3.1] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST treat a request with keep-alive as malformed [HTTP/3 4.2] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST treat a request with proxy-connection as malformed [HTTP/3 4.2] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST treat a request with transfer-encoding as malformed [HTTP/3 4.2] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST treat a request with upgrade as malformed [HTTP/3 4.2] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST send H3_FRAME_ERROR if a SETTINGS frame stops mid-parameter [HTTP/3 7.1] | - | did not get expected exception: QUICException |
| HTTP/3 servers treats a repeated setting identifier as an error whatever its value [HTTP/3 7.2.4] | - | did not get expected exception: QUICException |
| HTTP/3 servers MUST send H3_FRAME_UNEXPECTED if PUSH_PROMISE is received [HTTP/3 7.2.5] | - | predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 512) "Decompression of header block failed." |
| HTTP/3 servers MUST send H3_FRAME_ERROR if MAX_PUSH_ID has octets after its push ID [HTTP/3 7.1] | - | predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 261) "Unexpected frame type '0' received" |
| HTTP/3 servers MUST NOT buffer an over-long frame on a control stream [HTTP/3 7.1] | - | predicate failed on expected exception: QUICException / ApplicationProtocolErrorIsReceived (ApplicationProtocolError 262) "Received an invalid frame len 1073741824 for frame of type 7." |
|
lgtm, merge away 👍 I'll follow up later about these resilience tests |
Description
Add compliance and resilience to the Ktor frameworks. They both seem pretty ok on my pi though with wildly different arguments, so we'll see how it goes on the big server.
Also, increased the blockable I/O dispatcher threads and reduced worker threads in the tuned one.
PR Commands — comment on this PR to trigger (requires collaborator approval):
/benchmark -f <framework>/benchmark -f <framework> -t <test>/benchmark -f <framework> -t <t1>,<t2>,.../benchmark -f <framework> --save/benchmark -f <framework> -t <test> --save/benchmark -f <framework> --compare <other>/benchmark-multiple -f <fw1>,<fw2>,...-tand--savetoo; saved results land in a single commit/benchmark-multiple --save-fneeded: benchmark and save every framework the PR touches/benchmark-test -t <test><test>and save the results/spec -f <framework>specin itsmeta.json/spec -f <framework> --savesite/data/spec/(lands with the merge, like saved results)For
/benchmark, always specify-f <framework>; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory — one table per framework on multi runs. A new benchmark comment while a run is in flight queues behind it (one deep) instead of cancelling it. For multi-framework PRs (dependency bumps, same-language refactors) prefer/benchmark-multiple, which runs everything in a single job and commits all saved results together, so no run overwrites another.--compareworks on single-framework runs only.A
/benchmarkwithout-talso runs the compliance suites, and its--savesaves their scores with the results./specruns the suites alone; without-fit runs them for every framework the PR touches. See Compliance.What the deltas are measured against. By default, this framework's own results published on
main- answering "did this change help?". When you are tuning a variant or a successor entry,--comparere-bases them on another entry instead:The reply states which baseline it used, and profiles the other framework does not run show
n/arather than a delta.Run benchmarks locally
You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.
Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk) are built as self-contained Docker images on first run.