Repository navigation
Trillium production stack - #1615
Conversation
The standalone container runs on --network host whenever an entry subscribes to a compose profile, and it was left running while the gateway and production-stack stacks came up. An entry that also serves h2/h3 keeps 8443 (and 8080), so the stack's edge failed to bind, shared the port, or exited on the QUIC port, and the checks reached the standalone server instead of the stack.
The compose server never came up: it builds a StaticFileHandler over /data/static at startup, which the production-stack compose did not mount, and trillium-static exits on a missing directory. Mount it as the gateway stacks do, and have the edge wait for the server's UDS rather than for the container to start.
|
I need some time to take a look at this, thanks for the pr |
…uthsvc on Rust 1.100, unbreak the aspnet gateway entries (#1767) * validate.sh: stop the standalone container before the compose profiles The standalone container runs on --network host whenever an entry subscribes to a compose profile, and it was left running while the gateway and production-stack stacks came up. An entry that also serves h2/h3 keeps 8443 (and 8080), so the stack's edge failed to bind, shared the port, or exited on the QUIC port, and the checks reached the standalone server instead of the stack. * authsvc: ship the real binary under cargo 1.100's build-dir layout production-stack has failed on the bench host since the morning of 2026-11-13 on every entry (roadrunner-gateway #1757, morojs #1761, tako-gateway #1762): the shared authsvc container exits 0 with no output, the edge never sees it healthy, and the run ends with no results. rust:1-alpine moved to Rust 1.100 on 2026-11-12. Its cargo keeps each package's artifacts under target/release/build/<package>/<hash>/ rather than target/release/deps/, so the cache-priming step's `rm -rf target/release/deps/authsvc*` stopped matching anything. The placeholder built from `fn main() {}` stayed up to date, the second `cargo build` finished in 0.04s without compiling src/, and the image shipped the placeholder: a 418 KB binary that exits 0 at once. `cargo clean --release -p authsvc` removes the placeholder's artifacts wherever cargo keeps them, and a grep for the server's startup line fails the build if the placeholder is ever shipped again. Built against the current rust:1-alpine (1.100.0): a 1.2 MB binary that logs "authsvc listening on ...", answers /_health with 200 and an /_auth without a token with 401. * aspnet-minimal_nginx, aspnet-minimal_caddy: build against aspnet-minimal's Services and Types Both Dockerfiles still COPY aspnet-minimal's AppData.cs and Models.cs, which #1002 (2026-07-18) split into Services/ and Types/, so neither image has built since. They were the only failures in #1615's validation sweep, and their published results have not been refreshed since July. Copy Handlers.cs with Services/ and Types/, register DatabaseService, DatasetService, ItemService and the AppJsonContext resolver the way aspnet-minimal's Program.cs does, and give the nginx entry's /api routes DatabaseService's Postgres pool in place of AppData.PgDataSource. The caddy entry gains the StackExchange.Redis reference DatabaseService needs; it connects to nothing unless REDIS_URL is set, which gateway-h3 never does. --------- Co-authored-by: Jacob Rothstein <[email protected]>
|
Your validate.sh commit is on main now as part of #1767 (squashed, with you as co-author). Thanks for it. It turned out to be the fix for the cache-aside failure on #1761, where the leftover standalone morojs was answering part of the stack's traffic through SO_REUSEPORT. In #1767's run on the bench host, every entry it affects passed the gateway and production-stack checks, trillium and trillium-tuned included. That leaves this PR conflicting only in |
# Conflicts: # frameworks/trillium-tuned/Cargo.lock # frameworks/trillium/Cargo.lock
|
/benchmark-multiple --save |
|
👋 Benchmark request received. A collaborator will review and approve the run. |
Benchmark ResultsFrameworks: 2 | Test: ✅
|
| Test | Conn | RPS | Rate | CPU/req | p99 | p99.9 | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|---|---|---|---|
| baseline | 4096 | 1,321,859 | — | — | — | — | 3390.2% | 201MiB | +0.4% | +0.5% |
| pipelined | 4096 | 3,941,520 | — | — | — | — | 6293.7% | 150MiB | +0.2% | -18.5% |
| limited-conn | 4096 | 624,633 | — | — | — | — | 1884.9% | 138MiB | +0.2% | -3.5% |
| json-comp | 4096 | 117,131 | — | — | — | — | 6096.0% | 807MiB | +1.5% | +1.5% |
| json-comp | 16384 | 118,589 | — | — | — | — | 5994.8% | 784MiB | +0.6% | -2.9% |
| json-tls | 4096 | 1,048,553 | — | — | — | — | 5116.5% | 428MiB | +0.8% | +18.2% |
| 8gbit | 512 | 48,970 | 0.9794 |
46.59us | 139.0us | 6898.0us | 228.8% | 224MiB | +0.8% | -4.7% |
| static-tls | 1024 | 683,651 | — | — | — | — | 6024.6% | 494MiB | +2.3% | +6.2% |
| async-db | 1024 | 222,635 | — | — | — | — | 3113.5% | 201MiB | ~0% | +1.0% |
| fortunes | 1024 | 201,737 | — | — | — | — | 4689.4% | 303MiB | -5.7% | -8.2% |
| baseline-h2 | 256 | 4,090,664 | — | — | — | — | 6389.8% | 327MiB | +3.7% | +2.8% |
| baseline-h2 | 1024 | 3,436,518 | — | — | — | — | 6453.2% | 601MiB | -0.3% | -0.7% |
| static-h2 | 256 | 21,260 | — | — | — | — | 3918.3% | 1.8GiB | +0.2% | ~0% |
| static-h2 | 1024 | 17,762 | — | — | — | — | 4428.8% | 2.6GiB | ~0% | +23.8% |
| baseline-h2c | 256 | 4,069,118 | — | — | — | — | 6427.4% | 314MiB | -5.0% | +4.0% |
| baseline-h2c | 1024 | 3,562,309 | — | — | — | — | 6381.9% | 476MiB | -0.3% | -42.5% |
| baseline-h2c | 4096 | 3,481,833 | — | — | — | — | 6110.7% | 1.3GiB | +4.1% | +30.0% |
| json-h2c | 1024 | 2,058,299 | — | — | — | — | 6402.7% | 962MiB | +2.6% | -21.7% |
| json-h2c | 4096 | 1,840,089 | — | — | — | — | 6037.0% | 2.9GiB | -3.7% | +11.5% |
| baseline-h3 | 64 | 462,611 | — | — | — | — | 2583.1% | 297MiB | +4.1% | +8.8% |
| static-h3 | 64 | 220,372 | — | — | — | — | 2382.2% | 444MiB | -2.1% | -6.9% |
| gateway-64 | 512 | 68,957 | — | — | — | — | 4162.7% | 3.2GiB | -4.7% | +10.3% |
| gateway-64 | 1024 | 68,626 | — | — | — | — | 4306.6% | 5.5GiB | +6.9% | +3.8% |
| gateway-h3 | 64 | 76,398 | — | — | — | — | 3948.0% | 838MiB | -1.8% | +1.3% |
| gateway-h3 | 256 | 58,576 | — | — | — | — | 2817.8% | 1.9GiB | -1.4% | +11.8% |
| production-stack | 256 | 15,537 | — | — | — | — | 1103.0% | 1.1GiB | NEW | NEW |
| production-stack | 1024 | 46,799 | — | — | — | — | 3441.1% | 3.6GiB | NEW | NEW |
| unary-grpc | 256 | 3,059,755 | — | — | — | — | 6534.2% | 322MiB | +6.1% | +14.2% |
| unary-grpc | 1024 | 2,603,720 | — | — | — | — | 6438.7% | 574MiB | +5.4% | +4.4% |
| unary-grpc-tls | 256 | 2,778,948 | — | — | — | — | 6407.2% | 462MiB | +0.2% | +29.1% |
| unary-grpc-tls | 1024 | 2,451,838 | — | — | — | — | 6454.1% | 718MiB | +4.1% | +4.7% |
| echo-ws | 512 | 1,144,309 | — | — | — | — | 1743.3% | 116MiB | -0.7% | +1.8% |
| echo-ws | 4096 | 1,389,942 | — | — | — | — | 2291.7% | 236MiB | -1.5% | -11.9% |
| echo-ws | 16384 | 1,372,263 | — | — | — | — | 2368.5% | 323MiB | +1.4% | -4.4% |
| echo-ws-pipeline | 512 | 17,948,083 | — | — | — | — | 2088.3% | 107MiB | +1.3% | -10.8% |
| echo-ws-pipeline | 4096 | 21,916,169 | — | — | — | — | 2822.4% | 211MiB | +1.2% | -11.7% |
| echo-ws-pipeline | 16384 | 21,433,900 | — | — | — | — | 2822.9% | 305MiB | +15.6% | +22.0% |
| echo-ws-limited | 512 | 613,064 | — | — | — | — | 1572.2% | 144MiB | -0.7% | -1.4% |
| echo-ws-limited | 4096 | 629,297 | — | — | — | — | 1544.0% | 170MiB | -0.4% | +29.8% |
| latency-1m | 1024 | 986,592 | 0.9866 | 23.60us | 2603.0us | 15708.0us | 2482.1% | 141MiB | +1.7% | +12.8% |
| latency-10k | 1024 | 9,880 | 0.9880 | 24.30us | 101.0us | 805.0us | 22.3% | 139MiB | -0.2% | +0.7% |
| latency-500k-8cpu | 1024 | 352,076 | 0.7042 |
22.66us | 5548032.0us | 5679104.0us | 786.6% | 75MiB | -0.2% | -1.3% |
| async | 32000 | 493,339 | — | — | — | — | 2940.2% | 1.3GiB | -1.3% | +8.3% |
Composite score, main against this run, in the board's default view:
| Family | main | this run | Δ |
|---|---|---|---|
| HTTP/1.1 | 4257.7 (28 of 107) | 4253.7 (28 of 107) | -4.0 |
| HTTP/2 | 1589.3 (9 of 40) | 1588.8 (9 of 40) | -0.5 |
| HTTP/3 | 865.4 (7 of 19) | 852.8 (8 of 19) | -12.6 |
| Gateway | 601.8 (8 of 9) | 732.1 (8 of 9) | +130.3 |
| gRPC | 846.4 (6 of 13) | 879.8 (6 of 13) | +33.4 |
| WebSocket | 940.8 (24 of 38) | 958.9 (24 of 38) | +18.1 |
The results on main were measured in another session, so a few points up or down can be the machine and not the change.
✅ trillium-tuned
| Test | Conn | RPS | Rate | CPU/req | p99 | p99.9 | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|---|---|---|---|
| baseline | 4096 | 2,447,041 | — | — | — | — | 6710.7% | 433MiB | ~0% | +0.2% |
| pipelined | 4096 | 3,696,010 | — | — | — | — | 6321.8% | 371MiB | +2.8% | -4.9% |
| limited-conn | 4096 | 1,924,410 | — | — | — | — | 6484.8% | 524MiB | -2.1% | +3.1% |
| json-comp | 4096 | 118,276 | — | — | — | — | 6227.6% | 1.0GiB | +2.4% | ~0% |
| json-comp | 16384 | 99,867 | — | — | — | — | 6122.4% | 1.4GiB | -4.0% | -6.7% |
| json-tls | 4096 | 1,321,822 | — | — | — | — | 6542.0% | 523MiB | +1.3% | -0.8% |
| 8gbit | 512 | 49,117 | 0.9823 | 47.52us | 122.0us | 1653.0us | 216.9% | 234MiB | -0.4% | -1.7% |
| static-tls | 1024 | 784,391 | — | — | — | — | 6461.7% | 752MiB | +0.8% | -21.7% |
| async-db | 1024 | 245,053 | — | — | — | — | 3555.1% | 280MiB | +7.2% | -4.4% |
| fortunes | 1024 | 166,211 | — | — | — | — | 4449.4% | 375MiB | +1.6% | +0.3% |
| baseline-h2 | 256 | 5,385,385 | — | — | — | — | 6300.7% | 650MiB | -1.3% | -2.4% |
| baseline-h2 | 1024 | 4,062,132 | — | — | — | — | 6264.5% | 1.8GiB | -3.8% | +5.9% |
| static-h2 | 256 | 207,837 | — | — | — | — | 6384.1% | 14.2GiB | -0.6% | +2.9% |
| static-h2 | 1024 | 195,722 | — | — | — | — | 6330.4% | 17.5GiB | +0.5% | -11.6% |
| baseline-h2c | 256 | 5,559,206 | — | — | — | — | 6152.6% | 593MiB | -2.6% | +3.7% |
| baseline-h2c | 1024 | 4,247,239 | — | — | — | — | 6273.3% | 1.5GiB | -3.7% | -6.3% |
| baseline-h2c | 4096 | 3,762,724 | — | — | — | — | 6349.7% | 5.5GiB | -5.2% | +1.9% |
| json-h2c | 1024 | 2,006,958 | — | — | — | — | 6369.4% | 1.2GiB | -1.3% | ~0% |
| json-h2c | 4096 | 1,843,704 | — | — | — | — | 6242.4% | 3.3GiB | -1.5% | -2.9% |
| baseline-h3 | 64 | 465,636 | — | — | — | — | 2522.1% | 372MiB | +2.3% | -1.1% |
| static-h3 | 64 | 233,215 | — | — | — | — | 2443.6% | 697MiB | +3.1% | +5.3% |
| gateway-64 | 512 | 69,226 | — | — | — | — | 4033.5% | 3.4GiB | -2.7% | ~0% |
| gateway-64 | 1024 | 68,584 | — | — | — | — | 4337.0% | 5.3GiB | +1.4% | -10.2% |
| gateway-h3 | 64 | 77,688 | — | — | — | — | 3873.8% | 869MiB | +0.4% | -0.9% |
| gateway-h3 | 256 | 58,631 | — | — | — | — | 2846.2% | 1.7GiB | -1.7% | -15.0% |
| production-stack | 256 | 17,552 | — | — | — | — | 1141.5% | 1.3GiB | NEW | NEW |
| production-stack | 1024 | 46,114 | — | — | — | — | 3356.7% | 3.9GiB | NEW | NEW |
| unary-grpc | 256 | 3,823,615 | — | — | — | — | 6342.6% | 529MiB | -2.7% | -1.1% |
| unary-grpc | 1024 | 3,064,698 | — | — | — | — | 6349.6% | 1.6GiB | +0.8% | +6.7% |
| unary-grpc-tls | 256 | 3,728,130 | — | — | — | — | 6148.5% | 626MiB | -2.4% | +2.5% |
| unary-grpc-tls | 1024 | 2,823,406 | — | — | — | — | 6093.8% | 1.8GiB | -1.2% | +28.6% |
| echo-ws | 512 | 4,061,462 | — | — | — | — | 6466.6% | 173MiB | +0.3% | -2.3% |
| echo-ws | 4096 | 4,121,639 | — | — | — | — | 6446.0% | 294MiB | +1.0% | +4.6% |
| echo-ws | 16384 | 3,604,368 | — | — | — | — | 6106.2% | 780MiB | -0.7% | +3.2% |
| echo-ws-pipeline | 512 | 52,631,128 | — | — | — | — | 6495.3% | 174MiB | +1.2% | +0.6% |
| echo-ws-pipeline | 4096 | 53,286,433 | — | — | — | — | 6636.3% | 306MiB | +3.1% | +7.0% |
| echo-ws-pipeline | 16384 | 46,915,465 | — | — | — | — | 6209.9% | 922MiB | +0.1% | +23.8% |
| echo-ws-limited | 512 | 2,009,768 | — | — | — | — | 5396.4% | 200MiB | +1.5% | +3.1% |
| echo-ws-limited | 4096 | 2,472,798 | — | — | — | — | 6203.5% | 424MiB | +0.6% | -0.7% |
| latency-1m | 1024 | 997,593 | 0.9976 | 26.91us | 205.0us | 8204.0us | 2946.8% | 207MiB | ~0% | +0.5% |
| latency-10k | 1024 | 9,983 | 0.9983 | 29.57us | 118.0us | 175.0us | 28.3% | 207MiB | ~0% | -1.9% |
| latency-500k-8cpu | 1024 | 358,751 | 0.7175 |
22.24us | 6318080.0us | 6735872.0us | 785.3% | 101MiB | +1.8% | +26.2% |
| async | 32000 | 1,973,380 | — | — | — | — | 6429.4% | 1.4GiB | -2.0% | ~0% |
Composite score, main against this run, in the board's default view:
| Family | main | this run | Δ |
|---|---|---|---|
| HTTP/1.1 | 6035.7 (11 of 107) | 6021.4 (11 of 107) | -14.4 |
| HTTP/2 | 1978.6 (7 of 40) | 1939.2 (7 of 40) | -39.4 |
| HTTP/3 | 871.2 (6 of 19) | 897.7 (6 of 19) | +26.5 |
| Gateway | 605.6 (7 of 9) | 738.9 (7 of 9) | +133.3 |
| gRPC | 1101.8 (5 of 13) | 1085.3 (5 of 13) | -16.5 |
| WebSocket | 2830.4 (3 of 38) | 2856.0 (1 of 38) | +25.7 |
The results on main were measured in another session, so a few points up or down can be the machine and not the change.
Description
This builds on #1614 but I thought it was more likely to require discussion so I opened it as a distinct PR.
This both fixes a bug in the validation script and also implements "production-stack" for trillium.
validate.shstarts an entry's standalone container with--network hostwhenever the entry subscribes to a compose profile (gateway-64, gateway-h3, production-stack), and leaves it running while the compose stacks come up. The stacks bind the same host ports (8443 at the edge, 8080 for the server), so for an entry whose standalone server also serves h2/h3 on 8443:GET /api/items/1that should be a 401 is a 200 withserver: trillium/1.4.0.starting HTTP/3 QUIC listener: listen udp :8443: bind: address already in use, and the gateway checks pass against the standalone server instead of the stack.libuvcpp works around this inside the entry: see
frameworks/libuvcpp/main.ccnearinit_database("the harness does not stop it before the gateway phase") andproxy-production/nginx.conf(loopback binds plus reuseport, so the edge wins localhost). The fix belongs in the harness, so this stops the standalone container before the compose profiles. It usesdocker stoprather thanrm, so the finaldump_logscan still read it; the EXIT trap removes it as before. benchmark.sh already brings up one profile at a time, so the stacks now see the same host ports here as they do in a benchmark run.Because the gateway checks now reach the stacks rather than the standalone server, entries whose gateway stacks were broken but validated anyway will start failing those checks.
If this merges, libuvcpp can stop working around this harness bug and the maintainer would likely want to be pinged.
PR Commands — comment on this PR to trigger (requires collaborator approval):
/benchmark -f <framework>/benchmark -f <framework> -t <test>/benchmark -f <framework> --save/benchmark -f <framework> -t <test> --save/benchmark -f <framework> --compare <other>/benchmark-multiple -f <fw1>,<fw2>,...-tand--savetoo; saved results land in a single commit/benchmark-multiple --save-fneeded: benchmark and save every framework the PR touches/benchmark-test -t <test><test>and save the resultsFor
/benchmark, always specify-f <framework>; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory — one table per framework on multi runs. A new benchmark comment while a run is in flight queues behind it (one deep) instead of cancelling it. For multi-framework PRs (dependency bumps, same-language refactors) prefer/benchmark-multiple, which runs everything in a single job and commits all saved results together, so no run overwrites another.--compareworks on single-framework runs only.What the deltas are measured against. By default, this framework's own results published on
main- answering "did this change help?". When you are tuning a variant or a successor entry,--comparere-bases them on another entry instead:The reply states which baseline it used, and profiles the other framework does not run show
n/arather than a delta.Run benchmarks locally
You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.
Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk) are built as self-contained Docker images on first run.