Skip to content

Trillium production stack - #1615

Merged
MDA2AV merged 5 commits into
MDA2AV:mainfrom
jbr:trillium-production-stack
Nov 13, 2026
Merged

MDA2AV merged 5 commits into
MDA2AV:mainfrom
jbr:trillium-production-stack

Conversation

@jbr

@jbr jbr commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Description

This builds on #1614 but I thought it was more likely to require discussion so I opened it as a distinct PR.

This both fixes a bug in the validation script and also implements "production-stack" for trillium.

validate.sh starts an entry's standalone container with --network host whenever the entry subscribes to a compose profile (gateway-64, gateway-h3, production-stack), and leaves it running while the compose stacks come up. The stacks bind the same host ports (8443 at the edge, 8080 for the server), so for an entry whose standalone server also serves h2/h3 on 8443:

  • The edge fails to bind 8443, or shares it when both sides set SO_REUSEPORT, and the checks reach whichever process answers. production-stack then fails its auth wall, because the edge enforces it and the server doesn't. In trillium's run on [trillium.rs] implement async, {baseline,json}-h2c, echo-ws-limited #1367 (https://github.com/MDA2AV/HttpArena/actions/runs/33343130569), the unauthenticated GET /api/items/1 that should be a 401 is a 200 with server: trillium/1.4.0.
  • A Caddy edge with h3 enabled exits with starting HTTP/3 QUIC listener: listen udp :8443: bind: address already in use, and the gateway checks pass against the standalone server instead of the stack.

libuvcpp works around this inside the entry: see frameworks/libuvcpp/main.cc near init_database ("the harness does not stop it before the gateway phase") and proxy-production/nginx.conf (loopback binds plus reuseport, so the edge wins localhost). The fix belongs in the harness, so this stops the standalone container before the compose profiles. It uses docker stop rather than rm, so the final dump_logs can still read it; the EXIT trap removes it as before. benchmark.sh already brings up one profile at a time, so the stacks now see the same host ports here as they do in a benchmark run.

Because the gateway checks now reach the stacks rather than the standalone server, entries whose gateway stacks were broken but validated anyway will start failing those checks.

If this merges, libuvcpp can stop working around this harness bug and the maintainer would likely want to be pinged.


PR Commands — comment on this PR to trigger (requires collaborator approval):

Command Description
/benchmark -f <framework> Run every test the framework subscribes to
/benchmark -f <framework> -t <test> Run one test only
/benchmark -f <framework> --save Run and save results (updates the leaderboard on merge)
/benchmark -f <framework> -t <test> --save Run one test and save results
/benchmark -f <framework> --compare <other> Measure the deltas against another framework instead of this one
/benchmark-multiple -f <fw1>,<fw2>,... Benchmark several frameworks in one run — takes -t and --save too; saved results land in a single commit
/benchmark-multiple --save No -f needed: benchmark and save every framework the PR touches
/benchmark-test -t <test> Benchmark all enabled frameworks subscribed to <test> and save the results

For /benchmark, always specify -f <framework>; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory — one table per framework on multi runs. A new benchmark comment while a run is in flight queues behind it (one deep) instead of cancelling it. For multi-framework PRs (dependency bumps, same-language refactors) prefer /benchmark-multiple, which runs everything in a single job and commits all saved results together, so no run overwrites another. --compare works on single-framework runs only.

What the deltas are measured against. By default, this framework's own results published on main - answering "did this change help?". When you are tuning a variant or a successor entry, --compare re-bases them on another entry instead:

/benchmark -f genhttp --compare genhttp-kestrel

The reply states which baseline it used, and profiles the other framework does not run show n/a rather than a delta.


Run benchmarks locally

You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.

./scripts/validate.sh <framework>
./scripts/benchmark-lite.sh <framework> baseline
./scripts/benchmark-lite.sh --load-threads 4 <framework>

Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk) are built as self-contained Docker images on first run.

jbr added 3 commits October 22, 2026 16:08
The standalone container runs on --network host whenever an entry
subscribes to a compose profile, and it was left running while the
gateway and production-stack stacks came up. An entry that also serves
h2/h3 keeps 8443 (and 8080), so the stack's edge failed to bind, shared
the port, or exited on the QUIC port, and the checks reached the
standalone server instead of the stack.
The compose server never came up: it builds a StaticFileHandler over
/data/static at startup, which the production-stack compose did not
mount, and trillium-static exits on a missing directory. Mount it as the
gateway stacks do, and have the edge wait for the server's UDS rather
than for the container to start.
@MDA2AV

MDA2AV commented Nov 4, 2026

Copy link
Copy Markdown
Owner

I need some time to take a look at this, thanks for the pr

MDA2AV added a commit that referenced this pull request Nov 13, 2026
…uthsvc on Rust 1.100, unbreak the aspnet gateway entries (#1767)

* validate.sh: stop the standalone container before the compose profiles

The standalone container runs on --network host whenever an entry
subscribes to a compose profile, and it was left running while the
gateway and production-stack stacks came up. An entry that also serves
h2/h3 keeps 8443 (and 8080), so the stack's edge failed to bind, shared
the port, or exited on the QUIC port, and the checks reached the
standalone server instead of the stack.

* authsvc: ship the real binary under cargo 1.100's build-dir layout

production-stack has failed on the bench host since the morning of
2026-11-13 on every entry (roadrunner-gateway #1757, morojs #1761,
tako-gateway #1762): the shared authsvc container exits 0 with no output,
the edge never sees it healthy, and the run ends with no results.

rust:1-alpine moved to Rust 1.100 on 2026-11-12. Its cargo keeps each
package's artifacts under target/release/build/<package>/<hash>/ rather
than target/release/deps/, so the cache-priming step's
`rm -rf target/release/deps/authsvc*` stopped matching anything. The
placeholder built from `fn main() {}` stayed up to date, the second
`cargo build` finished in 0.04s without compiling src/, and the image
shipped the placeholder: a 418 KB binary that exits 0 at once.

`cargo clean --release -p authsvc` removes the placeholder's artifacts
wherever cargo keeps them, and a grep for the server's startup line fails
the build if the placeholder is ever shipped again. Built against the
current rust:1-alpine (1.100.0): a 1.2 MB binary that logs
"authsvc listening on ...", answers /_health with 200 and an /_auth
without a token with 401.

* aspnet-minimal_nginx, aspnet-minimal_caddy: build against aspnet-minimal's Services and Types

Both Dockerfiles still COPY aspnet-minimal's AppData.cs and Models.cs,
which #1002 (2026-07-18) split into Services/ and Types/, so neither image
has built since. They were the only failures in #1615's validation sweep,
and their published results have not been refreshed since July.

Copy Handlers.cs with Services/ and Types/, register DatabaseService,
DatasetService, ItemService and the AppJsonContext resolver the way
aspnet-minimal's Program.cs does, and give the nginx entry's /api routes
DatabaseService's Postgres pool in place of AppData.PgDataSource. The
caddy entry gains the StackExchange.Redis reference DatabaseService needs;
it connects to nothing unless REDIS_URL is set, which gateway-h3 never
does.

---------

Co-authored-by: Jacob Rothstein <[email protected]>
@MDA2AV

MDA2AV commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Your validate.sh commit is on main now as part of #1767 (squashed, with you as co-author). Thanks for it. It turned out to be the fix for the cache-aside failure on #1761, where the leftover standalone morojs was answering part of the stack's traffic through SO_REUSEPORT. In #1767's run on the bench host, every entry it affects passed the gateway and production-stack checks, trillium and trillium-tuned included.

That leaves this PR conflicting only in frameworks/trillium/Cargo.lock and frameworks/trillium-tuned/Cargo.lock. A rebase onto main drops the validate.sh commit by itself ("patch contents already upstream") and leaves the two trillium commits.

# Conflicts:
#	frameworks/trillium-tuned/Cargo.lock
#	frameworks/trillium/Cargo.lock
@MDA2AV

MDA2AV commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Update: I merged main into the branch instead of rebasing (6083cf9), so your history stays as it is. Your dependency update was already on main through #1614 and #1640, so the lockfiles are main's, and the PR now shows only the production-stack change. Pull before your next push.

@MDA2AV

MDA2AV commented Nov 13, 2026

Copy link
Copy Markdown
Owner

/benchmark-multiple --save

@github-actions

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Results

Frameworks: 2 | Test: all tests

✅ trillium

Test Conn RPS Rate CPU/req p99 p99.9 CPU Mem Δ RPS Δ Mem
baseline 4096 1,321,859 — — — — 3390.2% 201MiB +0.4% +0.5%
pipelined 4096 3,941,520 — — — — 6293.7% 150MiB +0.2% -18.5%
limited-conn 4096 624,633 — — — — 1884.9% 138MiB +0.2% -3.5%
json-comp 4096 117,131 — — — — 6096.0% 807MiB +1.5% +1.5%
json-comp 16384 118,589 — — — — 5994.8% 784MiB +0.6% -2.9%
json-tls 4096 1,048,553 — — — — 5116.5% 428MiB +0.8% +18.2%
8gbit 512 48,970 0.9794 ⚠️ 46.59us 139.0us 6898.0us 228.8% 224MiB +0.8% -4.7%
static-tls 1024 683,651 — — — — 6024.6% 494MiB +2.3% +6.2%
async-db 1024 222,635 — — — — 3113.5% 201MiB ~0% +1.0%
fortunes 1024 201,737 — — — — 4689.4% 303MiB -5.7% -8.2%
baseline-h2 256 4,090,664 — — — — 6389.8% 327MiB +3.7% +2.8%
baseline-h2 1024 3,436,518 — — — — 6453.2% 601MiB -0.3% -0.7%
static-h2 256 21,260 — — — — 3918.3% 1.8GiB +0.2% ~0%
static-h2 1024 17,762 — — — — 4428.8% 2.6GiB ~0% +23.8%
baseline-h2c 256 4,069,118 — — — — 6427.4% 314MiB -5.0% +4.0%
baseline-h2c 1024 3,562,309 — — — — 6381.9% 476MiB -0.3% -42.5%
baseline-h2c 4096 3,481,833 — — — — 6110.7% 1.3GiB +4.1% +30.0%
json-h2c 1024 2,058,299 — — — — 6402.7% 962MiB +2.6% -21.7%
json-h2c 4096 1,840,089 — — — — 6037.0% 2.9GiB -3.7% +11.5%
baseline-h3 64 462,611 — — — — 2583.1% 297MiB +4.1% +8.8%
static-h3 64 220,372 — — — — 2382.2% 444MiB -2.1% -6.9%
gateway-64 512 68,957 — — — — 4162.7% 3.2GiB -4.7% +10.3%
gateway-64 1024 68,626 — — — — 4306.6% 5.5GiB +6.9% +3.8%
gateway-h3 64 76,398 — — — — 3948.0% 838MiB -1.8% +1.3%
gateway-h3 256 58,576 — — — — 2817.8% 1.9GiB -1.4% +11.8%
production-stack 256 15,537 — — — — 1103.0% 1.1GiB NEW NEW
production-stack 1024 46,799 — — — — 3441.1% 3.6GiB NEW NEW
unary-grpc 256 3,059,755 — — — — 6534.2% 322MiB +6.1% +14.2%
unary-grpc 1024 2,603,720 — — — — 6438.7% 574MiB +5.4% +4.4%
unary-grpc-tls 256 2,778,948 — — — — 6407.2% 462MiB +0.2% +29.1%
unary-grpc-tls 1024 2,451,838 — — — — 6454.1% 718MiB +4.1% +4.7%
echo-ws 512 1,144,309 — — — — 1743.3% 116MiB -0.7% +1.8%
echo-ws 4096 1,389,942 — — — — 2291.7% 236MiB -1.5% -11.9%
echo-ws 16384 1,372,263 — — — — 2368.5% 323MiB +1.4% -4.4%
echo-ws-pipeline 512 17,948,083 — — — — 2088.3% 107MiB +1.3% -10.8%
echo-ws-pipeline 4096 21,916,169 — — — — 2822.4% 211MiB +1.2% -11.7%
echo-ws-pipeline 16384 21,433,900 — — — — 2822.9% 305MiB +15.6% +22.0%
echo-ws-limited 512 613,064 — — — — 1572.2% 144MiB -0.7% -1.4%
echo-ws-limited 4096 629,297 — — — — 1544.0% 170MiB -0.4% +29.8%
latency-1m 1024 986,592 0.9866 23.60us 2603.0us 15708.0us 2482.1% 141MiB +1.7% +12.8%
latency-10k 1024 9,880 0.9880 24.30us 101.0us 805.0us 22.3% 139MiB -0.2% +0.7%
latency-500k-8cpu 1024 352,076 0.7042 ⚠️ 22.66us 5548032.0us 5679104.0us 786.6% 75MiB -0.2% -1.3%
async 32000 493,339 — — — — 2940.2% 1.3GiB -1.3% +8.3%

Composite score, main against this run, in the board's default view:

Family main this run Δ
HTTP/1.1 4257.7 (28 of 107) 4253.7 (28 of 107) -4.0
HTTP/2 1589.3 (9 of 40) 1588.8 (9 of 40) -0.5
HTTP/3 865.4 (7 of 19) 852.8 (8 of 19) -12.6
Gateway 601.8 (8 of 9) 732.1 (8 of 9) +130.3
gRPC 846.4 (6 of 13) 879.8 (6 of 13) +33.4
WebSocket 940.8 (24 of 38) 958.9 (24 of 38) +18.1

The results on main were measured in another session, so a few points up or down can be the machine and not the change.

✅ trillium-tuned

Test Conn RPS Rate CPU/req p99 p99.9 CPU Mem Δ RPS Δ Mem
baseline 4096 2,447,041 — — — — 6710.7% 433MiB ~0% +0.2%
pipelined 4096 3,696,010 — — — — 6321.8% 371MiB +2.8% -4.9%
limited-conn 4096 1,924,410 — — — — 6484.8% 524MiB -2.1% +3.1%
json-comp 4096 118,276 — — — — 6227.6% 1.0GiB +2.4% ~0%
json-comp 16384 99,867 — — — — 6122.4% 1.4GiB -4.0% -6.7%
json-tls 4096 1,321,822 — — — — 6542.0% 523MiB +1.3% -0.8%
8gbit 512 49,117 0.9823 47.52us 122.0us 1653.0us 216.9% 234MiB -0.4% -1.7%
static-tls 1024 784,391 — — — — 6461.7% 752MiB +0.8% -21.7%
async-db 1024 245,053 — — — — 3555.1% 280MiB +7.2% -4.4%
fortunes 1024 166,211 — — — — 4449.4% 375MiB +1.6% +0.3%
baseline-h2 256 5,385,385 — — — — 6300.7% 650MiB -1.3% -2.4%
baseline-h2 1024 4,062,132 — — — — 6264.5% 1.8GiB -3.8% +5.9%
static-h2 256 207,837 — — — — 6384.1% 14.2GiB -0.6% +2.9%
static-h2 1024 195,722 — — — — 6330.4% 17.5GiB +0.5% -11.6%
baseline-h2c 256 5,559,206 — — — — 6152.6% 593MiB -2.6% +3.7%
baseline-h2c 1024 4,247,239 — — — — 6273.3% 1.5GiB -3.7% -6.3%
baseline-h2c 4096 3,762,724 — — — — 6349.7% 5.5GiB -5.2% +1.9%
json-h2c 1024 2,006,958 — — — — 6369.4% 1.2GiB -1.3% ~0%
json-h2c 4096 1,843,704 — — — — 6242.4% 3.3GiB -1.5% -2.9%
baseline-h3 64 465,636 — — — — 2522.1% 372MiB +2.3% -1.1%
static-h3 64 233,215 — — — — 2443.6% 697MiB +3.1% +5.3%
gateway-64 512 69,226 — — — — 4033.5% 3.4GiB -2.7% ~0%
gateway-64 1024 68,584 — — — — 4337.0% 5.3GiB +1.4% -10.2%
gateway-h3 64 77,688 — — — — 3873.8% 869MiB +0.4% -0.9%
gateway-h3 256 58,631 — — — — 2846.2% 1.7GiB -1.7% -15.0%
production-stack 256 17,552 — — — — 1141.5% 1.3GiB NEW NEW
production-stack 1024 46,114 — — — — 3356.7% 3.9GiB NEW NEW
unary-grpc 256 3,823,615 — — — — 6342.6% 529MiB -2.7% -1.1%
unary-grpc 1024 3,064,698 — — — — 6349.6% 1.6GiB +0.8% +6.7%
unary-grpc-tls 256 3,728,130 — — — — 6148.5% 626MiB -2.4% +2.5%
unary-grpc-tls 1024 2,823,406 — — — — 6093.8% 1.8GiB -1.2% +28.6%
echo-ws 512 4,061,462 — — — — 6466.6% 173MiB +0.3% -2.3%
echo-ws 4096 4,121,639 — — — — 6446.0% 294MiB +1.0% +4.6%
echo-ws 16384 3,604,368 — — — — 6106.2% 780MiB -0.7% +3.2%
echo-ws-pipeline 512 52,631,128 — — — — 6495.3% 174MiB +1.2% +0.6%
echo-ws-pipeline 4096 53,286,433 — — — — 6636.3% 306MiB +3.1% +7.0%
echo-ws-pipeline 16384 46,915,465 — — — — 6209.9% 922MiB +0.1% +23.8%
echo-ws-limited 512 2,009,768 — — — — 5396.4% 200MiB +1.5% +3.1%
echo-ws-limited 4096 2,472,798 — — — — 6203.5% 424MiB +0.6% -0.7%
latency-1m 1024 997,593 0.9976 26.91us 205.0us 8204.0us 2946.8% 207MiB ~0% +0.5%
latency-10k 1024 9,983 0.9983 29.57us 118.0us 175.0us 28.3% 207MiB ~0% -1.9%
latency-500k-8cpu 1024 358,751 0.7175 ⚠️ 22.24us 6318080.0us 6735872.0us 785.3% 101MiB +1.8% +26.2%
async 32000 1,973,380 — — — — 6429.4% 1.4GiB -2.0% ~0%

Composite score, main against this run, in the board's default view:

Family main this run Δ
HTTP/1.1 6035.7 (11 of 107) 6021.4 (11 of 107) -14.4
HTTP/2 1978.6 (7 of 40) 1939.2 (7 of 40) -39.4
HTTP/3 871.2 (6 of 19) 897.7 (6 of 19) +26.5
Gateway 605.6 (7 of 9) 738.9 (7 of 9) +133.3
gRPC 1101.8 (5 of 13) 1085.3 (5 of 13) -16.5
WebSocket 2830.4 (3 of 38) 2856.0 (1 of 38) +25.7

The results on main were measured in another session, so a few points up or down can be the machine and not the change.

@MDA2AV
MDA2AV merged commit 40b78bb into MDA2AV:main Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants