Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ The reply states which baseline it used, and profiles the other framework does n
| Category | Profiles | Description |
|----------|----------|-------------|
| Connection | `baseline`, `pipelined` *, `limited-conn` | Mixed GET/POST with query parsing (4K conns), 16× batched pipelining (reference-only, shown faded, excluded from the composite score), short-lived connections that close after 10 requests |
| Concurrency | `async` | `GET /delay/{ms}` with the wait named in the route, 32K held connections. Isolates what a framework does while a request is pending, with no database or I/O in the way. Scored for frameworks and engines, not for infrastructure, since the spec settled at a flat 10 ms over 32K connections |
| Concurrency | `async`, `connections-1m` *, `connections-1m-zrk` * | `GET /delay/{ms}` with the wait named in the route, 32K held connections. Isolates what a framework does while a request is pending, with no database or I/O in the way. Scored for frameworks and engines, not for infrastructure, since the spec settled at a flat 10 ms over 32K connections. `connections-1m` holds one million keep-alive connections at once with 250K req/s paced across them (one request per connection every 4 s), spread over 32 loopback addresses: opt-in and reference-only, it measures memory per held connection, CPU and tail latency, and needs a listener on 0.0.0.0 or [::]. `connections-1m-zrk` is the same test with zrk 3.0.2 as the load generator instead of gcannon |
| Efficiency | `latency-1m`, `latency-10k`, `latency-500k-8cpu` * | One million req/s offered at a fixed rate with zrk; the metric is CPU spent per request, read exactly from the container's cgroup rather than sampled. Everyone who finishes serves the same million, so what separates them is the bill. Scored on its own 0-100 formula rather than on rps, since the rate is pinned. `latency-10k` is the identical setup - same cores, connections and endpoint - at a near-idle 10K req/s, so what it reports is the standing cost of being a running server rather than of serving load, and it is scored the same way. Both score CPU at 0.50, p99 at 0.25 and mean latency at 0.25, the mean being the queueing signal: ten times the best mean is worth nothing. `latency-500k-8cpu` pins 500K req/s with the server on four cores plus their SMT siblings, so the load sits near saturation and the queue a server builds shows in its mean; reference-only while the first entries are measured |
| Workload | `json-comp`, `json-tls`, `8gbit`, `static-tls` * | JSON serialization, gzip/brotli compression, HTTP/1.1 over TLS, a 10 KB body echoed over TLS at a pinned 50K req/s so ingest and egress load together - scored on the same 0-100 formula as the latency profiles, since its rate is pinned too, 20-file static asset serving over TLS (reference-only for frameworks and engines, scored for infrastructure) |
| Database | `async-db` * | Async Postgres sequential scan. Reference-only: the database and its driver dominate it far more than the framework does, so it is measured and shown but no longer decides the ranking |
Expand Down
4 changes: 3 additions & 1 deletion docker/gcannon.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,10 @@ ARG GCANNON_REF=main
# stale git-clone layer. Without this, edits pushed to the gcannon repo
# are invisible until the user manually runs --no-cache.
ARG CACHE_BUST=0
# libssl-dev: gcannon 0.7.0 and later do the TLS handshake with OpenSSL (the
# runtime image's libssl3 serves the binary), and fail to build without it.
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc libc6-dev make git ca-certificates && rm -rf /var/lib/apt/lists/*
gcc libc6-dev make git ca-certificates libssl-dev && rm -rf /var/lib/apt/lists/*
WORKDIR /deps
RUN git clone --branch liburing-2.9 --depth 1 https://github.com/axboe/liburing.git && \
cd liburing && ./configure --prefix=/usr && make -j"$(nproc)" -C src && make install -C src
Expand Down
3 changes: 3 additions & 0 deletions docker/zrk.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@
# across entries and across rounds, so a generator that drifts under it
# invalidates every comparison it was measured against, including old ones.
# Bumping the version is a deliberate act that re-baselines the profile.
#
# connections-1m-zrk builds this file a second time as zrk:3.0.2, passing
# ZRK_VERSION and ZRK_SHA256 (tools/zrk.sh), and leaves zrk:local as it is.
FROM ubuntu:24.04 AS build
ARG ZRK_VERSION=2.3.0
ARG ZRK_SHA256=ab6f1c1ce34ce73f52afd106419d15e5c494bd1cedd475afa56babb85b09a816
Expand Down
2 changes: 2 additions & 0 deletions frameworks/actix/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
"baseline",
"async",
"latency-1m",
"connections-1m",
"connections-1m-zrk",
"latency-10k",
"latency-500k-8cpu",
"pipelined",
Expand Down
2 changes: 2 additions & 0 deletions frameworks/aspnet-minimal/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
"baseline",
"pipelined",
"limited-conn", "async", "latency-1m", "latency-10k", "latency-500k-8cpu",
"connections-1m",
"connections-1m-zrk",
"json-comp",
"json-tls",
"8gbit",
Expand Down
2 changes: 2 additions & 0 deletions frameworks/bun/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
"baseline",
"async",
"latency-1m", "latency-10k", "latency-500k-8cpu",
"connections-1m",
"connections-1m-zrk",
"pipelined",
"limited-conn",
"json-comp",
Expand Down
2 changes: 2 additions & 0 deletions frameworks/fib/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
"limited-conn",
"async",
"latency-1m",
"connections-1m",
"connections-1m-zrk",
"latency-10k",
"latency-500k-8cpu",
"json-comp",
Expand Down
2 changes: 1 addition & 1 deletion frameworks/genhttp-ioxide/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ FROM mcr.microsoft.com/dotnet/sdk:11.0 AS build
# GenHTTP built from source: the branch of Kaliumhexacyanoferrat/GenHTTP#953
# (update-ioxide-0.20.396) rather than the 11.3.0 release. Pinned to one commit,
# because a build on cached layers would otherwise keep whichever it cloned first.
ARG GENHTTP_COMMIT=28b0cd107bb36321408a3a1f892f3bb45225e9d7
ARG GENHTTP_COMMIT=5967f5c6bf64d3d67a0eed7a669fa2f8c846dd02
RUN git init -q /genhttp \
&& git -C /genhttp fetch -q --depth 1 https://github.com/Kaliumhexacyanoferrat/GenHTTP.git "$GENHTTP_COMMIT" \
&& git -C /genhttp checkout -q FETCH_HEAD
Expand Down
2 changes: 2 additions & 0 deletions frameworks/genhttp-ioxide/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
"baseline",
"async",
"latency-1m",
"connections-1m",
"connections-1m-zrk",
"latency-10k",
"latency-500k-8cpu",
"pipelined",
Expand Down
2 changes: 2 additions & 0 deletions frameworks/node/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
"enabled": true,
"tests": [
"baseline", "latency-1m", "latency-10k",
"connections-1m",
"connections-1m-zrk",
"pipelined",
"limited-conn",
"json-comp",
Expand Down
74 changes: 69 additions & 5 deletions scripts/benchmark.sh
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ FRAMEWORK="$FRAMEWORK_ARG"
# and any combination thereof.
_has_isolated_test=false
for t in baseline pipelined limited-conn json-comp json-tls upload \
static-tls async-db async latency-1m latency-10k latency-500k-8cpu \
static-tls async-db async latency-1m latency-10k latency-500k-8cpu connections-1m connections-1m-zrk \
baseline-h2 static-h2 baseline-h2c json-h2c \
baseline-h3 static-h3 \
unary-grpc unary-grpc-tls \
Expand Down Expand Up @@ -196,10 +196,40 @@ if { framework_subscribes_to "latency-1m" || framework_subscribes_to "latency-10
info "zrk: docker mode ($ZRK_IMAGE)"
fi

# connections-1m-zrk runs its own zrk (ZRK_CONN1M_VERSION, tools/zrk.sh): the
# same Dockerfile at another version, so the pinned image above is untouched.
# A failed build is not fatal here: the fan-out asks the image for --version
# and reports the profile as not run, and the other profiles go ahead.
if framework_subscribes_to "connections-1m-zrk" \
&& [[ " ${profiles_to_run[*]} " == *" connections-1m-zrk "* ]] \
&& [ -z "${ZRK_CONN1M_CMD:-}" ]; then
if _zrk_at_least "$ZRK" "$ZRK_CONN1M_VERSION"; then
ZRK_CONN1M_CMD="$ZRK"
else
if ! docker image inspect "$ZRK_CONN1M_IMAGE" >/dev/null 2>&1; then
info "building $ZRK_CONN1M_IMAGE (zrk $ZRK_CONN1M_VERSION) for connections-1m-zrk"
docker build -t "$ZRK_CONN1M_IMAGE" \
--build-arg ZRK_VERSION="$ZRK_CONN1M_VERSION" \
--build-arg ZRK_SHA256="$ZRK_CONN1M_SHA256" \
-f "$ROOT_DIR/docker/zrk.Dockerfile" "$ROOT_DIR/docker" \
|| warn "$ZRK_CONN1M_IMAGE build failed - connections-1m-zrk will not run"
fi
ZRK_CONN1M_CMD="docker run --rm --network host --cpuset-cpus=$GCANNON_CPUS --security-opt seccomp=unconfined --ulimit memlock=-1:-1 --ulimit nofile=1048576:1048576 $ZRK_CONN1M_IMAGE"
fi
info "zrk for connections-1m-zrk: $ZRK_CONN1M_CMD"
fi

# ── System tuning — NOW, after all image builds are complete ───────────────

if [ "${SKIP_TUNE:-}" != "true" ]; then
system_tune
# Only when this run will hold a million connections; see system.sh.
if { framework_subscribes_to "connections-1m" \
&& [[ " ${profiles_to_run[*]} " == *" connections-1m "* ]]; } \
|| { framework_subscribes_to "connections-1m-zrk" \
&& [[ " ${profiles_to_run[*]} " == *" connections-1m-zrk "* ]]; }; then
system_tune_connections_1m
fi
else
info "skipping system tuning as requested (SKIP_TUNE=true)"
fi
Expand Down Expand Up @@ -287,7 +317,7 @@ run_one() {
# three runs, which is not known until all three have happened. Every run is
# recorded here and the choice is made below.
local _mdir=""
case "$endpoint" in latency-1m|latency-10k|latency-500k-8cpu|8gbit) _mdir=$(mktemp -d) ;; esac
case "$endpoint" in latency-1m|latency-10k|latency-500k-8cpu|8gbit|connections-1m|connections-1m-zrk) _mdir=$(mktemp -d) ;; esac

BEST_M=()
local run
Expand Down Expand Up @@ -323,6 +353,14 @@ run_one() {
[[ "$line" == *=* ]] && m["${line%%=*}"]="${line#*=}"
done < <("${tool//-/_}_parse" "$endpoint" "$output")

# connections-1m reports the server's CPU over gcannon's measured
# window only (scripts/conn_hold.py), connections-1m-zrk over the fan-
# out's (scripts/zrk_fanout.py). The reading above spans the whole run,
# a million connects and closes included, so the window replaces it.
if [ -n "${m[cpu_usec_window]:-}" ]; then
CPU_ACCT_USEC="${m[cpu_usec_window]}"
fi

local rps_int=${m[rps]:-0}

# Best-of-N keeps the fastest run — except on a fixed-rate profile,
Expand Down Expand Up @@ -369,7 +407,11 @@ run_one() {
# for. tcp_tw_reuse lets those be recycled, but at 48K of a 64.5K-port
# range the connect path starts scanning and the ramp lands inside the
# measured window. Two seconds is plenty below that.
if [ "$CONNS" -ge 32768 ]; then sleep 15; else sleep 2; fi
# A million at once is longer again: the server has a million closes to
# work through before the next run starts connecting.
if [ "$CONNS" -ge 500000 ]; then sleep 30
elif [ "$CONNS" -ge 32768 ]; then sleep 15
else sleep 2; fi
done

# ── Latency-1M: pick the run by score, not by any single metric ─────
Expand Down Expand Up @@ -405,7 +447,7 @@ run_one() {
echo ""; echo "=== Best: ${best_rps} req/s (CPU: $best_cpu, Mem: $best_mem) ==="

# The fixed-rate profiles' headline number, and the check that it counts.
if [ "$endpoint" = "latency-1m" ] || [ "$endpoint" = "latency-10k" ] || [ "$endpoint" = "latency-500k-8cpu" ] || [ "$endpoint" = "8gbit" ]; then
if [ "$endpoint" = "latency-1m" ] || [ "$endpoint" = "latency-10k" ] || [ "$endpoint" = "latency-500k-8cpu" ] || [ "$endpoint" = "8gbit" ] || conn1m_profile "$endpoint"; then
# Two different failures that used to print the same line. "No CPU
# reading" is a cgroup problem; "no requests" is a generator or server
# problem, and saying the former when it is the latter sent the first
Expand Down Expand Up @@ -440,6 +482,22 @@ run_one() {
fi
fi

# connections-1m: whether the million was actually held. The rate alone
# cannot say it - a connection that never opened sends nothing, which reads
# as a rate shortfall - so the fan-out counts the server's established
# sockets after the window as well.
if conn1m_profile "$endpoint"; then
if [ -n "${BEST_M[hold_error]:-}" ]; then
warn "$endpoint did not run: ${BEST_M[hold_error]}"
else
echo "=== Connections: ${BEST_M[connections_open]:-n/a} open of $CONNS | connect errors in the window ${BEST_M[connect_errors]:-n/a} ==="
if [ -n "${BEST_M[connections_open]:-}" ] \
&& [ "${BEST_M[connections_open]}" -lt $(( CONNS * 99 / 100 )) ] 2>/dev/null; then
warn "held ${BEST_M[connections_open]} of $CONNS connections through the window"
fi
fi
fi

# 8gbit's generator reports only the bytes it read back, so that figure
# is the download half of the echo; the request body is a fixed size
# (ZRK_ECHO_BODY_BYTES in scripts/lib/tools/zrk.sh - keep the two in step),
Expand Down Expand Up @@ -532,7 +590,7 @@ save_result() {
# not the load, so the reason ships with the row rather than being inferred
# from an rps that looks merely slow.
local eff_extra=""
if [ "$profile" = "latency-1m" ] || [ "$profile" = "latency-10k" ] || [ "$profile" = "latency-500k-8cpu" ] || [ "$profile" = "8gbit" ]; then
if [ "$profile" = "latency-1m" ] || [ "$profile" = "latency-10k" ] || [ "$profile" = "latency-500k-8cpu" ] || [ "$profile" = "8gbit" ] || conn1m_profile "$profile"; then
local _eff_reqs=${BEST_M[status_2xx]:-0}
local _eff_per_req="null"
if [ -n "$best_cpu_usec" ] && [ "$_eff_reqs" -gt 0 ] 2>/dev/null; then
Expand All @@ -544,6 +602,12 @@ save_result() {
\"target_rate\": ${BEST_M[target_rate]:-0},
\"rate_ratio\": ${best_rate_ratio:-0},
\"p99_9_latency\": \"${BEST_M[p999_lat]:-}\""
# How much of the million was held, next to what it cost to hold it.
if conn1m_profile "$profile"; then
eff_extra="$eff_extra,
\"connections_open\": ${BEST_M[connections_open]:-null},
\"window_connect_errors\": ${BEST_M[connect_errors]:-null}"
fi
fi

local cpu_extra=""
Expand Down
Loading
Loading