You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current tests show how fast a framework is at moderate concurrency. Many production systems also hold very large numbers of mostly-idle connections: IM/chat, push notification gateways, IoT, WebSocket services, long-polling and API gateways. At that scale, the deciding factors are usually memory per connection, CPU overhead and tail latency, not peak RPS. Frameworks that look similar at a few thousand connections can differ by an order of magnitude at 1M.
Adding large-connection tests would make HttpArena a much stronger reference when choosing a framework for these workloads.
Proposal
Add a test category that runs each framework at 100k, 500k and 1M concurrent connections, for example:
Idle hold: open N keep-alive connections and hold them idle. Measure server RSS and memory per connection.
Sparse activity: N connections are open, but only a small fraction (e.g. 1–5%) send requests at any moment. Measure throughput, p50/p99/p999 latency, CPU usage and RSS.
Full load (optional): all N connections send requests. Measure throughput, latency and errors/timeouts.
Ramp-up (optional): measure how long it takes to establish N connections and how many fail.
The same idea could cover WebSocket at high connection counts as well.
Suggested metrics
Peak and steady-state RSS, and memory per connection
CPU utilization
Throughput (RPS)
Latency percentiles (p50 / p99 / p999)
Connection errors and timeouts
Implementation notes
Port limits: one client IP can open only about 64k connections to one server IP:port. The load generator needs multiple source IPs (IP aliases or several client machines) or multiple server ports.
System tuning: the server and client need raised ulimit -n, fs.file-max, net.core.somaxconn, net.ipv4.ip_local_port_range and TCP memory settings. Documenting these in the test setup would make results reproducible.
Fewer frameworks: frameworks could opt in to this category, so it doesn't add much to the total benchmark time.
Reacted by Diogo Martins, Andreas Nägeli and Nigro Simone
I'll check what can be done, 1 million is likely not possible when we run both load gen and server on the same box, I am inclined to 200k, maybe push a bit more.
This test is likely to be unscored though, this is mostly a edge type server test case, most frameworks here are not built to handle high connection count
I'll check what can be done, 1 million is likely not possible when we run both load gen and server on the same box, I am inclined to 200k, maybe push a bit more.
We can use multiple ports, for example, each server can listen on 50 ports.
This test is likely to be unscored though, this is mostly a edge type server test case, most frameworks here are not built to handle high connection count
Some frameworks would struggle to handle such a large number of connections and should perhaps be categorized separately. In fact, the current engine, infra, and gateway classifications might not be very clear.
I like the idea - and agree that servers are mostly idle having tons of keep alive connections open. The scoring would be interesting though - just memory?
I like the idea - and agree that servers are mostly idle having tons of keep alive connections open. The scoring would be interesting though - just memory?
@Kaliumhexacyanoferrat My own tests ranked TPS, CPU, and memory efficiency ratios(Energy Efficiency Ratio), as shown in the image: CPU EER and MEM EER items. However, I don't have a good idea for the overall score yet; you guys are more professional in this area.
Activity
MDA2AV commentedon Nov 13, 2026
I'll check what can be done, 1 million is likely not possible when we run both load gen and server on the same box, I am inclined to 200k, maybe push a bit more.
This test is likely to be unscored though, this is mostly a edge type server test case, most frameworks here are not built to handle high connection count
lesismal commentedon Nov 14, 2026
We can use multiple ports, for example, each server can listen on 50 ports.
That's how I do it in my own benchmark library:
https://github.com/lesismal/go-websocket-benchmark/blob/main/config/config.go#L73
Some frameworks would struggle to handle such a large number of connections and should perhaps be categorized separately. In fact, the current engine, infra, and gateway classifications might not be very clear.
Kaliumhexacyanoferrat commentedon Nov 14, 2026
I like the idea - and agree that servers are mostly idle having tons of keep alive connections open. The scoring would be interesting though - just memory?
lesismal commentedon Nov 14, 2026
@Kaliumhexacyanoferrat My own tests ranked TPS, CPU, and memory efficiency ratios(Energy Efficiency Ratio), as shown in the image: CPU EER and MEM EER items. However, I don't have a good idea for the overall score yet; you guys are more professional in this area.
github actions runs the benchmarks but only 2 CPU for each server in github action.
https://github.com/lesismal/go-websocket-benchmark/actions/runs/42801023661
MDA2AV commentedon Nov 16, 2026
first draft #1815