[frankmcsherry]

Diagnostics for the DDIR server as ordinary DDIR collections. They're read with peek and tail like any answer, instead of through one-off commands (#989's profile, #1006's sizes) or the WebSocket console. Design note: https://claude.ai/artifact/VoTk6wN3XzR3LfXYkz1w1S

How it works. With DDIR_DIAGNOSTICS=1, each worker registers timely's "timely" logger, DD's "differential/arrange" logger and rendering's "ddir/labels" logger before building any dataflow. At each tick, the feeder (interactive/src/diag.rs) folds the buffered events into per-operator state, then gives each installed diag.* source its changes at the epoch being closed. Programs import the sources the way they import generated sources such as iota:N. Each count carries a magnitude:

Source	Row	Count
diag.operator	((worker, op), (program, statement, name, address))	+1 when created; −1 at shutdown or when its dataflow is dropped
diag.time	((worker, op), ())	the operator's own nanoseconds scheduled, while it lives
diag.time_retired	((worker, program, statement, name), ())	at shutdown, an operator's total moves here from diag.time
diag.records	((worker, op), ())	updates held, from batch, merge and drop events

Views. interactive/server/diag.ddp is installed as program diag. Its exports are diag.time_by_statement, diag.time_by_program (live plus retired time), diag.records_by_statement and diag.operators_by_program. Every view leaves out program diag itself.

Retention stays bounded. Everything keyed by an operator is retracted when the operator, or its dataflow, is dropped. Time moves into diag.time_retired, so a program installed and dropped every epoch still adds up. That state is bounded by the number of distinct (program, statement, name) triples. Over 300 install/drop cycles, the diagnostics' own held updates stayed flat at about 1,640.

From #989: its statement labels. Lowering records which statement built each item, kept aligned through optimize, and rendering logs which operator ids each item became. Export arrangements are now labelled export <name>. #989's profile command is superseded.

Held back: #1006's heap-byte accounting walks corgi's value layout from outside corgi, so it is left out to be owned by corgi, either as a corgi heap_bytes or as sizes carried in DD's batch events. diag.trace_bytes waits for it.

Agreement with them:

• time on the txn resolver: 361.5 ms against profile's 361.8 ms, with every statement's share within 0.3 points;
• updates held: 52,737,336 against sizes' 52,737,206, the differences being one tick of lag;

Overhead with DDIR_DIAGNOSTICS=1; it's off by default:

• the loggers and feeder alone cost about 4%;
• the views program costs a fixed ~0.5 ms per tick. On the txn bench that's +27–32% at batch 1,000 (about 2 ms cycles) and +5.5% at batch 10,000.
• On LDBC SF0.1 it's +1.5% in all.

Installing the views on demand is a possible follow-up.

What changes:

• DDIR_DIAGNOSTICS=1 now feeds diag.* and installs diag.ddp, instead of starting the WebSocket diagnostics dataflow, and the server no longer depends on the diagnostics crate.
• Until the console reads the diag.* traces, it loses its DDIR_DIAG_PORT listener, the live wall-clock view during a long tick, channel and message counts, and batcher stats.

Known limits:

• Events during a tick appear at the next one.
• Dataflow scaffolding, about 5–10% of a program's time, has no statement.
• Names are byte lists, so text peek shows them as numbers; frames decode them.
• Not yet included: diag.batcher, and any byte accounting (see above).

Checks:

• cargo test -p interactive -p ddir-server, including tests/diag.rs and tests/labels.rs, on master-next fceb62f: 196 passed, none failed.
• Clippy reports nothing on new lines.
• All 16 LDBC CI steps pass with diagnostics off, including authored/check.py.

🤖 Generated with Claude Code

https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o