Plan the voice demo's live call timeline

quest/README.md

@@ -16,4 +16,4 @@ its scope, until it is scheduled.
 ## Required
 
 - [m0: immediate priorities](/quest/m0/README.md) - hosted worklets once a release ships assets(), lean agent context, and checks that pass locally
-- [m1: next](/quest/m1/README.md) - the Voice AI demo page and its live buffer view
+- [m1: next](/quest/m1/README.md) - the Voice AI demo page and its live call timeline

quest/m1/README.md

@@ -2,11 +2,11 @@
 
 ## Goal
 
-The demos after m0: a Voice AI demo you can talk to, which shows why MoQ beats
-WebRTC for voice agents.
+The demos after m0: a Voice AI demo you can talk to, which shows live how MoQ
+moves and buffers the call and why it beats WebRTC for voice agents.
 
 ## Required
 
 - [An @moq/watch release ships per-frame arrivals](/quest/m1/watch-arrivals-release.md) - condition: the reply-side data is on npm
-- [The voice agent publishes voice.json](/quest/m1/voice-timeline-track.md) - condition: word timings and prompt-side arrivals on wtf
-- [Voice buffers under the words](/quest/m1/voice-buffers.md) - the live transcript draws each word's audio frames, marking the ones late for a typical WebRTC jitter buffer
+- [The voice agent publishes the full voice.json](/quest/m1/voice-timeline-track.md) - condition: the agent's side of the call, timed, on wtf
+- [A live timeline of the voice call](/quest/m1/voice-buffers.md) - every frame, buffer, VAD decision, and model step on one axis, marking frames late for a typical WebRTC jitter buffer

quest/m1/voice-buffers.md

@@ -1,40 +1,65 @@
-# [M] Voice buffers under the words
+# [L] A live timeline of the voice call
 
 ## Goal
 
-Under each word of the live `/voice` transcript, a strip draws that word's
-audio frames from real measurements: when each arrived against when it plays.
-A frame that arrived later than a typical WebRTC jitter buffer waits turns
-red: an estimate from the MoQ arrivals, not a measured WebRTC receiver. Both
-directions: your prompt as the agent received it, and the agent's reply as
-your browser received it. Nothing is simulated or injected.
+The `/voice` demo draws the whole call on one live, scrolling timeline, from
+real measurements: your mic frames as captured, those frames reaching the
+agent, its VAD and turn decisions, the words STT recognizes and when, the LLM
+and TTS working, each reply frame published, in flight, and buffered in your
+browser, and the playhead. A reader watches packets cross before the AI
+recognizes them as words, sees how far ahead MoQ holds the reply in memory, and
+can tell at a glance where a slow reply spent its time, which answers "is the
+delay MoQ?".
+
+Frames that arrived later than a typical WebRTC jitter buffer waits turn red:
+an estimate from the MoQ arrivals, not a measured WebRTC receiver. Nothing is
+simulated or injected.
 
 ## Plan
 
-Decided 2026-10-09:
-
-- **Real data only.** No "network blip" button and no injected loss. On a
-  clean network the strips are all on time, and the reply strip still shows
-  how far ahead the agent sent its audio.
-- **A WebRTC deadline line, computed from the real arrivals.** A frame is
-  "late for WebRTC" when its arrival spread exceeds a typical WebRTC jitter
-  target. Measure lateness as arrival minus timestamp, relative to the
-  smallest one seen, so the sender's and receiver's clocks never need to
-  agree. Choose and cite the target when implementing.
-- **Draw it like `src/components/voice/`.** Reuse its segment colors and
-  word layout on a sliding window of live, append-only data, in place of its
-  fixed axis and simulated `FLIGHT`.
-- **Reply side** reads `@moq/watch`'s per-frame `arrivals()` handle on the
-  audio decoder and maps frames to words with the agent's TTS word timings.
-  It yields each frame's `{timestamp, arrival, late}` and a gap item
-  (`timestamp` and `duration` in `Time.Milli`) where media will never
-  arrive. Its queue is unbounded, so the page keeps its own window. There is
-  no gap at the join: the strip starts on the first frame read.
-- **Prompt side** reads the agent's `voice.json` track on the response
-  broadcast: word timings from STT and the arrival time of each request
-  audio frame.
+Decided 2026-10-23:
+
+- **One shared wall-clock axis**, not per-direction panels or strips under
+  words: every lane on the same time, so a frame's capture, arrival, and
+  recognition line up vertically. A sliding window around now; buffered reply
+  audio sits to the right of the playhead.
+- **All at once.** Build the page after the agent-side and reply-side data
+  exist, rather than shipping lanes piecemeal.
+- **Real data only**, and keep the WebRTC mark: a frame is late for WebRTC
+  when its arrival spread exceeds a typical WebRTC jitter target. Measure
+  lateness as arrival minus media timestamp, relative to the smallest seen,
+  so it needs no clock agreement. Choose and cite the target when
+  implementing.
+- **Draw it like the post.** Reuse `src/components/voice/`'s colors and
+  vocabulary (buffered, consumed, waiting, lost; packets crossing a network
+  band; words over their audio) so the live view reads as the post's diagrams
+  come alive.
+
+Sources, as of planning:
+
+- Browser, available now: `@moq/publish`'s captured audio frames;
+  `@moq/watch`'s audio `buffered` ranges and playback `timestamp`; the
+  connection's PROBE RTT; RTVI turn events and per-service `metrics` on the
+  transcript track.
+- Reply-side arrivals: `@moq/watch`'s per-frame `arrivals()`, which also
+  yields a gap item where media will never arrive. Draw a frame as lost only
+  from such a gap; until then a missing frame is in flight.
+- Agent side: `voice.json` on the response broadcast, extended to carry
+  everything that happens on the agent with its time.
+
+The browser cannot see its own QUIC send buffer. Show "captured, not yet at
+the agent" from the agent's arrivals instead.
+
+Placing agent events on the browser's axis needs a clock offset. Both
+directions carry media timestamps with send and arrival times, so an
+NTP-style estimate from the minimum transit each way is a good starting point.
+The two minimum transits should sum to about the PROBE RTT, and their
+asymmetry bounds the error; show agent events only once that bound is small.
+
+Build on [moq-dev/moq.dev#194](https://github.com/moq-dev/moq.dev/pull/194)
+(transcript auto-scroll, steadier cat) if it is still open.
 
 ## Required
 
 - [An @moq/watch release ships per-frame arrivals](/quest/m1/watch-arrivals-release.md) - the reply-side data
-- [The voice agent publishes voice.json](/quest/m1/voice-timeline-track.md) - word timings and the prompt-side data
+- [The voice agent publishes the full voice.json](/quest/m1/voice-timeline-track.md) - the agent's side of the timeline

quest/m1/voice-timeline-track.md

@@ -1,19 +1,24 @@
-# [XS] The voice agent publishes voice.json
+# [XS] The voice agent publishes the full voice.json
 
 ## Goal
 
 The moq.pro demo agent on wtf publishes a `voice.json` track on each session's
-response broadcast, carrying word timings for both directions and the arrival
-time of each request audio frame.
+response broadcast carrying the agent's whole side of the call, each with its
+time: request frame arrivals, VAD and turn decisions, word spans and when STT
+recognized them, LLM and TTS activity, and each reply frame as published.
 
 This quest tracks a condition outside the repository. When it holds, delete
 this quest and every `Required` entry that links it.
 
 ## Plan
 
-moq.pro tracks the track as
-[its own quest](https://github.com/moq-dev/moq.pro/blob/main/quest/m2/voice-timeline.md),
-which owns its schema. It merged in
-[moq-dev/moq.pro#2319](https://github.com/moq-dev/moq.pro/pull/2319), but wtf
-has not been deployed since. Check by subscribing to it during a staging
-session.
+moq.pro owns the schema and the Pipecat fork hooks behind it:
+
+- [The demo agent publishes voice.json](https://github.com/moq-dev/moq.pro/blob/main/quest/m2/voice-timeline.md) -
+  version 1 (arrivals and word spans) merged in
+  [moq-dev/moq.pro#2319](https://github.com/moq-dev/moq.pro/pull/2319) and
+  awaits its wtf proof
+- [voice.json carries the agent's whole pipeline](https://github.com/moq-dev/moq.pro/blob/main/quest/m2/voice-timeline-pipeline.md) -
+  the rest
+
+Check by subscribing to the track during a staging session.