Skip to content

Idle-time auto-compaction, timed against the provider prompt-cache window (Claude Code pattern) #99

Open

Description

@terenceponce

mise ~/.config/mise/config.toml tools: [email protected]
Owner ask (Oct 30, from Claude Code's behavior): when the agent has been IDLE for a while, auto-compact BEFORE the provider's prompt cache expires - so the next turn starts small and the compaction cost never lands on the user's critical path.

Why this is a natural fit here

  • The agent is an event-driven GenServer - an idle timer (Process.send_after, cancelled/rescheduled on user activity, suggester's epoch-guard pattern) is architecturally clean and does NOT touch ADR 0006's render discipline (agent-side, not render-side).
  • /compact (issue /compact: trigger compaction manually from the command surface #92) just built the machinery: the shared eviction pass, the feedback entry, the idle-only guard, the no-op-below-trigger decision. This feature is a timer-driven caller of that same pass.
  • The accounting layer (ADR 0022) anchors on the provider usage frame - where available it can read cached-token counts to OBSERVE real cache behavior per provider instead of hardcoding TTL assumptions.

What it buys TODAY vs at STAGE 2 (be honest in the ADR)

  • Today (compaction = local drop-oldest, no API call): the real present-day win is moving the eviction cost off the user's next send - which matters because of the recorded O(n^2) eviction follow-up (10.1s at 20k messages): a huge idle session currently stalls its first send on prompt-time compaction. Idle-time eviction pays that stall while nobody is watching. The request bytes are identical either way - so today the feature is about latency placement, not token cost.
  • Stage 2 (compaction = LLM summarization, already reserved inert in settings): the full Claude Code economics arrive - the summarization is an API call whose latency and input cost land on the critical path; running it during idle removes the latency, and TIMING it before the cache expires makes the summarize request itself ride the warm prefix cache (cheap input). The cache-expiry timing only becomes material when compaction is an API call.

Design questions the lane settles (record in an ADR 0022 amendment)

  1. Idle threshold: fixed (e.g. 5 min) vs provider-aware (usage-frame cached_tokens observed decay) - prefer observable over assumed where the frame exposes it.
  2. Trigger interaction: idle compaction fires only at/above the same 80% trigger (below it there is nothing to gain and context to lose) - confirm or argue.
  3. Activity definition: what resets the timer (composer input, any key, command use) - suggester's epoch pattern.
  4. Persistence: eviction + transcript recording happen exactly as /compact does them (ADR 0014 §5 semantics); a crash mid-idle-compaction must leave a loadable transcript.
  5. The O(n^2) eviction loop (banked follow-up m1) is IN SCOPE here if the idle path makes it hot - a huge session compacting on a timer is exactly the case that hits it.

Done when

Driven e2e: an over-threshold session left idle auto-compacts within the recorded threshold with the standard feedback entry, WITHOUT any render/tick activity while idle (probe: idle renders stay 0, agent timer is not a renderer); user activity during the window cancels/reschedules (epoch-proven); below-threshold idle sessions never compact; the first send after idle compaction is stall-free where the pre-idle path stalled (measure on a large fixture); ADR 0022 amendment records threshold policy + trigger interaction + today-vs-stage-2 economics; mix ci green with pasted exit + counts.

Scheduling note

Filed for the post-v0.3.0 era (the cut is locked to the current queue: #67 stage 2 -> #75 images -> prep -> tag). Pairs naturally with Stage-2 summarization design - if that lane is framed first, this trigger should be designed into it rather than bolted after.

Owner refinement (Oct 30, from watching Claude Code)

The observed behavior: Claude's prompt cache lasts about an HOUR, and the compaction fires when the agent has been idle for ALMOST that hour - so the summarization request rides the still-warm cache before it expires. The framing that matters: harvest the cache before it's wasted. The prefix has already been paid for; letting the TTL lapse unused throws that away, and a post-expiry compaction then pays full input price on the whole history. An idle timer tuned to (TTL - margin) converts otherwise-expiring cache life into a cheap compaction.

Design implication: the idle threshold is not a fixed comfort number - it is DERIVED from the provider's cache TTL (for Claude: the 1-hour tier). Providers differ (Anthropic 5-min default / 1-hour extended; OpenAI implicit; Z.AI/Synthetic their own), so the observable path already noted - reading cached_tokens from the usage frame to infer live cache behavior per provider - is how the lane derives the threshold rather than hardcoding one. Confirmed parked until the post-v0.3.0 era; design this INTO the Stage-2 summarization lane.

Activity

  1. terenceponce commented on Nov 1, 2026

    @terenceponce
    MemberAuthor

    Design parked at the gate — this issue stays open.

    ADR 0033 (Proposed) is on main (PR #162, docs-only, hostile design review arc FIX-FIRST → CLEARED ×2). Nothing is implemented; the implementation lane opens only on a read of ADR 0033 §9 — the five open questions:

    1. The derivation itself — the issue's lean was observation-derived thresholds; §3 argues chain-over-derivation (misses are retroactive, capacity eviction impersonates expiry) with observation as the telemetry gate + deferred calibration. Confirm the reversal or direct observation-first.
    2. No-telemetry providers — hard-off (no idle compaction without cached_tokens evidence) vs a degraded fixed-interval variant.
    3. Activity granularity — any-key {:activity, ui} forwarding vs submit-only.
    4. The 300 s default + its silence — conservative (never misses a harvest), but on an hour-tier provider stage 2 pays a summarize per 300 s window; and timing assumptions stay silent (vs /compact answers 'no context window configured' - per-model window coverage is missing #111's visible-window precedent) — confirm or ask for a note.
    5. The m1 gate + the stage-2 contract — m1 stays its own slice with a >1 s fire-time measure as the pull-in trigger; §8 pins the stage-2 fire as a fail-closed, fail-bounded async summarize that prompts queue behind. Amend-under 0033, or re-open at stage 2?

    Headline design: a cache-TTL-derived, epoch-guarded idle timer in the agent (the one deliberate bend of the agent's no-timers contract — ADR 0007 note promised at acceptance), firing the same eviction_pass as prompt-time and /compact, armed only past the 80% trigger AND where cached_tokens telemetry proves the provider caches at all (that passthrough doesn't exist yet — plan step 1). Today's economics are honestly latency-placement (the m1 O(n²) stall moves off the first send); the warm-cache harvest arrives with stage 2, whose timer contract §8 pins as stage-invariant.

    Implementation plan with effort (M+ ≈ 2–3 focused days without m1) + the questions with recommendations: jobs/dashfloat/tasks/0099-idle-compaction/plan.md in the ops repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions