Skip to content

fix: memory keeps every byte and stops freezing the daemon - #379

Merged
saucam merged 3 commits into
mainfrom
feat/lossless-memory
Oct 13, 2026
Merged

saucam merged 3 commits into
mainfrom
feat/lossless-memory

Conversation

@saucam

@saucam saucam commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

What this fixes

Memory is meant to be verbatim, but it was quietly lossy, and its "background" embedding was freezing the daemon.

Problem (measured) Fix
Embedding ran on the daemon's main thread: each batch of 8 episodes froze every session, client and Telegram update for ~450 ms. The model runs on a worker thread. Longest stall while embedding 16 long episodes: 14–74 ms.
Tool output was cut at 64 KiB, because full-text indexing costs ~9 ms per MiB on the event path (220 ms for a 10 MiB log). Storing and indexing are split. The full body is kept, zstd-compressed (~5 ms and ~5% of the size for 10 MiB). Only a bounded view is indexed: the head, the tail and the middle's error/warning lines, up to 256 KiB. get_episode pages through the full body.
The embedding model reads 512 tokens and drops the rest, so semantic recall saw only the start of each episode, often not the tool result at all. Each episode is embedded in up to 8 windows, placed on the head, the tail and diagnostic lines, and scored by its best window. A failure buried mid-way through a 21K-char log moved from rank 8 to rank 1 on a plain-English query.
With output compression on, the hint said "memory has the raw output", but only the compressed text was stored. The wrapper spills the raw output to a private file named in the hint. Memory stores it in place of the compressed text.

New settings

  • memory.embeddings.enabled (default true, env CODEOID_MEMORY_EMBEDDINGS): turn semantic recall off. The model is never downloaded or loaded, and recall uses full-text search only.
  • memory.embeddings.maxWindows (default 8): embedding windows per episode. Each costs about 45 ms of worker-thread CPU.

Both appear in the Memory settings tab.

Not in this PR

  • Full-text search requires every query word to match, so plain-English questions ("why did X fail") usually get no keyword hits and rely on embeddings alone. This predates this PR; the search rework planned in docs/conductor-session-resolution.md covers it.
  • Episodes written before this change stay as they were (capped, one embedding). There is no backfill.
  • Retention: nothing is deleted. storageStats() gives a size readout for a future UI.

Testing

  • Typecheck and lint pass. 3,027 daemon tests and 593 web tests pass, including 19 new ones in memory-lossless.test.ts.
  • Live against the real model: the event-loop stall and deep-window recall numbers above.
For agents: where things are
  • src/daemon/memory/lossless.ts: indexView (bounded index view), embedWindows (window placement).
  • src/daemon/memory/store.ts: new tables episode_payloads (zstd body) and episode_vectors (windows 1..n; window 0 stays in episodes.embedding), plus getFullContent, setWindowEmbeddings and storageStats. The vector matrix now has one row per window; ids repeats per window.
  • src/daemon/memory/engine.ts: ingest splits body and view; the embed pump batches windows (32 texts per call); recall scores best window per episode (vectorScores passed to the ranker); embeddingsEnabled / maxEmbedWindows options.
  • src/daemon/memory/embedder-worker.ts + WorkerEmbedder in embedder.ts: the off-thread model.
  • src/daemon/compress/spill.ts: write/take/prune spill files. The raw=<uuid> reference is anchored to the trailing hint line. Spills are pruned after 24 h at boot.
  • src/daemon/memory/chunker.ts: the 64 KiB cap is gone; spilled raw output is swapped in.
  • src/daemon/memory/tools.ts: get_episode takes offset / max_chars (default 64K, max 256K).

🤖 Generated with Claude Code

saucam and others added 2 commits October 12, 2026 05:53
Memory claimed to be verbatim but cut tool output at 64 KiB, embedded only
the first ~512 tokens of each episode, and ran the embedding model on the
daemon's main thread. Measured, those were the real costs:

- Embedding a batch of 8 episodes froze the event loop for ~450 ms. The
  model now runs on a Bun Worker; the longest stall while embedding 16 long
  episodes dropped to 14-74 ms.
- Full-text indexing a body costs ~9 ms per MiB on the event path (220 ms
  for 10 MiB), which is why output was capped. Storage and indexing are now
  separate: the full body is kept zstd-compressed (~5 ms, ~5% of the size
  for 10 MiB) in episode_payloads, and only a bounded view (head, tail and
  the diagnostic lines of the middle, <=256 KiB) is indexed. get_episode
  pages through the full body.
- bge-small reads 512 tokens and drops the rest, so semantic recall never
  saw most tool results. Episodes are now embedded in up to 8 windows,
  placed on the head, tail and diagnostic lines, and scored by their best
  window. A failure buried 10K chars into a 21K-char log went from rank 8
  to rank 1 on a natural-language query.
- With output compression on, the hint promised the raw output was in
  memory, but only the compressed text was stored. The wrapper now spills
  the raw output to a private file named in the hint, and the chunker
  stores it in the episode.

memory.embeddings.enabled (default true, CODEOID_MEMORY_EMBEDDINGS) turns
semantic recall off entirely: the model is never loaded and recall runs on
full-text search. memory.embeddings.maxWindows (default 8) sets windows per
episode.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@saucam
saucam merged commit 7bae800 into main Oct 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants