improvement(search): run Search retirement as an operator command instead of a deploy step

[waleedlatif1]

Summary

Deploys no longer run the Search retirement (0027–0029). It becomes an operator-run, resumable, paced maintenance command. Supersedes #8521, which proposed a background scheduler; that is not needed for a one-off cleanup.

Why: a long cleanup inside the deploy migration generated heavy WAL. Checkpoints became WAL-triggered back to back, commits waited on synchronous replication, and application writes hit lock and statement timeouts. It also held the release until it finished. The retirement is optional storage reclamation once live Search is on, so it does not belong on the deploy's critical path.

Changes

• Deploy registry: packages/db/script-migrations/index.ts drops 0029. The registry rule allows deleting an entry, never renaming or reordering one. A deploy's migration run no longer creates or touches the cleanup tables.
• Operator command (0027_retire_search_embeddings.ts, entry point unchanged):
  • --pause-ratio N (default 2): after each page, pause N × the page's duration, capped at one minute (it was 1× capped at 5 s). Pages are timed through their commit, so a slow synchronous replica or a checkpoint stall lengthens the pause.
  • --max-rows N (default 2000, range 25–8,000): lowers the starting row limit and the ceiling it can grow to.
  • --maintenance: also runs the HNSW rebuilds and vacuum, and journals 0029 with its superseded names. A plain run only retires, journals nothing, and resumes the saved cursor.
• 0029 is now retireAllSearchEmbeddings(pacing), used by the command and by tests.
• search-embedding-retirement.md is rewritten as an operator runbook:
  • the command and its flags;
  • a direct or session-pooled connection only, run as the migration role;
  • off-peak timing;
  • what to watch;
  • that Ctrl-C is a safe pause and the run resumes;
  • that self-hosted operators can run it too.

Behaviour changes

• Deploy migrations stop at 0026. Databases mid-way through the retirement keep their progress row and cursor, and the command resumes them.
• Pacing is gentler by default: 2× the page time, capped at one minute.
• Nothing changes for application code. The retirement only ever touched Search KBs' documents and chunks, which live Search does not read.

Not changed: findings from #8521's review that no longer apply

• Target snapshot before any time check. The snapshot reads only knowledge_base IDs in one repeatable-read transaction. It is cheap, and it now runs only when an operator starts the command.
• Full marker recheck on resuming a finished retirement. It walks the captured KB IDs, not documents or chunks, so it is cheap. It runs only in the operator command, so no deploy waits on it.
• Phase-change pages skipping the slow-page halving. The halving is skipped on purpose, because a phase change's recheck says nothing about page cost. The pause after the page still applies.

Precedent

• strong_migrations, Backfilling data: batch and throttle a backfill separately from the migration.
• GitLab batched background migrations take long data work off deploy migrations, throttled on database health. GitLab also runs automatic reindexing in a weekend window.
• Shopify maintenance_tasks are operator-run tasks outside deploys, with a cursor, pause/resume and throttling.
• gh-ost throttling and pt-online-schema-change pace chunked work on lag and load.
• Stripe online migrations: backfills run offline from deploys.
• PostgreSQL 17 docs on WAL and checkpoints and synchronous replication.
• PlanetScale: the only scalable delete.

Test plan

• New: a full registry run never starts the retirement. 0016_backfill_search_vectors.integration.ts asserts that the cleanup tables do not exist after runScriptMigrations(sql). It went red with 0029 re-registered.
• New: the pause ratio is honoured. 0027_retire_search_embeddings.integration.ts uses a statement trigger that makes each delete page take about 100 ms; with a ratio of 3, consecutive pages must start at least 380 ms apart. It went red with the pause removed (123 ms).
• Updated: the legacy-checkpoint test uses retireAllSearchEmbeddings() instead of filtering the registry.
• packages/db suites against real Postgres 17 + pgvector: 128 integration tests and 104 unit tests pass.
• Smoke test of the operator command: the flags are accepted, an out-of-range --max-rows is rejected, a plain run on an empty database finishes and journals nothing, and --maintenance journals.
• bun run lint, packages/db type-check, and bun run check:audits (53) all pass.

[waleedlatif1]

@greptile

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Oct 1, 2026 4:50pm UTC

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[greptile-apps]

Confidence Score: 4/5

[Critical risk] Moves search retirement from automatic deploy to manual operator command.

The PR appears safe to merge, with a non-blocking runbook correction needed about overlapping plain runs.

Findings

1.  Plain runs lack the documented lock ▶

Summary

The PR removes Search retirement from deploy migrations and makes it an operator-run, paced command with optional index maintenance and completion journaling.

• Adds configurable page pauses and row limits while retaining the saved retirement cursor.
• Updates integration coverage and replaces the deployment guidance with an operator runbook.
• The runbook overstates the concurrency protection of a plain run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  D[Deploy] --> R[Script registry through 0026]
  O[Operator command] --> P{--maintenance?}
  P -- No --> C[Paced retirement; save cursor]
  P -- Yes --> C
  C -->|maintenance run| M[Rebuild indexes and vacuum]
  M --> J[Journal 0029 and superseded names]

Loading

_{Reviews (1) · Last reviewed commit: "improvement(search): run Search retireme..."}

[cubic-dev-ai]

No issues found across 6 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>This PR changes a database schema or migrates data. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.

Re-trigger cubic</sub>

[cubic-dev-ai]

No issues found across 6 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}