[PSeitz]

Why

The Indexer hands finished batches to a separate IndexSerializer actor and yields the CPU. The serializer may not be scheduled immediately, leaving a gap before serialization starts. Running serialization and packaging inside the Indexer removes those actor handoffs and their scheduling delays. We want to finish as fast a possible to release memory.

Changes

• Serialize and package indexing batches inside the Indexer, then send them directly to the Uploader.
• Package merged splits inside the MergeExecutor using the same packaging helper.
• Remove the IndexSerializer and Packager actors, their intermediate messages, and their pipeline supervision wiring.
• Propagate final-flush serialization and packaging errors from the Indexer.
• Count flush time toward the cooperative indexing sleep budget so flushing does not delay the next scheduled work period.

Mailboxes and backpressure

Actor	Previous capacity	New capacity
DocProcessor	10	10
Indexer	10	10
IndexSerializer	0	Removed
Packager (indexing and merge)	1	Removed
MergeExecutor	1	1
Uploader	0	0

Capacities count messages, which can contain batches. Capacity 0 is a rendezvous: sending waits until the receiver accepts the message.

The serializer could process one batch while the Indexer built the next, despite its zero-capacity mailbox. Serialization and packaging now finish before the Indexer processes more documents. Removing the packager also removes its queued batch and separate in-progress work.

This trades pipeline overlap and buffering for fewer scheduling handoffs. Uploader backpressure reaches the Indexer or MergeExecutor directly. Upload concurrency remains semaphore-controlled, and handoff waits for acceptance rather than upload completion.

Operational changes

• Serialization I/O metrics move from the index_serializer component to indexer.
• Delete-task pipeline observations report packaging failures under delete_task_executor; the separate packager entry is removed.

Verification

• Affected crate suites: 290 tests passed, one test skipped.
• In-memory indexing: 14 tests passed.
• Failpoints: 10 tests passed.
• Formatting and diff checks passed.
• Production throughput and latency under sustained load have not been benchmarked.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-31T16:54:29.856312Z	88ef471	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d5227d37ee

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Finalize all writers before packaging splits

When a workbench contains many partition builders (especially with QW_ENABLE_IN_MEMORY_INDEXING), this packages each finalized split while the iterator still owns all remaining SingleSegmentIndexWriters, and every resulting hotcache_bytes allocation remains retained in packaged_splits. At a heap-limit flush this layers accumulating hotcaches over the still-live writer/RAM-directory buffers and can push indexing into an OOM; the removed serializer avoided this by finalizing every builder before the packager allocated any hotcaches. Preserve that two-pass ordering inside the Indexer.

Useful? React with 👍 / 👎.