Skip to content

lore: fail lore indexing when emails are left unindexed - #137

Open
12458 wants to merge 1 commit into
facebookexperimental:mainfrom
12458:lore-pipeline-failures
Open

12458 wants to merge 1 commit into
facebookexperimental:mainfrom
12458:lore-pipeline-failures

Conversation

@12458

@12458 12458 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

The lore indexing pipeline logs these failures instead of failing the run:

  • A worker cannot read an email. For example, the m blob (the email file in a public-inbox commit) is missing.
  • A batch of emails cannot be stored in the lore table.
  • The commits of a stored batch cannot be recorded as indexed.
  • A parser thread (which reads emails) or an inserter task (which stores them) fails or panics.

process_lore_commits_pipeline then returns Ok(()). semcode-index --lore prints Indexed N new emails and exits 0, even though the archive is only partly indexed. For example, if 1,000 new emails are requested and one batch of 100 fails to store, the command still prints Indexed 1000 new emails. Searches miss those 100 emails, and automation sees a successful run.

No email is lost for good: the next run retries the failed ones. But the operator isn't told that another run is needed.

After this change, the run fails with an error such as:

Error indexing lkml/0: lore indexing incomplete: 100 emails could not be stored; indexing again retries them (900 emails stored)

Stored emails stay stored and recorded. Failed commits stay unrecorded, so the next run reads them again. The error includes how many emails were stored.

process_lore_commits_pipeline now returns the number of rows it appended to the lore table, and the error carries the same count. semcode-index adds up that count for every archive, whether it finished or failed. That total is the New emails indexed count and decides whether compaction and the full-text search (FTS) update run.

Emails found already stored (such as crossposts) and commits without a message add no rows, so they no longer count as new emails. Before, a finished archive counted every new commit, and a failed one would have counted nothing.

In two cases, a run left commits unrecorded even though nothing went wrong. The run now handles both:

  • Duplicate message IDs. Sometimes two batches hold emails with the same message_id. One batch stores the email. The other batch defers its copy: it does not store it, and it does not record its commit. That commit used to wait for the next run. Now the run retries deferred emails after every inserter has finished.

    To support this, insert_lore_emails() now returns LoreInsertOutcome { failed, deferred, appended }: the emails that failed, the emails that were deferred, and the rows appended. When a batch holds several copies of a failed or deferred message_id, every copy is reported, so no copy's commit is recorded while its message is not stored.

  • Commits without an m file. public-inbox records a removed message as a commit with a d file and no m file. Such a commit was read again, with a warning, on every run. Now it is recorded as indexed. A tree that cannot be decoded is still treated as unreadable, not as a removed message. Removing the deleted message from the index is out of scope.

This change also fixes a possible hang. If every inserter died, the parsers waited forever. Now they stop with an error.

This is the lore counterpart of #135. The error summary follows its format, and join_failed() and the summary helpers move to src/indexer.rs so that both pipelines share them.

Tests, in src/indexer.rs:

  • unreadable_email_fails_the_run_and_is_retried: one commit's blob is deleted. The run fails. The other two emails are stored and recorded. After the blob is restored, the next run stores the third email.
  • unstored_emails_fail_the_run_and_are_retried: the lore table is missing. The run fails and records nothing. The next run stores every email.
  • unrecorded_commits_fail_the_run_and_are_retried: the lore_indexed_commits table is missing. The run fails, but the emails are stored. The next run records the commits and appends no rows.
  • failed_parsers_fail_the_run: the parsers cannot open the archive. The run fails.
  • malformed_tree_is_not_recorded: a commit whose tree cannot be decoded fails the run and stays unrecorded.
  • already_stored_email_is_not_counted_as_appended: a failed run whose only readable email was already stored reports no appended rows.
  • commit_without_a_message_is_recorded: a commit with a d file is recorded, and the run succeeds.
  • duplicate_message_ids_are_stored_once_and_recorded: one message ID appears in four commits. The run stores one email and records all four commits.
  • every_copy_of_a_pending_id_is_deferred: every copy of a deferred message ID in a batch is reported as deferred.
  • incomplete_summary_lists_each_failure: checks the exact error text.

The tests of finished runs also check the appended count. For example, the crosspost test checks that the second archive appends only its one new email.

Without the new error check, the first four tests fail.

Validation:

  • cargo fmt --check
  • cargo clippy --all-targets -- -D warnings
  • cargo test

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 17, 2026
@12458
12458 force-pushed the lore-pipeline-failures branch 4 times, most recently from 77f606f to 8871556 Compare November 18, 2026 19:27
The lore pipeline only logged emails it could not read, batches it
could not store, commits it could not record as indexed, and parser or
inserter threads that failed or panicked. It then returned Ok, so
semcode-index printed "Indexed N new emails" and exited 0 for an
archive that was only partly indexed. Nothing was lost for good -- the
unrecorded commits were read again by the next run -- but automation
could not tell that a run was incomplete.

Count these failures and return a LoreIndexIncomplete error at the end
of the run, for example:

  lore indexing incomplete: 1 email could not be read; indexing again
  retries them (2 emails stored)

Emails the run stored stay stored and recorded. Their count is carried
by the error, so a failed archive's new emails still trigger compaction
and the FTS update.

Two cases left commits unrecorded without anything having failed. Make
the run finish them, so success means every commit is recorded:

- An email whose message_id another batch held was deferred to the next
  run. Retry deferred emails once every inserter has finished.
  insert_lore_emails() now returns failed and deferred emails apart.
- public-inbox records a removed message as a commit with a 'd' file and
  no 'm'. It was re-read and warned about on every run. Record it as
  indexed instead.

The main task no longer holds the commit and result channels, so if
every parser or every inserter stops, the other side gets an error
instead of blocking forever on a full channel.

Co-authored-by: Chris J Arges <[email protected]>
@12458
12458 force-pushed the lore-pipeline-failures branch from 8871556 to a2c5161 Compare November 20, 2026 01:26
@12458
12458 marked this pull request as ready for review November 20, 2026 03:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant