Repository navigation
Conversation
12458
force-pushed
the
lore-pipeline-failures
branch
4 times, most recently
from
November 18, 2026 19:27
77f606f to
8871556
Compare
The lore pipeline only logged emails it could not read, batches it could not store, commits it could not record as indexed, and parser or inserter threads that failed or panicked. It then returned Ok, so semcode-index printed "Indexed N new emails" and exited 0 for an archive that was only partly indexed. Nothing was lost for good -- the unrecorded commits were read again by the next run -- but automation could not tell that a run was incomplete. Count these failures and return a LoreIndexIncomplete error at the end of the run, for example: lore indexing incomplete: 1 email could not be read; indexing again retries them (2 emails stored) Emails the run stored stay stored and recorded. Their count is carried by the error, so a failed archive's new emails still trigger compaction and the FTS update. Two cases left commits unrecorded without anything having failed. Make the run finish them, so success means every commit is recorded: - An email whose message_id another batch held was deferred to the next run. Retry deferred emails once every inserter has finished. insert_lore_emails() now returns failed and deferred emails apart. - public-inbox records a removed message as a commit with a 'd' file and no 'm'. It was re-read and warned about on every run. Record it as indexed instead. The main task no longer holds the commit and result channels, so if every parser or every inserter stops, the other side gets an error instead of blocking forever on a full channel. Co-authored-by: Chris J Arges <[email protected]>
12458
force-pushed
the
lore-pipeline-failures
branch
from
November 20, 2026 01:26
8871556 to
a2c5161
Compare
12458
marked this pull request as ready for review
November 20, 2026 03:01
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The lore indexing pipeline logs these failures instead of failing the run:
mblob (the email file in a public-inbox commit) is missing.process_lore_commits_pipelinethen returnsOk(()).semcode-index --loreprintsIndexed N new emailsand exits 0, even though the archive is only partly indexed. For example, if 1,000 new emails are requested and one batch of 100 fails to store, the command still printsIndexed 1000 new emails. Searches miss those 100 emails, and automation sees a successful run.No email is lost for good: the next run retries the failed ones. But the operator isn't told that another run is needed.
After this change, the run fails with an error such as:
Stored emails stay stored and recorded. Failed commits stay unrecorded, so the next run reads them again. The error includes how many emails were stored.
process_lore_commits_pipelinenow returns the number of rows it appended to the lore table, and the error carries the same count.semcode-indexadds up that count for every archive, whether it finished or failed. That total is theNew emails indexedcount and decides whether compaction and the full-text search (FTS) update run.Emails found already stored (such as crossposts) and commits without a message add no rows, so they no longer count as new emails. Before, a finished archive counted every new commit, and a failed one would have counted nothing.
In two cases, a run left commits unrecorded even though nothing went wrong. The run now handles both:
Duplicate message IDs. Sometimes two batches hold emails with the same
message_id. One batch stores the email. The other batch defers its copy: it does not store it, and it does not record its commit. That commit used to wait for the next run. Now the run retries deferred emails after every inserter has finished.To support this,
insert_lore_emails()now returnsLoreInsertOutcome { failed, deferred, appended }: the emails that failed, the emails that were deferred, and the rows appended. When a batch holds several copies of a failed or deferredmessage_id, every copy is reported, so no copy's commit is recorded while its message is not stored.Commits without an
mfile. public-inbox records a removed message as a commit with adfile and nomfile. Such a commit was read again, with a warning, on every run. Now it is recorded as indexed. A tree that cannot be decoded is still treated as unreadable, not as a removed message. Removing the deleted message from the index is out of scope.This change also fixes a possible hang. If every inserter died, the parsers waited forever. Now they stop with an error.
This is the lore counterpart of #135. The error summary follows its format, and
join_failed()and the summary helpers move tosrc/indexer.rsso that both pipelines share them.Tests, in
src/indexer.rs:unreadable_email_fails_the_run_and_is_retried: one commit's blob is deleted. The run fails. The other two emails are stored and recorded. After the blob is restored, the next run stores the third email.unstored_emails_fail_the_run_and_are_retried: the lore table is missing. The run fails and records nothing. The next run stores every email.unrecorded_commits_fail_the_run_and_are_retried: thelore_indexed_commitstable is missing. The run fails, but the emails are stored. The next run records the commits and appends no rows.failed_parsers_fail_the_run: the parsers cannot open the archive. The run fails.malformed_tree_is_not_recorded: a commit whose tree cannot be decoded fails the run and stays unrecorded.already_stored_email_is_not_counted_as_appended: a failed run whose only readable email was already stored reports no appended rows.commit_without_a_message_is_recorded: a commit with adfile is recorded, and the run succeeds.duplicate_message_ids_are_stored_once_and_recorded: one message ID appears in four commits. The run stores one email and records all four commits.every_copy_of_a_pending_id_is_deferred: every copy of a deferred message ID in a batch is reported as deferred.incomplete_summary_lists_each_failure: checks the exact error text.The tests of finished runs also check the appended count. For example, the crosspost test checks that the second archive appends only its one new email.
Without the new error check, the first four tests fail.
Validation:
cargo fmt --checkcargo clippy --all-targets -- -D warningscargo test