Repository navigation
Add further information to output - #80
Conversation
What does this change?When a job completes with no words, the only information available today is Page errors that were previously silentThese are added to
New countsThree new fields on the job, the job response (
These aren't errors: a manifest can include a cover or blank page with no OCR, and that shouldn't be reported as a failure. They're reset on reprocess. ExampleA completion notification for a two-canvas manifest where one ALTO file is missing and the other has no words: {
"jobId": "1/my-manifest",
"status": "Completed",
"finished": "2026-11-05T10:00:00+00:00",
"totalPages": 2,
"totalWordCount": 0,
"errors": "https://example.org/canvases/c1: no text found at https://example.org/alto/c1.xml",
"invocationCount": 1,
"skippedCanvases": 0,
"pagesWithoutTextSource": 0,
"pagesWithoutWords": 1
}Behaviour changeJobs whose text files return 404 (or are empty) now complete with Consumers
Tests
Database MigrationNote
|
c261f21 to
a72c673
Compare
What does this change?
When a job completes with no words, the only information available today is
totalWordCount: 0. Several different causes look identical, and most of them aren't recorded anywhere (dlcs/iiif-presentation#677 is an example of this being hard to diagnose). This PR records why pages produced no text, so callers can tell the user.Page errors that were previously silent
These are added to
errorson a completed job, one line per page, alongside the fetch exceptions that were already recorded there:{canvasId}: no text found at {textUri}. Applies to ALTO/hOCR, VTT and annotation pages.TextBuilder.AddPage/AddTranscriptPagenow returnfalsewhen no provider handles the page, instead of only logging a warning. Now:{canvasId}: unsupported text format at {textUri} (profile '…', format '…').New counts
Three new fields on the job, the job response (
GET /textbuilder/{id}) andJobCompletionNotification:skippedCanvasesManifestReducer.Reducenow returns this alongside the pages, in a newReducedManifestrecord. Always 0 forsourceDatajobs.pagesWithoutTextSourcepagesWithoutWordsTextBuilder.WordCount.These aren't errors: a manifest can include a cover or blank page with no OCR, and that shouldn't be reported as a failure. They're reset on reprocess.
Example
A completion notification for a two-canvas manifest where one ALTO file is missing and the other has no words:
{ "jobId": "1/my-manifest", "status": "Completed", "finished": "2026-11-05T10:00:00+00:00", "totalPages": 2, "totalWordCount": 0, "errors": "https://example.org/canvases/c1: no text found at https://example.org/alto/c1.xml", "invocationCount": 1, "skippedCanvases": 0, "pagesWithoutTextSource": 0, "pagesWithoutWords": 1 }Behaviour change
Jobs whose text files return 404 (or are empty) now complete with
errorsset, where previouslyerrorswasnull. The job status is unchanged (Completed). Anything that treats a non-nullerrorson a completed job as a failure will see this.Consumers
errorsfor completed jobs, and to show them as a warning onfinishedPipelines.Tests
TextBuilderTests, plus new tests inTextBuildJobTests(missing text, unsupported format, page without words, skipped canvases, notification contents) andManifestReducerTests(skipped count).ExecuteAsync_AltoReturnsNull_PageSkippedSilentlyandExecuteAsync_VttFetchReturnsNull_PageSkippedSilentlyasserted the old silent behaviour, so they've been renamed and updated to expect the new error.Database Migration
Note
AddDiagnosticsToBuilderJobskipped_canvases,pages_without_text_sourceandpages_without_words(integer, not null, default 0) to thejobstable🤖 Generated with Claude Code