Skip to content

Harden hybrid-reference reading, bound form painting, and make the PDF/A conversion read what the profiles name - #37

Open
CastleSoft wants to merge 15 commits into
MPCoreDeveloper:mainfrom
CastleSoft:hardening-2
Open

CastleSoft wants to merge 15 commits into
MPCoreDeveloper:mainfrom
CastleSoft:hardening-2

Conversation

@CastleSoft

Copy link
Copy Markdown

What this changes

Three robustness fixes and one correctness measurement, found by the checks in #32. Stacked on #36: it contains
the commits of #32 to #36 until those merge, so review the last two commits.

  • A hybrid-reference file is read, and an object stream no longer overrides the table. A hybrid file's classic
    table lists the objects it keeps in object streams as free; the table kept the first entry it met, so the free
    entry hid the real one in /XRefStm and the file read as having no pages. And a stream still holds the copies it
    held when it was written, so an object a later revision rewrote (or freed) came back depending on which
    object was resolved first.
  • A page that paints forms without end is read up to a budget. PdfEngineOptions.MaxFormPaintsPerPage (100,000)
    and 256 MiB of decoded form content per page. A 3.6 KB document of six levels of forms, each painting the next 40
    times, did not finish reading.
  • The PDF/A conversion sets right, removes or refuses what the profiles name in annotations, actions, forms,
    colour and content: see chapter 24.
  • The Arlington check blocks, with an accept list whose entries each say why.

Why, and what is wrong today

robustness.yml converts the veraPDF corpus for its profile and asks veraPDF 1.30.4 whether the result conforms.
On f529b02: 395 of 2,074 conversions were written as PDF/A and rejected by veraPDF, 17 of them from sources
that already conformed — the converter damaged them, chiefly by replacing a CMYK output intent with sRGB. The
analysis never decoded a content stream or read an annotation's flags, a bookmark, or an action's /Next chain.

Measured locally with the same veraPDF on the ten clause groups this changes (PDF_A-2b: 150 inputs meant to fail,
51 meant to pass): the failing inputs went to 48 refused, 76 conforming and 26 still failing; the passing inputs
to 39 conforming, 3 refused (over-refusals) and 0 damaged. The 26 are the limits docs/validation.md lists by clause
(colorant consistency, fonts, a few single rules). The full-corpus figure comes from the first robustness.yml run on
this branch and sets the --max-false-success ratchet.

The test that fails without this change

  • PdfCrossReferencePrecedenceTests (4): the hybrid file with no pages, the stale and the resurrected object, the
    stream read before /Prev. All four fail on 71485c0.
  • PdfFormBudgetTests (5): the text walk and the image walk report FormPaintLimitExceeded; the budget is per page.
    The two limit tests fail with the walkers reverted.
  • PdfAArchivalChecksTests (33): 22 fail on the previous analysis. PdfATests gains the refusal of RGB content
    under a gray intent, which the old test called "the validator's question".
  • PdfFormEditorTests.States_resources_on_the_appearance_of_a_button_that_draws_no_text fails without the fix.

Checklist

  • dotnet build .\SharpCorePDF.slnx -c Release is clean on both legs, with zero warnings
  • A test fails when this change is reverted, and it is in this pull request (checked, per group, above)
  • PublicAPI.Unshipped.txt updated in the same commit: MaxFormPaintsPerPage, PdfAFeatureRepaired, FormPaintLimitExceeded
  • CHANGELOG.md has an entry under [Unreleased]
  • Documentation updated: manual chapter 24, docs/validation.md, docs/release-checklist.md
  • No internal story, epic or requirement identifier anywhere
  • Not drive-by: no reordering, renaming or reformatting outside what the change needs

Anything else a reviewer should know

  • Behaviour change a caller can see. A page painted in DeviceRGB is now refused under an output intent that is not
    RGB; a document that states its own intent keeps it instead of being relabelled sRGB. A page authored by this
    library paints in DeviceRGB by default, so a caller wanting a CMYK intent authors in CMYK. SubmitForm is no longer
    refused (every part permits it); ResetForm, Hide, SetOCGState, Trans, GoTo3DView, SetState, NoOp are.
  • A new public enum member shifts nothing (= 68, = 69, appended), but a caller that switches over
    PdfWarningKind without a default meets two kinds it has not seen. Release note: this is a minor version.
  • Arlington is blocking and unverified locally (no Docker daemon here). Its accept list is the baseline the last
    run reported; if the first run on this branch finds a rule not in it, the job fails, which is the point.
  • Scoped Stryker over the touched files scores 99.6% on the changed lines; the two survivors are killed by a test added afterwards.

…ng and corpora

A writer that only reads its own output back cannot see the mistakes it shares with itself.
validation.yml now puts the example output and a validation corpus (written by
tools/SharpCorePDF.Validation, with a manifest of the text, passwords, permissions, revisions and
signature coverage each document must have) through qpdf, pypdf, pdfminer.six, MuPDF, pdf.js,
fontTools, OTS, PDFium and pyHanko, checks the incremental-update invariants, and replays the fuzz
seeds and four hostile documents. robustness.yml runs libFuzzer over the parser and the DCT, CCITT,
LZW, Flate and RunLength decoders, reads the veraPDF and pdf.js corpora one process per file, and
judges the PDF/A writer with veraPDF over the veraPDF corpus. compliance.yml cross-validates PDF/A-4.

The checks found defects that are not fixed here. Each is pinned by a skipped test
(PdfSignatureAttackTests, PdfIncrementalInvariantTests) or a strict --known-defect flag that fails
the run once the defect is fixed, and docs/validation.md lists them with their evidence.

No shipped package changes. The constitution's statement that no Python validator runs in CI is
corrected as a proposed PATCH amendment (2.0.1) for the maintainer to ratify. Local AI assistant
state (.claude/, CLAUDE.local.md) is ignored.
pdf.js 6 has no PDFDocumentProxy.destroy(), so the encryption check's pdf.js leg reported every document as unopenable: the call threw inside the try that was meant to catch a wrong password. The loading task owns the document and destroying it releases both. The fix was made before the previous commit but an editor saved a stale copy of the file over it.
Signature wrapping: the validator required only that the byte range start at 0 and end at the end of
the file, so bytes written into the unsigned gap could close the signature value early and add
entries while the digest still matched, and the signature was reported valid and covering the whole
document. The gap must now be exactly one hexadecimal string equal to the dictionary's /Contents;
anything else is MalformedByteRange, integrity Unknown and CoversWholeDocument false.

Decompression bombs: Flate, LZW and RunLength decoded without a limit, so a 1.1 MB document took the
process past 1 GiB. PdfEngineOptions.MaxDecodedStreamBytes (128 MiB) now stops them; decode buffers
above 16 MiB no longer stay in the shared array pool, and buffer growth no longer overflows.

Stack overflow: a chain of streams whose /Length is the next stream was resolved by recursion, and
60,000 links ended the process. The chain is followed no deeper than MaxNestingDepth, then the
stream is found by its endstream keyword, as for a /Length cycle.

Allocations sized by the file: an object stream's /N is bounded by what the stream can hold, and a
JPEG whose output is above the caller's budget is measured from its frame header instead of being
decoded first; the working sizes are computed in 64 bits.

Tests that fail without the fixes: PdfSignatureAttackTests (gap wrapping, gap shape),
PdfFilterTests (limits per filter, content stream past the limit), PdfReaderRecoveryTests (length
chain, indirect lengths, object stream /N), PdfJpegFilterTests (oversized frames, working
buffers). The three hostile documents of validation.yml are read under a 1 GiB heap, and their
--known flags are removed.
…fused

Four defects turned documents into exceptions that were not PdfExceptions, so a caller that handles
the contract's refusal still saw the read fail:

- A JPEG scan selected Huffman tables with 4-bit selectors and a progressive scan named its spectral
  selection, and both were used as indexes unchecked (IndexOutOfRangeException). ParseScan now refuses
  a selector above 3, and a progressive scan unless Ss <= Se <= 63 and Ah, Al <= 13.
- The empty name, a solidus on its own, is a name PDF allows, and four documents of the pdf.js test
  corpus use it as a key; PdfDictionary.Set refused it, so they did not open at all.
- A glyph drawn upside down (a negative font size) gave PdfGlyph.BoundingBox a bottom above its top,
  and the rectangle refused it, failing text extraction for the page.
- PdfDate.TryParse threw for a time-zone offset beyond fourteen hours, so one malformed /CreationDate
  failed the inspection of the whole document; such an offset is now read as no offset.

PdfContainerTests.Refuses_an_entry_without_a_key asserted the second defect and is rewritten: a null
key is still refused, the empty name is kept. Tests that fail without the fixes:
PdfJpegFilterTests (table selectors, progressive scan bounds), PdfObjectModelTests (empty-name key,
in a dictionary and in a document), PdfTextScannerTests (upside-down glyph), PdfSerializationTests
(offsets past fourteen hours). The pdf.js corpus and the hostile documents now read with no fault,
crash or timeout, and their --known flags are removed.
…t identifier

UseIncrementalUpdate ran the full rewrite into an appended writer, so every object of the document,
the signature dictionary of a signed source included, was written again under a new number. Every
earlier byte was kept, but the file grew by its own size per edit, and validators that look at what
changed after a signature (pyHanko, Acrobat) saw the signed field as rewritten: pyHanko placed a
signature followed by an edit as CONTIGUOUS_BLOCK_FROM_START instead of ENTIRE_REVISION.

In an appended revision the edited document's copier now keeps the numbers of its objects, the
catalog, the page tree, the pages and the information dictionary are written under the numbers they
already have, and the writer leaves out an object that would be written exactly as the source states
it. Rotating a page of a signed document now appends 716 bytes and three objects instead of 20,969
bytes and twelve; the signature dictionary is not in the revision.

The trailer also kept neither half of the source's /ID. ISO 32000-1 14.4 makes the first element
permanent across incremental updates; it is now the source's, and only the second element is new.

Tests that fail without the change: PdfIncrementalInvariantTests (the identifier, an update that
appends only what it changed, a signature dictionary left out of the revision); the corpus checks
validate-incremental.py and validate-signatures.py now pass with no --known-defect flag. The manual's
editing chapter described every edit as a complete rewrite and now says what the option does.
…CI runs found

- A merge copied a page that states no /MediaBox, and inherits none, without one, so the result broke the
  rule that every page has a media box and each reader assumed its own size. The page is written with the
  A4 box the inspection already describes it with, and PageSizeAssumed says so.
- A merge or an edit rewrote every real it copied to five decimals: a Type 3 font matrix of 0.000012 came
  out as 0.00001. A real read from a document now keeps its text when that is a well-formed PDF real and
  is written back from it. A real too large for the formatter's 32-byte buffer failed the write of the
  whole document with InvalidOperationException; the first run of the document fuzz target found it, and
  such a value is now formatted on the allocating path.
- The image copy route let InvalidDataException escape for damaged Flate data, with or without a
  predictor (the flate fuzz target's first find); it now refuses it with PdfSourceFormatException.
- The Arlington service names its verdict element arlingtonReport, which validate-arlington.py now reads.
  Its first run found that a free text annotation lacked the /DA ISO 32000-1 requires and annotation
  appearances lacked /Resources; both are now written.

Tests that fail without the changes: PdfMergeFidelityTests (media box, copied reals, any magnitude),
PdfContentReaderTests.Refuses_to_copy_an_image_whose_flate_data_is_damaged, and
PdfDocumentEditorTests.States_the_default_appearance_and_resources_of_a_free_text_annotation and
States_resources_on_an_appearance_that_needs_none. qpdf now passes every example output with no
--known-defect flag, so the qpdf step lists none.
…e edges

Recovery
- A cross-reference table that parses can still be wrong: bytes inserted or removed after it was written
  move every object. Reconstruction now also runs when the catalog's offset, or most of a sample of
  offsets, does not land on the n g obj header of its own object.
- Reconstruction indexes the objects of every object stream the scan finds, and takes /Root from the
  dictionary of the last cross-reference stream, so a PDF 1.5 file whose cross-reference stream is lost
  reads instead of being refused with "no document catalog". The test that pinned the refusal now pins
  the rebuild.

Hardening
- AESV2 always uses a sixteen-byte key; a crypt filter that declares fewer made the file unopenable.
- The whole-buffer predictor path sizes a row in 64 bits like the streaming decoder: a declared
  Columns x Colors that wrapped to zero looped forever under TIFF prediction, and to a negative number
  threw ArgumentOutOfRangeException.
- A stream with a wrong /Length is cut where its zlib data is complete, by its Adler-32, instead of at the
  first endstream whose truncated bytes inflate; at most sixteen candidates are inflated.
- A reference to a generation the table does not list resolves to null (ISO 32000-1, 7.3.10).
- A revision appended to a file whose newest section is a cross-reference stream, by an edit or a
  signature, ends in a cross-reference stream instead of a classic table. qpdf and pyHanko accept an edit
  and two signatures appended this way.

Corpus fault
- The public-corpora job faulted on isartor-6-1-11-t01-fail-a.pdf: a file specification that only names an
  external file is listed as an attachment with ByteLength 0, and copying it threw ArgumentException.
  It now writes no bytes; an id the document does not have is still refused.

Each change has a test that fails without it. Full suite passes on net11.0 and net10.0; the pdf.js corpus
run reports no fault, crash or timeout.
The scoped run over the files of the last two commits scored 70.8% on their changed lines. The survivors
that were missing assertions now have them:

- PdfCrossReferenceStreamWriterTests compares every row of an appended cross-reference stream byte for
  byte, its /Index subsections, the width of each /W field at the byte boundaries, its /Size and the
  trailer entries it carries or drops.
- The free text /DA test uses three different colour channels; the merge warning names the page as a
  person counts it; the damaged-Flate refusal says why.
- The predictor limit is pinned at the largest row accepted and one column more, and a zero-bit row is
  refused.
- The endstream scan leaves the parser behind the keyword it chose, and the sixteen-candidate cap is pinned
  at its boundary.
- A signature on a classic-table document keeps a classic table, and only an edit's stream lists the head
  of the free list.

PdfWriter's stream path no longer sets /Size before the stream writer, which derives it itself; the
statement was the survivor nothing could kill. The remaining survivors are equivalent mutants
(ConfigureAwait flips, bounds checks on offsets the scan already validated) or fallbacks for damaged files
noted in the pull request.
…ection does

The second robustness run, on 54c08d5, found two more faults now that the first ones are fixed:

- flate: the image copy route caught the decompressor's InvalidDataException but not the ZLibException
  it throws for other damaged input (crash-0b3d19d6). Both are now refused with
  PdfSourceFormatException. ZLibException is not public, so it is recognised by name rather than by
  catching every IOException, which would also swallow the destination's own failures.
- document: a page declaring /Rotate 91 made the edit engine throw ArgumentOutOfRangeException
  (crash-7201f51f). The edit engine read /Rotate as an integer and required a multiple of 90, while the
  inspection reads any number and rounds to the nearest quarter turn, so a real /Rotate was also read as
  none. The edit engine now uses the inspection's reading, and the quarter turns are reduced before they
  become an integer, so a declared value beyond int's range no longer wraps.

Tests that fail without the changes: PdfContentReaderTests.Refuses_to_copy_an_image_whose_flate_data_is_damaged
(the fuzz input's payload) and PdfMergeFidelityTests.Edits_a_page_whose_rotation_is_not_a_quarter_turn.
Both crash inputs and the two from the first run replay with no fault. Scoped Stryker over the two changed
files kills every mutant on the changed lines.
The third robustness run, on 7c97a38, found a stack overflow (crash-d34275a3): a merge walks the outline
recursively, one frame per level, and a source nested about fifty thousand levels deep ended the
process. The walk now stops at 256 levels and reports BookmarksTruncated, and a bookmark reached a
second time - a /First or /Next pointing back up the tree, or a child two parents share - is copied once
and reported with BookmarkSkipped, which also ends cycles and the repeated walks of shared subtrees.

Tests: PdfMergeOutlineLimitsTests (depth limit, shared child, self cycle). The crash input replays with
no fault; the full suite passes.
The robustness run on f529b02 found a crash in the document fuzz target (crash-e154541f).
Reconstruction keeps the damaged table's entries beside the ones its scan finds, and the search for a
cross-reference stream's trailer sliced the document at each of their offsets unchecked. An offset past
the end of the file, or one beyond the range of an index that wraps negative when cast, threw
ArgumentOutOfRangeException instead of letting the rebuild go on to find the catalog. Such an entry is now
passed over. The catalog search needs no such check: PdfLexer clamps the position it starts at, which a
scoped Stryker run confirmed by leaving the same guard there unkillable, so it was not added.

Test that fails without the change:
PdfReaderRecoveryTests.Rebuilds_a_table_that_states_an_offset_past_the_end_of_the_file, for an offset
past the end and one that wraps negative. The crash input replays with no fault. On the changed lines the
two surviving mutants are equivalent: offset 0 is the header and an offset equal to the length gives an
empty window, so neither boundary changes what is read.

The outline-depth fix of f529b02 is now in CHANGELOG.md and the fixed-defects table of
docs/validation.md, which it had not been.
A hybrid-reference file lists the objects it keeps in object streams as free in its classic table, so a
reader that knows no streams skips them, and describes them in the stream the trailer's /XRefStm names. The
table kept the first entry it met, so the free entry hid the real one and such a file read as having no pages.
The stream may now answer for the numbers its own section's table frees, and is read before the revision
/Prev points at, which is the order ISO 32000-1 7.5.8.4 gives.

Separately, loading an object stream cached every object inside it, and Resolve consulted that cache before
the table. A stream still holds the copies it held when it was written, so an object a later revision rewrote
in the file body came back stale, and one it freed came back from the dead, depending on which object was
resolved first. Only the numbers the table places in a stream are taken from it now, and the table is
consulted first.

PdfCrossReferencePrecedenceTests builds each case as a document; all four fail on the previous reader.
… the profiles name

The checks of the previous pull requests found a page that never finishes reading, and a PDF/A conversion that
writes files veraPDF rejects.

Forms. A form that paints the next form forty times, six levels down, is under the depth limit and the cycle
guard, and a 3.6 KB document asked for four billion paints. The text walk and the image walk now count a
page's paints against PdfEngineOptions.MaxFormPaintsPerPage (100,000) and its decoded form content against
256 MiB, report FormPaintLimitExceeded, and cache a form painted again. Field appearances state /Resources even
when empty, as annotation appearances already did; the Arlington model of PDF 1.7 requires it.

PDF/A. robustness.yml converted the veraPDF corpus and asked veraPDF to judge each result: 395 of 2,074 were
written and rejected, 17 of them from sources that already conformed. The analysis never decoded a content
stream or read an annotation's flags, a bookmark, or an action's /Next chain. It now sets the Print flag
right (PdfAFeatureRepaired), refuses hidden annotations and types a profile does not list, drops the actions
of widgets and fields and an XFA form, finds forbidden and named actions along chains and on bookmarks, pages
and the open action with the set the profiles list (SubmitForm is permitted), refuses DeviceCMYK or DeviceRGB
that the output intent does not cover (through Separation and DeviceN alternates and transparency groups),
undefined operators and forbidden graphics-state keys, and keeps a usable output intent the source already
states instead of relabelling it sRGB.

Measured locally with veraPDF 1.30.4 on the ten clause groups changed (PDF_A-2b, 150 inputs meant to fail,
51 meant to pass): failing inputs 48 refused, 76 conforming, 26 still failing; passing inputs 39 conforming,
3 refused, 0 damaged. The 26 are listed by clause in docs/validation.md. A page painted in DeviceRGB under an
intent that is not RGB is now refused, which PdfATests used to call the validator's question.

Tests that fail without the changes: PdfFormBudgetTests, PdfAArchivalChecksTests (22 of 33 fail on the previous
analysis) and PdfFormEditorTests.States_resources_on_the_appearance_of_a_button_that_draws_no_text.
Scoped Stryker over the touched files scores 99.6% on the changed lines; the two survivors, a content array,
are now killed. The Arlington check is blocking, with an accept list that gives a reason for each rule.
…uccesses at the measured count

The first Robustness run of this branch found a crash: a 378-byte file whose MediaBox digits overflow a double
reads as infinite, and the inspection, the page reader and the edit engine each passed that to a constructor
that refuses it, so an ArgumentOutOfRangeException escaped. A number that is not finite is now unreadable, so
the page is described as A4 with a remark, as one with no box is. Assumes_a_size_for_a_page_whose_media_box_overflows
and Edits_a_page_whose_media_box_overflows fail without the change.

The same run measured the PDF/A conversion over the whole veraPDF corpus: 229 false successes, down from 395,
and 2 conforming sources damaged, down from 17. robustness.yml now fails above 229, and docs/validation.md
records the figures and the clauses they fall in.

Scoped Stryker over the four files changed scores 100% (424 mutants).
The walk of the form's field tree was bounded in depth but followed every path to the bottom, so a form
whose levels each list the next level twice has two to the power of its depth paths. Forty levels, in a
2.6 KB file, and the signature validation never finished; the document fuzz target found it as a timeout on
the second Robustness run of this branch. The other field-tree walks already keep a set of the fields they
have read; this one now does as well.

Finishes_looking_for_signatures_in_a_form_whose_levels_each_list_the_next_twice waits 30 seconds for forty
levels and fails without the change. Scoped Stryker over SignatureValidator.cs scores 100% (179 mutants).
@CastleSoft

Copy link
Copy Markdown
Author

Two commits were added to this pull request after it was opened (e4c615e and 402bc7f), both from the first Robustness run on the branch.

  • A crash the document fuzz target found. A MediaBox whose digits overflow a double reads as infinite, and the inspection, the page reader and the edit engine each passed it to a constructor that refuses it. Such a box is now unreadable and the page is described as A4 with a remark. Two tests fail without the change.
  • A timeout the same target found. The signature search walked every path of the form's field tree, so a form whose levels each list the next twice (forty levels, 2.6 KB) never finished. Each field is now read once. One test fails without the change.
  • The PDF/A ratchet is set. The full-corpus run measured 229 false successes, down from 395, and 2 conforming sources damaged, down from 17. robustness.yml now fails above 229; docs/validation.md has the breakdown by clause.

Scoped Stryker over the changed files scores 100% for both; the suite is 1,416 tests on net10.0 and net11.0 with zero warnings.

Not yet re-run: the Fuzz document job on the latest commit. Three other walks bound only depth (the embedded-files name tree in PdfEditEngine.Document.cs and two in AcroFormWriter.cs) and could hang the same way; fuzzing has not reached them, so they are unchanged here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant