Skip to content

fix: restore a green test baseline (JVM heap, license headers, docs) - #35

Merged
easel merged 1 commit into
DocumentDrivenDX:mainfrom
dmautz1:fix/ci-test-baseline
Oct 27, 2026
Merged

easel merged 1 commit into
DocumentDrivenDX:mainfrom
dmautz1:fix/ci-test-baseline

Conversation

@dmautz1

@dmautz1 dmautz1 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Summary

The coverage job fails on every PR, on main's code too (115 failed, 4 errors, the same list on #33 and #34). This PR fixes the five root causes behind them.

Root cause Failures Fix
Spark driver heap stuck at 1 GB. The pytest process shares one local JVM, and whichever test starts it first sets the heap. The conformance engine and the JDBC lane start it without spark.driver.memory, so the JVM gets the 1 GB default, and the factory's later 4 GB request is silently ignored. The sample-data ingestion suite then runs out of memory (Java heap space), the JVM dies, and every later Spark or Sail test fails with Connection refused. ~55 tests/conftest.py sets PYSPARK_SUBMIT_ARGS=--driver-memory 4g pyspark-shell as a default, which applies however the JVM is started. Checked directly: a session started like the conformance engine's goes from a 1.0 GB to a 4.0 GB max heap.
Vendored loader bundles changed. The dual-license commit (6eb4572) added an SPDX line to the top of document_loader/canonical/run.ts and read-publication.ts. Both are pinned by sha256 in release.json, and both also need // @bun on line 1. 30 Restored the original vendored bytes. All 5 artifact hashes match again.
Headers in golden SQL. The same commit added the header to golden SQL files, and four suites compare against them exactly: LDP structure, LDP project, dbt-dag, and Databricks parse. 18 A shared tests/golden_text.read_golden() strips the header. It generalizes the helper that test_golden.py already had (from 6dfb864), and test_golden.py now uses it too.
Module map out of date. The implementation-plan module map was missing document_loader/ and excel_support.py. 1 Added both rows.
Unordered row comparison. The domain-pack lineage checks have no ORDER BY but were compared in order. Spark local[2] can return the rows in either order. 1 When the SQL has no ORDER BY, the rows are compared ignoring order.

The CLAUDE.md document_loader/ docs failure was already fixed in #34.

Testing (local)

  • tests/unit, tests/docs and tests/test_golden.py: 3,397 passed, 0 failed. The same set fails 31 tests on main.
  • The golden and conformance suites touched here: 65 passed, 35 skipped (Databricks workspace not configured).
  • tests/integration/test_sample_data_ingestion.py on the Spark backend, run for the small-legal, manufacturing and supply-chain packs: 6 passed. The full 16-pack Spark leg takes more than 25 minutes locally, so CI here is the full check.
  • ruff is clean.

🤖 Generated with Claude Code

The coverage job failed 115 tests + 4 errors on every PR; four root causes:

- Spark driver heap: the pytest process shares one local JVM and whichever
  test launches it first fixes the heap. The conformance engine (and the
  JDBC lane) launch without spark.driver.memory, so the JVM got the 1g
  default and the factory's later 4g request was silently ignored. The
  sample-data ingestion suite then OOMed (Java heap space), the JVM died,
  and ~55 later Spark/Sail tests failed "Connection refused". conftest now
  defaults PYSPARK_SUBMIT_ARGS to --driver-memory 4g for every launch path.
- Vendored loader bundles: the dual-license commit prepended an SPDX line
  to document_loader/canonical/run.ts and read-publication.ts, which are
  pinned by sha256 in release.json (and must keep `// @bun` on line 1).
  Restored the vendored bytes (30 document_loader tests).
- Golden SQL fixtures gained the same header; four suites compare goldens
  exactly. Shared tests/golden_text.read_golden() strips it, generalizing
  the helper test_golden.py already had (18 golden-drift tests).
- Docs: the implementation-plan module map now lists document_loader/ and
  excel_support.py.

Also: the domain-pack scenario checks compare rows in order, but two
`lineage` queries have no ORDER BY, so Spark local[2] could return them in
either order; compare unordered when the SQL defines no order.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@easel
easel merged commit ad969d1 into DocumentDrivenDX:main Oct 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants