Repository navigation
fix: restore a green test baseline (JVM heap, license headers, docs) - #35
Merged
Merged
Conversation
The coverage job failed 115 tests + 4 errors on every PR; four root causes: - Spark driver heap: the pytest process shares one local JVM and whichever test launches it first fixes the heap. The conformance engine (and the JDBC lane) launch without spark.driver.memory, so the JVM got the 1g default and the factory's later 4g request was silently ignored. The sample-data ingestion suite then OOMed (Java heap space), the JVM died, and ~55 later Spark/Sail tests failed "Connection refused". conftest now defaults PYSPARK_SUBMIT_ARGS to --driver-memory 4g for every launch path. - Vendored loader bundles: the dual-license commit prepended an SPDX line to document_loader/canonical/run.ts and read-publication.ts, which are pinned by sha256 in release.json (and must keep `// @bun` on line 1). Restored the vendored bytes (30 document_loader tests). - Golden SQL fixtures gained the same header; four suites compare goldens exactly. Shared tests/golden_text.read_golden() strips it, generalizing the helper test_golden.py already had (18 golden-drift tests). - Docs: the implementation-plan module map now lists document_loader/ and excel_support.py. Also: the domain-pack scenario checks compare rows in order, but two `lineage` queries have no ORDER BY, so Spark local[2] could return them in either order; compare unordered when the SQL defines no order. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
coveragejob fails on every PR, onmain's code too (115 failed, 4 errors, the same list on #33 and #34). This PR fixes the five root causes behind them.spark.driver.memory, so the JVM gets the 1 GB default, and the factory's later 4 GB request is silently ignored. The sample-data ingestion suite then runs out of memory (Java heap space), the JVM dies, and every later Spark or Sail test fails withConnection refused.tests/conftest.pysetsPYSPARK_SUBMIT_ARGS=--driver-memory 4g pyspark-shellas a default, which applies however the JVM is started. Checked directly: a session started like the conformance engine's goes from a 1.0 GB to a 4.0 GB max heap.document_loader/canonical/run.tsandread-publication.ts. Both are pinned by sha256 inrelease.json, and both also need// @bunon line 1.tests/golden_text.read_golden()strips the header. It generalizes the helper thattest_golden.pyalready had (from 6dfb864), andtest_golden.pynow uses it too.document_loader/andexcel_support.py.lineagechecks have noORDER BYbut were compared in order. Sparklocal[2]can return the rows in either order.ORDER BY, the rows are compared ignoring order.The CLAUDE.md
document_loader/docs failure was already fixed in #34.Testing (local)
tests/unit,tests/docsandtests/test_golden.py: 3,397 passed, 0 failed. The same set fails 31 tests onmain.tests/integration/test_sample_data_ingestion.pyon the Spark backend, run for the small-legal, manufacturing and supply-chain packs: 6 passed. The full 16-pack Spark leg takes more than 25 minutes locally, so CI here is the full check.🤖 Generated with Claude Code