Skip to content

Reset phase 2A: triage the 144 quarantined Bun files - #149

Merged
easel merged 14 commits into
mainfrom
reset/phase2-tests
Nov 12, 2026
Merged

easel merged 14 commits into
mainfrom
reset/phase2-tests

Conversation

@easel

@easel easel commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

What this does

Phase 2A of the reset: triage the 144 Bun test files that #143 (the Rust port, merged with CI skipped) left failing on a clean Linux run and quarantined in scripts/known-failures.txt. Every file was run, locally on macOS (all 144) and on Linux in CI (78 files before the run was cancelled to free runners). The two agree on every one of those 78 files, so the macOS run is used for the rest.

Result: known-failures.txt goes from 144 entries to 130, and every remaining entry now has a one-line reason (a failing assertion or error, tagged with the likely shared cause). The 130 are possible regressions of public behaviour (see "Findings" below), not stale or environment-bound tests.

Category table

Category Files What was done
1. Tests of deleted TypeScript engine internals 0 files, 1 test case No file imports a deleted module (checked statically for all 144: every relative import resolves). One test case did read deleted source: tests/actions-manifest.test.ts "portable modules contain no host-only imports" iterated over src/extensions/actions/{manifests,framed-json}.ts, which #143 deleted. Its list now names the two modules that remain (that case passes). No test file was deleted: nothing else was obsolete with confidence, and the brief says not to delete behaviour tests to get green.
2. Public behaviour differs after the Rust port (possible real regressions) 130 Stay quarantined with a precise reason each. Six fixes below moved other files out.
3. Environment-bound or too slow for the PR job 8 Moved to scripts/slow-tests.txt (nightly runs them after test:prepare/build:reference): arrow/ipc, core-ideals/relationship-{extras,postgresql,tablespec-projection}, projections/ddd-postgresql (file killed after 5-10 min, no assertion failure of their own), binding/postgresql-indexes (only failure is its 60 s timeout), domain-packs/download (needs the browser WASM package), actions-reference/process-recovery (needs the action test prerequisites). Whether they pass in nightly is reported below.
Fixed or already passing 6 core/field-ideals, core/nullability-ideals: stale quarantine, they pass (accessor refusal was fixed by 2b32fe1 in #144/#145 after they were listed). domain-packs/schema-browser-embedding: real source bug, fixed. binding/stable-ids, security/evaluate, security/protection/policy-v2: pass after the accessor-refusal helper (below).

Fixes (each cites why the new behaviour is the intended one)

  • docs/helix/05-deploy/schema-browser/static-validators.ts imported generateDomainPackLoaderSchema from src/domain-packs/loader, which Complete the Rust port and prepare UMF 0.10.11 [skip ci] #143 turned into declarations only (moved to the Rust-backed root). Now imports from src/index-node-v4, like scripts/build-domain-pack-loaders.ts. This broke the schema-browser build, not only a test.
  • Accessor refusal. 2b32fe1 ("Assert accessor refusal in both validation entry points", in Add the generated access executor used by prepared PostgreSQL reads #144/Admit prepared PostgreSQL reads and receipt-backed writes #145) established that the Rust admission may either throw AdmissionError('Accessor properties are not invoked during admission') or return an invalid result, and added expectAccessorRefusal for field-ideals/nullability-ideals. Applied the same helper (plus expectAccessorRefusalBy for APIs whose refusal is a status) in binding-uuid, binding/stable-ids, core/key-ideals, core/relationship-ideal, security/evaluate, security/protection/{domains,policy-v2}. Three files pass; the others move on to later, unrelated assertions and keep their quarantine with the new reason.
  • src/extensions/actions/plan-identity.ts was stale on main (Original action-plan producer identity is stale), so bun run test:prepare, the nightly job and the SDK source build all stopped there. Regenerated with bun scripts/actions-plan-identity.ts.
  • .github/workflows/ci.yml: separate concurrency group per trigger, so a nightly/manual run is not cancelled by a later push to the branch.

Findings: possible real regressions (please read)

Roughly by shared cause (counts are files, approximate where a file has several causes):

  1. Rust panic aborts the host process. tests/core-ideals/cardinality-parquet-projection.test.ts: crates/umf-core/src/cardinality_parquet_projection.rs:610 calls Option::unwrap() on None; a panic across the addon boundary aborts Bun (exit 134), taking the caller's whole process down.
  2. YAML writer loses data. core/core.test.ts: the YAML writer emits a block scalar without an indentation indicator, so strings with leading spaces read back changed (writeJsonValue({c:' lead\n second\n '},'yaml') reads back as lead\n second). The same cluster shows as wrongly indented line in block scalar when the PostgreSQL catalog tests read their own YAML (3 files), plus core/native-json (YAML 1.1 underscored integers come back quoted) and json-schema/native.
  3. Unknown meaning no longer blocks completeness / bounds not enforced. binding-uuid (an unknown logical facet no longer makes inspectBinding(...).complete false), actions-manifest (validateBulkOutcome accepts a 129-emoji key module), security/protection/domains (accepts a 4 MB closure document instead of PROTECTION_DOMAIN_BOUND), binding/table-operations (no TABLES_OPERATION_GENERATED etc. diagnostics for invalid declarations), actions/interpreter (strict reference admission no longer throws).
  4. Avro: umf.avro extension reports SEMANTICS_UNCHECKED (warning), which makes inspectAvro(...).complete false for ordinary schemas and makes tests that treat warnings as errors throw (8 files incl. avro/bundle, avro/corpus, core-ideals/relationship-avro, facets-avro).
  5. Number profile: values the old TS engine accepted or diagnosed are now thrown as Number outside the core interoperable numeric profile / Parsing would change the decimal value (7 files: core/javascript-numeric, core/facet-ideals, jsonld/from-rdf, openapi/extraction, core-ideals/key-sqlserver, ...).
  6. Admission/limits behave differently: JSON nesting exceeds the admission limit, Source structural limit exceeded, Serialized text limit exceeded, literal text budget exceeded thrown where tests expect a result (about 13 files).
  7. Status flips in both directions. Refusal became acceptance (Expected "blocked"/"refused", got "projected"/"resolved"/"evaluated") in about 9 files (key-avro-projection, tablespec/avro-integers, linkml/slot-*, shacl/lists, security/protection/{disclosure,query-properties}, security/write); acceptance became refusal (Expected "candidate"/"compiled"/"transformed", got "blocked"/"refused") in about 5 (jsonld/legacy, generalized-rdf/dataset, lifecycle/g01-fill, lifecycle/g11-field-replacement, security/protection/functions).
  8. SQL Server index projection: running tests/sqlserver/indexes.test.ts rewrites fixtures/sqlserver/index-projection.json; the new output has a stray newline inside a diagnostic string (Key "PK_Items"\n (PK) is not enforced...) and reorders nativeSchema. fixtures/postgresql/ddl-declarations.json is likewise rewritten with different content by postgresql/declarations.test.ts.
  9. Blank-node labels (g_15 vs g_14) are not reset per case (owl/list-axioms, owl/declarations, rdf/turtle); this is the same cause as the Rust rdf_parity case already excluded from nextest.

Also found: 50 test files overwrite tracked fixtures (Bun.write('fixtures/...')) every time they run, whether or not they pass; running a quarantined file dirties the checkout. Not changed here.

CI and timing

  • PR CI on the final commit (run 43157274708): all 37 jobs green (size-guard, native-addon, rust-build, rust 1-8, ts 1-24, python 3.11/3.14; nightly skipped by design). An earlier run on the previous commit failed only ts (7): security/protection/policy-v2 passes but one of its tests takes 132 s on the PR job's addon (120 s limit), so it moved to slow-tests.txt.

  • PR job: the 24 ts shards ran 13.4 min median, 15.2 min max in the final run (about 335 job-minutes); python 10.0 / 13.0 min; rust partitions 1.9-8.5 min after a 5.6 min build; about 4-5 min of each is setup (apt, playwright install). I did not change the shard count: more shards only lengthen the runner queue, which dominates wall time on this org (the nightly dispatch sat queued for hours). Nothing was moved out of the PR job to hit a target except the 8 files above.

  • Nightly (gh workflow run ci.yml --ref reset/phase2-tests; ci.yml exists on main, so dispatch is allowed). It had never completed. Dispatches so far, each fixing what the previous one hit: (1) Original action-plan producer identity is stale at bun run test:prepare (regenerated); (2) Could not resolve bindings/core/wasm/pkg/umf_core_wasm.js (the WASM package is now built first, Chromium and Go installed early); (3) build:reference failed on four evidence-map allocations whose records phase 1 deleted, then build:docs on stale Actions-guide excerpts and a guide page linking to deleted evidence (fixed here); (4) setup, WASM, test:prepare (the SDK source build, PostgreSQL WASM, smithy), typecheck, build:reference and build:docs all passed for the first time, then the full Bun inventory hit the 150 minute job limit. Now: the limit is 360 minutes, the quarantined files run as four parallel informational shards (quarantine), every nightly suite runs with !cancelled() so one failure does not hide the rest, and cargo test --workspace skips the single known rdf_parity case. Dispatch 5 (run 43166980553) was still running when this was written: the quarantine shards had mostly finished, the Bun inventory, browser suites, SDK package test and cargo test --workspace had not. Its result is not known yet; whether the 8 files moved to slow-tests.txt pass there is part of that result.

Not in this PR

Fixing the 130 behaviours; deleting any behaviour test; changing shard counts; removing the fixture-writing from the tests; history rewrite, tags, releases.

🤖 Generated with Claude Code

easel and others added 12 commits November 11, 2026 19:50
…d failed test:prepare

bun run test:prepare (and so the nightly job and the SDK source build) stopped at
'Original action-plan producer identity is stale'. Regenerated with
bun scripts/actions-plan-identity.ts.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
#143 moved the loader schema generator to the Rust-backed root and left src/domain-packs/loader.ts as
declarations only, so the static validators build failed with 'Export named generateDomainPackLoaderSchema
not found' (tests/domain-packs/schema-browser-embedding.test.ts). The same import scripts/build-domain-pack-loaders.ts uses.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…d-ideals/nullability-ideals already do

2b32fe1 ('Assert accessor refusal in both validation entry points', in #144/#145) established that the Rust
admission may either throw AdmissionError('Accessor properties are not invoked during admission') or return an
invalid result. Apply the same helper to the other tests that pass getters. policy-v2 passes fully; the others
move on to later, unrelated assertions and stay quarantined with the new reason.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…int list

tests/actions-manifest.test.ts read src/extensions/actions/{manifests,framed-json}.ts, which #143 deleted
when the Rust port took over; the lint now covers the two modules that remain.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…sible regressions, with reasons

- 130 files stay quarantined, each with a one-line reason (a failing assertion or error, tagged with a shared cause).
- 8 files move to slow-tests.txt: no assertion failure of their own, they need more time (relationship
  projection matrices, Arrow IPC, DDD PostgreSQL) or a prepared environment (WASM package, action prerequisites).
- 6 files were fixed or already pass (see the previous commits).
- Remove the temporary triage step from the ts job.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… skip the known rdf_parity case; drop four dead evidence allocations

The nightly job stopped at test:prepare ('Could not resolve bindings/core/wasm/pkg/umf_core_wasm.js').
The same job then runs build:reference, which needed Go and failed on four evidence-map allocations whose
records phase 1 deleted (uuid, typed-reporting, typed-sdk, actions-certificate: those operations now have no
recorded evidence). cargo test --workspace skips the single known-failing rdf_parity case.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The nightly job's build:docs step refuses stale excerpts; refreshed with bun scripts/actions-docs/build.ts --refresh
(the example imports from src/index-node-v4, the V4 root since #143; the public-types list follows the current exports).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…e: no links to deleted evidence

- build:docs failed on 'Missing repository link ../../evidence/...' in support-and-evidence.md. The page cited
  qualification counts from the deleted evidence; it now states what CI tests and points at known-failures.txt.
  Dead links to removed files in the guide pages became plain text.
- The nightly job's suites (Bun, quarantined Bun, browsers, SDK package, cargo test) run with !cancelled(), so one
  failing suite no longer hides the others.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@easel easel changed the title Reset phase 2A: triage the quarantined Bun files Reset phase 2A: triage the 144 quarantined Bun files Nov 12, 2026
easel and others added 2 commits November 11, 2026 23:35
… addon (120 s limit)

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…orkflow_call guard

The last nightly run got through setup, WASM, test:prepare, typecheck and the reference/guide build (first time),
then hit the 150 min limit inside the full Bun inventory. The inventory keeps its single job with a 6 h limit; the
quarantined files (informational) run as four parallel shards from the cached addon. The workflow_call trigger and the
github.workflow guard are the same lines release.yml's pull request adds, so the two merge cleanly.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@easel
easel marked this pull request as ready for review November 12, 2026 06:57
@easel
easel merged commit a1c2b30 into main Nov 12, 2026
80 of 81 checks passed
@easel
easel deleted the reset/phase2-tests branch November 12, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant