Skip to content

A benchmark named in any script, a figure recorded under an alias, and DeepEval's ARC challenge set - #42

Merged
jithinAB merged 4 commits into
masterfrom
evals/reading-fixes
Nov 17, 2026
Merged

jithinAB merged 4 commits into
masterfrom
evals/reading-fixes

Conversation

@jithinAB

@jithinAB jithinAB commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

A benchmark named in any script, a figure recorded under an alias, and DeepEval's ARC challenge set

This PR has four fixes, each found by reading the code against the catalogue (Bud Meta Harness model-finder notes, §4 items 1, 2, 6 and 7).

  • accept-when: names in any script, in both ports. selection._LINE accepted only [A-Za-z0-9 _.()+-], so τ-bench.score > 50 was not a comparison in Python, while accept.rs reads it (it treats everything before the first >, < or = as the name). τ-bench is the catalogue's own alias for tau-bench. The Python reader now takes the same name. spec/accept-when.yaml gains the τ-bench.score > 50 row and the refused > 50 row (no name), so both ports are held to them.
  • benchmarks.record writes under the row's own id. A figure recorded under an alias (qwen2.5:7b-instruct) created a second workspace row keyed by the alias. _layer merges rows by id, so the distribution's row and the alias row both became candidates. record now finds the row the name refers to (the workspace's first, then the distribution's) and writes there.
  • The shipped arc row runs the challenge set. DeepEval's ARC defaults to ARCMode.EASY, and the row says "challenge set", so a measured figure was the easy set's. The row now passes with: {mode: challenge}, which _settings_for already reads into ARCMode.CHALLENGE.
  • The deepeval extra declares datasets and pandas. Every DeepEval benchmark imports datasets when it is built, and MMLU imports pandas. DeepEval's own package declares neither, so run_deepeval failed on its first line in an environment installed from the extra. uv.lock is updated.

Proof

  • cargo test -p pact-loader --lib accept:: (the shared table) and pytest tests/test_an_accept_line_means_one_thing.py: 41 passed.
  • Full adapter suite on this branch: 3424 passed and 145 skipped. The only failures were test_both_ports_read_every_way_a_spend_cap_is_written.py, because the worktree had no adapters/typescript/node_modules for the TypeScript port; with it installed that file gives 19 passed. The headline counts come from scripts/sync-counts.sh (5314 tests). New tests: the table rows; test_a_figure_recorded_under_an_alias_lands_on_the_rows_own_id, which fails before because two rows answer to the alias; the ARC mode assertion in test_the_shipped_benchmarks_say_how_each_is_measured; and test_deepeval_comes_with_what_its_benchmarks_import.
  • Merged master (A measured figure says how it was measured, and every runtime holds a line to its interval #41, A runtime may serve a workspace without its broken agents (--leave-out-broken-agents) #44) on 2026-11-17: conflicts only in the counts, CHANGELOG and spec/accept-when.yaml (both sides kept); scripts/sync-counts.sh rewrote the counts (1722 Rust + 3613 adapter), and everything passed on a37968d. After the merge, the spend-cap file and the shared-table and count tests give 95 passed under Node 24; under the shell's Node 20 the TypeScript port exits with node: bad option: --experimental-strip-types, which is the environment, not this change.

Purpose checklist (BMH PACT v97 §0.2)

  1. P1 Declared: with: on the shipped catalogue row. No runtime-only setting.
  2. P2 No hidden divergence: the Python and Rust grammars agree on non-ASCII names.
  3. P3 Honest gaps: no new fields.
  4. P4 Host choices recorded: not applicable.
  5. P5 Model-visible ⟺ logged: not applicable.
  6. P6 Cross-runtime evidence: new rows in the spec/accept-when.yaml shared vectors.
  7. P7 Versioned and checked: not applicable.
  8. P8 Reusable: catalogue and recorder fixes hold for any workspace.
  9. P9 Platform rules: not applicable (upstream).
  10. P10 Justified: portability. One grammar in every port, one row per model, and figures measured on the set the row names.
  11. P11 Lane hygiene: one change, on master.

🤖 Generated with Claude Code

…run ARC's challenge set

An accept-when name is everything before the comparison in the Python port too,
as accept.rs reads it, so tau-bench's published alias `τ-bench` is a name in
both (new table rows). benchmarks.record writes under the id of the row the name
answers to, so an alias no longer makes a second candidate row. The shipped arc
row passes `with: {mode: challenge}`, since DeepEval's ARC defaults to the easy
set. The deepeval extra declares datasets and pandas, which DeepEval's
benchmarks import and its package does not declare.
# Conflicts:
#	CHANGELOG.md
#	README.md
#	docs/90-REVIEW.md
#	site-docs/index.md
#	site-docs/status/verified.md
#	spec/accept-when.yaml
@jithinAB
jithinAB marked this pull request as ready for review November 17, 2026 00:15
@jithinAB
jithinAB merged commit 45715d1 into master Nov 17, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant