Repository navigation
A benchmark named in any script, a figure recorded under an alias, and DeepEval's ARC challenge set - #42
Merged
Conversation
…run ARC's challenge set
An accept-when name is everything before the comparison in the Python port too,
as accept.rs reads it, so tau-bench's published alias `τ-bench` is a name in
both (new table rows). benchmarks.record writes under the id of the row the name
answers to, so an alias no longer makes a second candidate row. The shipped arc
row passes `with: {mode: challenge}`, since DeepEval's ARC defaults to the easy
set. The deepeval extra declares datasets and pandas, which DeepEval's
benchmarks import and its package does not declare.
# Conflicts: # CHANGELOG.md # README.md # docs/90-REVIEW.md # site-docs/index.md # site-docs/status/verified.md # spec/accept-when.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A benchmark named in any script, a figure recorded under an alias, and DeepEval's ARC challenge set
This PR has four fixes, each found by reading the code against the catalogue (Bud Meta Harness model-finder notes, §4 items 1, 2, 6 and 7).
accept-when:names in any script, in both ports.selection._LINEaccepted only[A-Za-z0-9 _.()+-], soτ-bench.score > 50was not a comparison in Python, whileaccept.rsreads it (it treats everything before the first>,<or=as the name).τ-benchis the catalogue's own alias for tau-bench. The Python reader now takes the same name.spec/accept-when.yamlgains theτ-bench.score > 50row and the refused> 50row (no name), so both ports are held to them.benchmarks.recordwrites under the row's own id. A figure recorded under an alias (qwen2.5:7b-instruct) created a second workspace row keyed by the alias._layermerges rows by id, so the distribution's row and the alias row both became candidates.recordnow finds the row the name refers to (the workspace's first, then the distribution's) and writes there.arcrow runs the challenge set. DeepEval'sARCdefaults toARCMode.EASY, and the row says "challenge set", so a measured figure was the easy set's. The row now passeswith: {mode: challenge}, which_settings_foralready reads intoARCMode.CHALLENGE.deepevalextra declaresdatasetsandpandas. Every DeepEval benchmark importsdatasetswhen it is built, and MMLU importspandas. DeepEval's own package declares neither, sorun_deepevalfailed on its first line in an environment installed from the extra.uv.lockis updated.Proof
cargo test -p pact-loader --lib accept::(the shared table) andpytest tests/test_an_accept_line_means_one_thing.py: 41 passed.test_both_ports_read_every_way_a_spend_cap_is_written.py, because the worktree had noadapters/typescript/node_modulesfor the TypeScript port; with it installed that file gives 19 passed. The headline counts come fromscripts/sync-counts.sh(5314 tests). New tests: the table rows;test_a_figure_recorded_under_an_alias_lands_on_the_rows_own_id, which fails before because two rows answer to the alias; the ARC mode assertion intest_the_shipped_benchmarks_say_how_each_is_measured; andtest_deepeval_comes_with_what_its_benchmarks_import.spec/accept-when.yaml(both sides kept);scripts/sync-counts.shrewrote the counts (1722 Rust + 3613 adapter), andeverythingpassed on a37968d. After the merge, the spend-cap file and the shared-table and count tests give 95 passed under Node 24; under the shell's Node 20 the TypeScript port exits withnode: bad option: --experimental-strip-types, which is the environment, not this change.Purpose checklist (BMH PACT v97 §0.2)
with:on the shipped catalogue row. No runtime-only setting.spec/accept-when.yamlshared vectors.🤖 Generated with Claude Code