Repository navigation
feat(scenarios): the scenario scorer moves into @objectstack/scenarios and judges capability tags, agent-action exposure, write outcomes and per-persona answers - #30594
Conversation
…na run grading Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu Co-authored-by: Claude <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu Co-authored-by: Claude <[email protected]>
An ablation showed record-triggered-flow-exists could never decide a verdict on its own: a build with no such flow already fails the binding check. Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu Co-authored-by: Claude <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu Co-authored-by: Claude <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu Co-authored-by: Claude <[email protected]>
📓 Docs Drift CheckThis PR changes 2 package(s): 39 hand-written doc(s) name something this change touched — list omitted above 15 rows. Re-derive on the tree named below: ⛔ 13 release-owned page(s) also affected — read-only, see AGENTS.md Documentation Guardrails. What this run could not see
Coarse fallback — 35 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin cf781c2383720f9402a6355dac14b1692ef81b70 && git checkout cf781c2383720f9402a6355dac14b1692ef81b70
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin a2437b560bd8872feece6ec8b57c6fdf21cff1f4 4dd3edc52bdfe115b3b86dce74a3b31df4450e4a && git checkout -B drift-repro a2437b560bd8872feece6ec8b57c6fdf21cff1f4 && git merge --no-ff 4dd3edc52bdfe115b3b86dce74a3b31df4450e4a
node scripts/docs-audit/affected-docs.mjs --json a2437b560bd8872feece6ec8b57c6fdf21cff1f4
|
…enario-tag-checks
…int into @objectstack/scenarios Pure rename, so history follows the files; imports are rewired in the next commit. Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu Co-authored-by: Claude <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu Co-authored-by: Claude <[email protected]>
…liases Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu Co-authored-by: Claude <[email protected]>
…eset for the moved scorer The unreleased CLI changeset for the scorer port is folded in: the modules it described no longer ship from @objectstack/cli. Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu Co-authored-by: Claude <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu Co-authored-by: Claude <[email protected]>
|
epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu Contract review — PR #30594Head-sha: 4dd3edc This is the record of an isolated, read-only contract-tier reviewer the epic seat ran on this head (the seat posts it; the reviewer wrote nothing).
The deletion loses no true user-facing statement: every still-true fact of the old note is restated in the new one. |
|
epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu 席位核对Verdict: ACCEPT at head
|
Fixes #30291
Clause-②: no
Metadata-protocol: none
维护者速读
packages/cli迁入私有包@objectstack/scenarios(裁决 D)。新增封闭的能力标签检查表、Agent 动作暴露检查、写操作结果判分和按角色判分,并提供给运行器用的入口@objectstack/scenarios/score。MetadataEvalCase类型挪到corpus.ts并原名重导出,行为不变。评分器此前没有从 CLI 发布过。回滚即撤销本 PR。packages/qa/scenarios/src/score/scenario-tags.ts(标签表与动作暴露判定)、packages/qa/scenarios/src/score/index.ts(运行器入口)、.changeset/30290-scenario-scorer-port.md的删除(见下方"需要你做的")。pnpm --filter @objectstack/scenarios test,预期 11 个文件、163 个用例全部通过。.changeset/30290-scenario-scorer-port.md。它写的是"CLI 新增评分器",迁移后这句话已不成立。check-empty-changeset对这类有意更正会一直保持红色,直到有人确认。What changes
Per the maintainer's ruling D on #30291 (「同意 D」), the deterministic scenario scorer lives in the private
@objectstack/scenariospackage (packages/qa/scenarios), beside the card format it judges. No model call anywhere (cost ruling on objectstack-ai/cloud#3600: 「目前阶段尽量不要用云端的token,api的更贵」).Moved with
git mv, so history follows:scenario-score,scenario-graphandscenario-run-grade(merged in feat(cli): port the deterministic scenario scorer — shape score, artifact graph reader and Run grader — with a discrimination self-test #30332),scenario-tagsandscenario-operation-grade, and their tests. They go frompackages/cli/src/lint/topackages/qa/scenarios/src/score/andtest/score/.@objectstack/cliends with no scenario code and no mirrored card types.The graders take the card schema's own types:
ScenarioExpect,z.inferofOutcomeSchemaandPostStateSchema, aPickofScenarioCard, and the loader'sTruth. Theexpect, outcome, persona, agent-action and truth mirrors are gone.@objectstack/scenarios/score(private subpath) is the runner's entry (Scenario library T3/T4: local Claude Code runner — Build, Run (问数 + operations per persona over the app's MCP), Iterate — writing run records, zero cloud-billed tokens #30292). It has three functions, and each takes a card the loader returned:scoreCardArtifact(card, artifact);gradeCardOperation(card, operationId, observation);gradeCardAnswer(card, { question, persona?, turn }). The persona's hidden objects and fields are read off the card.Capability-tag checks (
scenario-tags.ts): a closed table of nine rows, listed in the package README for the mid-tier card authors.unscored, never a pass. So is a row whose card gives it nothing to judge. Both are counted nowhere.resolveFlowTriggerKind,APPROVAL_NODE_TYPE,SINGLE_OPTION_TYPES,summaryFunctionReadsField,effectiveSharingModel,standaloneActionOwnerKey). Collections are read throughresolveStackCollectionfrom its owner,@objectstack/core.Agent-action exposure: each declared action must exist on its object, carry
ai.exposed: true(the MCP doors' opt-in) and, where the card saysconfirm, carryai.requiresConfirmation: true(the flag the confirmation gate enforces, not the delete-mode listing heuristic). A missing action fails all of its checks.Write-operation grader:
where, and the audit actor matches;ErrorCode, with zero writes;An unobserved fact fails.
Per-persona Run grading:
withheldtruth is must-not-reveal: a decline is correct, and a stated all-rows figure is a leak;Dependencies of the private package:
@objectstack/core,@objectstack/objectql,@objectstack/plugin-sharing(workspace). Its vitest config aliases each to source (check:test-source-alias).Gates re-homed:
@objectstack/cli#testto the scenarios entry and task, which also declarepackages/cli/src/lint/corpus.ts(the golden builds and the drift pin read it);second-object-read-exposurepin row forscenario-graph.tsfollows the file (samenot-a-served-readclass).CLI:
MetadataEvalCaseis declared incorpus.ts, beside the corpus it types, andmetadata-eval.tsre-exports it under the same name. That keeps the scenarios type-check program from pulling in the whole lint command.Changesets: one changeset (
@objectstack/scenariosand@objectstack/cli, patch). The unreleased.changeset/30290-scenario-scorer-port.mdis deleted on purpose: it announces a scorer in the CLI that this PR moves out, so it is a deliberate correction of a pending release note.check-empty-changesetstays red on it until a person confirms.Body item 2 (
os lint --evalreads cards) is cancelled by the ruling.Tests
pnpm --filter @objectstack/scenarios test: 11 files, 163 tests. Moved suites unchanged in what they assert. Addedtest/score/entry.test.ts(the runner entry over the realcrm_account_with_contactscard).scenario-cards.test.tsloads the control cards through the loader:expectmust equal whatcorpus.tsderives, with a floor of exactly 3 controls;pnpm --filter @objectstack/scenarios typecheck: exit 0.pnpm --filter @objectstack/core exec vitest run src/security/second-object-read-exposure.pin.test.ts: 4/4.pnpm --filter @objectstack/cli typecheck, build, andvitest run test/metadata-eval.test.ts src/lint/: 43/43.Acceptance notes
travel_expense.json(merged meanwhile) declares 12 tags. Five score:master-detail-inline-edit,roll-up-summary,select-options,automation.record-triggered-flowandaccess-security.field-permissions. The rest areunscored:records-forms.formula-field,records-forms.validation-rule,approvals.multi-step-approval,access-security.row-level-security,dashboards.report,ai.ask-dataandai.agent-actions. The README table is what the remaining five card authors align to, or extend.dispatch-gatesreads 4324 changed lines: it counts with--no-renames, so every moved file counts twice. The rename-aware count is 1806, and that is how GitHub counts. Two controls: refactor(lint): move the i18n translatable-key walk from the CLI into @objectstack/lint #30375 is 6567 no-renames vs 1561 rename-aware vs GitHub 1561; refactor(ui)!: fold the providers upload context into fields and data-objectstack, retire @object-ui/providers #30550 is 3228 vs 2482 vs GitHub 2482.scripts/cross-package-test-inputs.mjsandturbo.jsonare recorded as accepted on Scenario library T2b: capability-tag checks in the ported scorer +os lint --evalreads scenario cards #30291. They now point at the scenarios package.Verification
One ablation per moved module, at the new home (scripts/ablation-replace.mjs; tests import src relatively, so no dist leg)
Every mutation landed (anchor 1 → 0) and was restored (blob == HEAD,
git diff HEADempty):The full per-check battery of the previous round (18 mutations) ran against the same assertions before the move.
Gates
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commandson the patch-round diff: 91 commands, all run, exit codes captured before any pipe.check-empty-changeset --base origin/main: exit 1, the deliberate-correction refusal on30290-scenario-scorer-port.md(above).check:dual-build-cjs-loadsandcheck:i18n-coverage: exit 3PREREQUISITE NOT MET. They read every package's built output; NOT MEASURED, CI owns them.check:cross-package-test-inputs,check:ci-filter-parity,check:turbo-task-graph,check:undeclared-dep-imports,check:test-source-aliasandcheck:workspace-manifest-cycles.dist/index.d.ts,dist/console.d.tsanddist/hook-body.d.tsreaches 39 files, none a scenario file, and no bare@objectstack/scenariosspecifier.dist/lint/holds onlycorpus,hook-body-lowering,metadata-evalandscore.!in the title and no(narrowing)arm.Generated by Claude Code