feat(validate): catch broken references and show findings on the PR

[scott-lowe-vapi]

Value

V.A.L.U.E. tier: small — a behavior change: validate, and so apply and the Validate resources check, now fail on configs they used to pass.

• Problem: a reference that names no file fails in one of three ways, depending on the field (improvements.md docs: document orphan-YAML gate + --allow-new-files in README and AGENTS #31):

  • it's silently dropped: model.toolIds, artifactPlan.structuredOutputIds;
  • it's sent raw and rejected partway through a push: squad members, hook tools, personalityId, scenarioId;
  • or it's deferred to a linking pass.

  validate never checked references, so the Validate resources check from ci: validate every org's resources on every pull request #76 couldn't catch a typo'd tool name either. Warnings were also invisible in CI: they don't fail the check, and nobody reads the job log.

• Who it affects: everyone who edits resource files by hand or with a coding agent, and reviewers of their PRs.

• What changes:

  • New src/validate-refs.ts, run by validate (so by apply and CI) and by push:

    Rule	Severity	Catches
    dangling-reference	error	A name with no local file and no state entry. Uses the shared reference walk, plus scenario judges' evaluations[].structuredOutputId.
    override-tool-by-name	error	A tool name in toolIds inside assistantOverrides, membersOverrides or targetOverrides, where push never resolves names.
    unresolved-credential	warning	A credential name missing from the state file. The message names the org's bootstrap pull.
    reference-by-uuid	warning	A UUID reference, which breaks promotion. Stock personalities are exempt.

  • validate now also runs reference-to-ignored, as push already did. It reads the committed state file and stays offline.

  • On GitHub Actions, every finding becomes an annotation, so it shows on the file in the PR, warnings included.

  • push reports the new rules alongside its existing validators: warnings by default, blocking under --strict.

  • Docs:

    • a rule-by-rule table in troubleshooting;
    • the validate row in the commands guide;
    • AGENTS.md: never edit the state file to make a reference resolve;
    • improvements.md docs: document orphan-YAML gate + --allow-new-files in README and AGENTS #31 marked resolved by validation.

Evidence of value

The starter example with two typos, scheduler → schedular in the squad and booking-confirmed → booking-confirmd in a judge:

	Before (#76)	After
npm run validate	0 error(s) — ✅ Validation passed	2 error(s), one dangling-reference per typo, naming the file and the missing name

• New tests/validate-refs.test.ts covers:
  • names that resolve to local files and to state;
  • a typo in each reference field;
  • ignored references;
  • UUIDs and stock personalities;
  • all three override keys;
  • credentials as names, as UUIDs, and known to state.
• Annotation format: escaping of %, newlines, : and , is tested in tests/validate.test.ts.
• End to end: tests/ci-validate-workflow.test.ts runs the CI step with GITHUB_ACTIONS=true on the typo'd squad. The step fails, and the ::error points at resources/clinic/squads/front-desk.yml.
• Mutation: dropping the judge collection or the override walk fails two tests.
• Every example org still passes. The cross-org promotion example (which has no state files) now shows an unresolved-credential warning, which is accurate.

Testing plan

• npm test (523 tests) and npx tsc --noEmit pass.
• Behavior to expect: apply validates before it pulls. So a reference to a resource created in the dashboard and never pulled now stops apply; the message says to pull first. Before, apply went on to pull and push, and the reference resolved only if the pull happened to produce that exact name.
• Not tested: a live apply or push. Neither code path changed except for the added findings.
• Not in this PR: a check for secrets committed in resource files. It goes in its own PR, because a false positive there would block a customer's merge.

Refs TEST-141

🤖 Generated with Claude Code

[scott-lowe-vapi]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat: identify every gitops API request with a User-Agent #78
• feat(validate): catch broken references and show findings on the PR #77 👈 (View in Graphite)
• ci: validate every org's resources on every pull request #76
• ci: lint workflows and test the PR check's key-sharing decision #75
• test(docs): check that doc links, commands and flags exist #74
• fix(cleanup): never delete resources matched by .vapi-ignore #73
• docs(agents): one compact guide for Claude, Codex and Cursor #72
• docs: edit the guides for new readers and fix inaccuracies #71
• docs: restructure the README into a short landing page and topic guides #70
• docs: prepare for public release: scrub internal names, fix examples, add community files #69
• feat(promotion): gate promotion out of an org on a passing check #66
• fix(promotion): commit the files of transitions that applied when a later one fails #65
• test(promotion): pin the exact files promoting an assistant writes to the target #68
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.