Skip to content

feat(check): parse vapi-checks.yml check configuration - #60

Merged
scott-lowe-vapi merged 1 commit into
mainfrom
feat/check-config
Oct 3, 2026
Merged

feat(check): parse vapi-checks.yml check configuration#60
scott-lowe-vapi merged 1 commit into
mainfrom
feat/check-config

Conversation

@scott-lowe-vapi

@scott-lowe-vapi scott-lowe-vapi commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Value

V.A.L.U.E. tier: project — PR 4 of 10 for inline simulation PR checks (TEST-141); this PR adds a config parser with no caller yet.

  • Problem: a customer needs one place to say which agents to test, with which suites, in which org, and how strictly. promotion.yml can't hold it: promotionConfigParse requires a pipeline of two or more orgs, and most customers have one.
  • Who it affects: gitops users setting up PR checks, single-org and CI-org alike. The PR check (PRs 5–8) and the promotion gate (PR 10) both read this file, so the gate runs the same tests as the PR check.
  • What changes:
    • src/check-config.ts parses a root vapi-checks.yml into typed check definitions:
      • org, plus an optional runOrg and baseUrl;
      • targets (assistants/<id> or squads/<id>, nested IDs allowed);
      • suites and/or simulations;
      • bindings (promotion's shape, via promotionBindingsParse);
      • extra paths;
      • run settings with repo-wide defaults: transport chat, 1 iteration, 20 min, toolMocks: strict, stripWebhooks: true.
    • vapi-checks.example.yml is the commented starting point: a single-org check, plus a commented CI-org check.
    • Every key is validated, and unknown keys fail, so a typo can't silently run fewer tests. A mode: key fails with a message explaining that checks always build from the branch, since a "run what's deployed" mode was deliberately left out.

Evidence of value

Check Result
Valid configs Minimal check (defaults applied, runOrg = org, credentials bind / phones omit); default and per-check overrides; a CI-org check with EU baseUrl and bindings
Rejections 31 invalid shapes, each asserted against its exact message naming the field (table in tests/check-config.test.ts)
Shipped example Parses as-is, and the commented CI-org check parses once uncommented, so the docs can't drift from the parser
npm test 398 → 404 passing; npm run build clean

Testing plan

  • tests/check-config.test.ts covers:
    • parse results for valid configs;
    • one table row per rejection: version, empty or unknown keys at each level, mode, slugs, target shape, extension and .. in IDs, duplicates, missing suites/simulations, each setting's range, baseUrl, paths, bindings;
    • the example file;
    • checksConfigLoad, which returns null when no vapi-checks.yml exists (checks are opt-in).
  • Not tested: nothing uses the config yet. Resolving targets and suites against real files is PR 5, and the baseUrl fallback order (.env.<runOrg>, $VAPI_BASE_URL, api.vapi.ai) is applied at run time in PR 7.

Stacked on #59.

Refs TEST-141

🤖 Generated with Claude Code

@scott-lowe-vapi
scott-lowe-vapi force-pushed the refactor/config-free-engine-modules branch from 3e0e277 to 0518af1 Compare October 1, 2026 23:11
This was referenced Oct 1, 2026
Merged
Merged
Merged
Merged
@scott-lowe-vapi Graphite App

scott-lowe-vapi commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

This was referenced Oct 1, 2026
Merged
Merged
Merged
Open
@scott-lowe-vapi
scott-lowe-vapi marked this pull request as ready for review October 1, 2026 23:52
@scott-lowe-vapi Graphite App

scott-lowe-vapi commented Oct 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Merge activity

  • Oct 3, 5:59 AM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Oct 3, 6:05 AM UTC: Graphite rebased this pull request as part of a merge.
  • Oct 3, 6:05 AM UTC: @scott-lowe-vapi merged this pull request with Graphite.

@scott-lowe-vapi
scott-lowe-vapi changed the base branch from refactor/config-free-engine-modules to graphite-base/60 October 3, 2026 06:02
@scott-lowe-vapi
scott-lowe-vapi changed the base branch from graphite-base/60 to main October 3, 2026 06:03
@scott-lowe-vapi @claude
feat(check): parse vapi-checks.yml check configuration
c9a6467 
Add the config contract for inline simulation PR checks: a root
vapi-checks.yml declaring, per check, the org whose files to read, an
optional run org and base URL, the targets (assistants/<id> or
squads/<id>), the suites and simulations to run, credential and phone
bindings (promotion.yml's shape), extra trigger paths, and run settings
(transport, iterations, timeout, tool-mock policy, webhook stripping)
with repo-wide defaults.

It is a separate file from promotion.yml because single-org customers
can't have a valid promotion pipeline. Unknown keys are rejected,
including `mode`: checks always build the target from the branch.

Nothing reads the config yet; the dry-run CLI that uses it follows.

Refs TEST-141

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@scott-lowe-vapi
scott-lowe-vapi merged commit 8da229f into main Oct 3, 2026
3 checks passed
scott-lowe-vapi added a commit that referenced this pull request Oct 3, 2026
@scott-lowe-vapi
feat(check): build inline simulation-run payloads and add a dry-run C…
d92189b 
…LI (#61)

## Value

**V.A.L.U.E. tier:** project — PR 5 of 10 for inline simulation PR checks ([TEST-141](https://linear.app/vapi/issue/TEST-141/gitops-run-simulation-suites-against-pr-changes-inline-as-ci-checks)); this PR adds an offline dry run, and nothing is sent.

- **Problem:** to test a PR's changes without deploying them, the check has to turn the branch's files into one inline `POST /eval/simulation/run` body: the target with every tool, handoff and structured output, plus every scenario, judge and personality. It has to assemble that body the way push and the runtime would, or the check tests something other than what ships.
- **Who it affects:** gitops users, who can now see exactly what a check would send (`npm run check -- core --dry-run --print-payload`) before any minutes are spent. Every later PR (mock policy, live runs, workflow, promotion gate) builds on this payload.
- **What changes:**
  - **`src/check-payload.ts`** (with `check-payload-assistant.ts` and `check-payload-refs.ts`) builds the body from `orgResourcesRead` and the org states. It's pure, makes no network calls, and collects every problem so one dry run reports them all.
    - **Tools:**
      - runtime tool order: `model.tools`, then `toolIds`, then `toolRefs`. This is what `callAssistantsGet` does, and the parity run's likely `endCall` difference came from getting it wrong;
      - `##` comments stripped, and UUID references resolved through state;
      - a `toolRefs` pin wins over a duplicate `toolIds` entry, with a warning that the version pin is ignored;
      - `knowledgeBase` tools are kept by run-org UUID, since the API refuses them inline.
    - **Squads:** members are inlined (from a file, by UUID, or inline), and handoffs to members switch from ID to member name.
    - **Assistants:**
      - hook `do[].toolId` becomes an inline tool;
      - `artifactPlan.structuredOutputIds`, plus structured outputs that link the assistant through their own `assistant_ids`, are inlined;
      - linkage and server fields are stripped.
    - **Simulations:**
      - suites and simulations are deduplicated into entries with unique names of at most 80 characters;
      - judges' `structuredOutputId`s are inlined;
      - local personalities are inlined, and stock ones pass by ID.
    - **Credentials** are bound by name to the run org's UUIDs using promotion's binding policy (`bind` / `omit`).
    - **Fails the build, naming the field:**
      - a missing file;
      - a handoff leaving the target, or an assistant target that hands off;
      - legacy `assistantDestinations` by ID;
      - members without a name, or with the same name;
      - two inlined tools with the same type and name, which stored would keep but inline would silently drop;
      - tools referenced by ID inside overrides (under strict mocks);
      - over chat: audio judges, scenario hooks, or no required text judge;
      - any reference still a name after the build (the backstop for #31's silent drops);
      - a body over 4.5 MB.
    - **Warns** when text mentions `handoff_to_…` but a handoff is auto-named, because generated names differ inline and stored.
  - **`src/check-cmd.ts`:** `npm run check -- <check>|--all --dry-run [--print-payload [dir]]`.
    - Offline: no API key needed, and it honours `VAPI_GITOPS_ROOT`.
    - Exits 0 when every payload builds, and 2 on a usage, config or build error.
    - Without `--dry-run` it exits 2; live runs land in PR 7.
  - `package.json` gets a `check` script, and the README and AGENTS.md command tables get `npm run check` rows.
  - New fixture `tests/fixtures/check-parity/`: the TEST-141 parity squad written as gitops files.
- **Deliberately not here:** the fail-closed mock policy (dead servers, default error mocks, the tool-type allowlist), which is the next PR. Until then the payload carries tools as written, and there's no live path.

## Evidence of value

**The builder reproduces the payload that scored 15/15 in the parity run.**
- **How:** the dental squad from the 2026-10-01 parity experiment was written as gitops files (`.md` assistants with `toolIds`, a handoff tool by `assistantId`, judges by `structuredOutputId`). `npm run check -- core --dry-run --print-payload` was run on it, and the output was diffed against the inline body the experiment sent, rebuilt from `parity.mjs`.
- **Result:** the only differences are these.

| Path | Experiment sent | Built from files | Why |
|---|---|---|---|
| `members[*].assistant.model.tools` order | `[lookup_patient, handoff, endCall]`, `[check_availability, endCall]` | `[endCall, lookup_patient, handoff]`, `[endCall, check_availability]` | **Intended:** the runtime order of the stored arm (`model.tools` then `toolIds`). Same tools, byte-for-byte, order aside |
| `squad.name`, `personality.name` | `inline`, `caller` | `Bright Smile Dental`, `Dental caller` | Fixture names |
| `iterations`, `transport` | 5, (API default) | 1, `vapi.webchat` | `vapi-checks.yml` defaults |

- Every prompt, judge, tool mock, tool definition and handoff destination is identical. The handoff by `assistantId: scheduler` came out as `assistantName: "Scheduler"`, exactly what the experiment sent.

**Tests:** `npm test` goes from 404 to 430 passing (26 new), and `npm run build` is clean.

## Testing plan

- **`tests/check-payload.test.ts`** (21 tests, temp-dir fixtures):
  - the parity fixture end to end: tool order, member-name handoffs, `.md` prompt, inline judges, entries, transport;
  - `.md` body as the only system message;
  - missing `toolIds`, and `toolIds` by UUID with server fields stripped;
  - the `toolRefs` pin and warning;
  - `knowledgeBase` kept by UUID, and failing with no UUID;
  - duplicate tool names;
  - hook tools;
  - structured outputs from `artifactPlan` and from `assistant_ids` (by slug and by UUID);
  - an assistant target that hands off;
  - squad members by file, UUID and inline;
  - every squad problem reported at once;
  - strict vs off overrides;
  - credentials bound, omitted and missing;
  - a leftover reference, with free-form `parameters` ignored;
  - the chat rules, and voice allowing them;
  - stock and local personalities, name truncation and dedupe;
  - missing suites, simulations, scenarios, personalities and targets;
  - the auto-handoff warning;
  - the 4.5 MB limit.
- **`tests/check-cmd.test.ts`** (5 tests):
  - a dry run with `--print-payload`;
  - `--all` with one broken check exiting 2;
  - no config exiting 0;
  - usage, selection, config and live-mode errors exiting 2;
  - a missing state file as a warning.
- **Not tested:**
  - **Any live run:** nothing is sent until PR 7, so API acceptance of a built body is shown only for the parity fixture (the experiment's 201).
  - **Real customer squads:** override merge semantics and integration tool shapes are untested.
  - Phone-number bindings beyond the unit-tested promotion helper.
  - EU base URLs.

Stacked on #60.

Refs TEST-141

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vtkovapi vtkovapi vtkovapi approved these changes
@dhruva-vapi dhruva-vapi Awaiting requested review from dhruva-vapi

Assignees

@scott-lowe-vapi scott-lowe-vapi

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@scott-lowe-vapi @vtkovapi @dhruva-vapi