fix(sim): report failed, canceled and incomplete runs instead of a false pass by scott-lowe-vapi · Pull Request #58 · VapiAI/gitops

scott-lowe-vapi · 2026-10-01T22:40:43Z

Value

V.A.L.U.E. tier: project — PR 2 of 10 for inline simulation PR checks (TEST-141). Stacked on #57.

Problem: npm run sim reported every run as passed. It read a results field the simulation-run API doesn't return and counted status === "pass" (items are passed/failed), so it always summarised 0/0 and exited 0, failing runs included.
Who it affects: anyone gating on npm run sim, locally or in CI. The PR check (later in this stack) reuses this verdict, so it has to be right.
What changes:
- A strict verdict (src/sim-result.ts).
- Item fetching that handles both response shapes and late results.
- The run link printed, plus each failing judge with expected vs extracted values.
- --timeout and Ctrl-C both cancel the run.
- Exit codes: 0 passed, 1 failed, 2 usage, 3 incomplete.
- A config-free client that never retries run creation on a 5xx, because the run may already be queued.

Evidence of value

Same stub API, one passed and one failed item:

	Result
`main`'s `sim.ts`	`{"pass":0,"fail":0}` → exits 0 (false green)
This branch	`failed — 1 of 1 simulations failed` → exits 1

Live, against a test org (chat transport):

Suite	Exit	Output
Designed to fail (judge: "open 24 hours?")	1	`✗ … open-24h (expected = true, got false)` — run
Designed to pass (judge: "open 8–5 on Fridays?")	0	`passed — 1 of 1 simulations passed` — run

The temporary resources were deleted afterwards.

Testing plan

tests/sim-result.test.ts is a verdict table covering:
- the old false-green shape (no results);
- 0 items, a short item list, and a count mismatch;
- a failed item, with the failing judge listed;
- canceled items;
- all required evaluations skipped, and an optional skip alongside a scored required judge;
- missing itemCounts, and a run that hasn't ended.
tests/sim-run.test.ts runs runSimulation against a local HTTP stub:
- pass, and fail using the bare-array item shape;
- late item results;
- timeout cancels the run, and an interrupt cancels it with the 400 "already ended" swallowed;
- no retry of a 502 on create;
- --no-watch;
- pagination with overlapping pages deduped.
npm run build and npm test pass (377 tests).
Not tested: a live run that's still running when it's canceled (cancel was only exercised against the stub), and voice transport (the live runs used chat). Default behaviour change: unknown CLI arguments are now an error (exit 2) instead of being silently ignored.

Refs TEST-141

🤖 Generated with Claude Code

scott-lowe-vapi · 2026-10-01T23:51:37Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

scott-lowe-vapi · 2026-10-03T05:59:29Z

Merge activity

Oct 3, 5:59 AM UTC: A user started a stack merge that includes this pull request via Graphite.
Oct 3, 6:01 AM UTC: Graphite rebased this pull request as part of a merge.
Oct 3, 6:01 AM UTC: @scott-lowe-vapi merged this pull request with Graphite.

…lse pass `npm run sim` read a `results` field the simulation-run API doesn't return and counted `status === "pass"` (items are `passed`/`failed`), so every watched run summarised as 0/0 and exited 0, including failing runs. - src/sim-result.ts: strict, pure verdict. Passed only when the run ended, every expected item exists and passed, and each had a required evaluation that was actually scored; otherwise failed or incomplete with a reason. - src/sim.ts: read run items (paginated or bare array, deduped by id), wait for late item results, print the run link from the create response and the failing judges, cancel the run on --timeout (default 20 min) or Ctrl-C. - src/vapi-client.ts: config-free client; run creation is never retried on a 5xx, since the run may already be queued. - src/sim-cmd.ts: exported simCommandRun; exit 0 passed, 1 failed, 2 usage, 3 incomplete; --no-watch prints the link and exits 0. - Docs: run/item shapes and the verdict in docs/learnings/simulations.md, sim rows in README and AGENTS.md, improvements.md #33. Refs TEST-141 Co-Authored-By: Claude Opus 5.5 <[email protected]>

…g-free modules (#59) ## Value **V.A.L.U.E. tier:** project — PR 3 of 10 for inline simulation PR checks ([TEST-141](https://linear.app/vapi/issue/TEST-141/gitops-run-simulation-suites-against-pr-changes-inline-as-ci-checks)); this PR is a behaviour-preserving refactor. - **Problem:** the PR check (PRs 4–7) has to read an org's resource files and resolve org API keys. All the code that does that sits behind `config.ts`, which parses `argv`, binds one org and can `process.exit` at import time. Copying it into the check would let push and the check drift apart, so the check could test a different payload from the one push sends. - **Who it affects:** gitops users, whose PR checks must test exactly what push would deploy; and maintainers, who get one copy of the loader instead of two (three for the `.md` system-prompt injection). - **What changes:** code moves into two config-free modules. `resources.ts`, `config.ts` and `promote-cmd.ts` delegate to them and keep their exports, logs and error messages. - **`src/resource-parse.ts`:** - from `resources.ts`: `FOLDER_MAP`, `VALID_EXTENSIONS`, frontmatter/file parsing, the directory scan and the loader; - from `config.ts`: `.vapi-ignore` reading and matching; - new: `orgResourcesRead(rootDir, org)`, which returns every resource of an org keyed `type:id`; - the three copies of the `.md` body → system message injection become `markdownResourceParse`. - **`src/org-connection.ts`:** `envValue`, `tokensParse(envName)`, `connectionLoad` and `childRun`, moved from `promote-cmd.ts` and parameterised on the root dir, the token variable and the base URL. - **`api.ts`:** takes `VapiApiError`, `parseApiMessage`, `shouldRetry` and the backoff constants from `vapi-client.ts`, and re-exports `VapiApiError`. Both clients now share one error class (so `instanceof` checks see one class) and one retry rule. - **`promotion.ts`:** - imports `FOLDER_MAP` / `VALID_EXTENSIONS` instead of keeping its own copies; - exports `SLUG_RE`, `promotionBindingsParse`, `promotionBindingsResolve`, `promotionBindingsApply` and `PromotionBindingsResolved`, for the check config (PR 4) and the payload builder (PR 5). - Also dropped: the unused `FOLDER_TO_TYPE` map in `resources.ts`. ## Evidence of value **The output is byte-identical before and after.** The same fixture org was run at both refs against a local stub API. Nothing reached a real org. The fixture: - an `.md` assistant whose frontmatter has a system message the body must replace; - a nested assistant; - an ignored assistant; - a backup copy and an unsupported file; - a `.ts` tool and a YAML tool; - a squad, a structured output and all four simulation types; - a two-org `promotion.yml`. | Command | `fix/sim-false-green` (06a1492) | This branch (0518af1) | |---|---|---| | `push ev-dev --dry-run` (new-file gate refusal) | identical | identical | | `push ev-dev --dry-run --allow-new-files` (full plan, every would-POST) | identical | identical | | `loadResources` JSON for every type (full parsed data, incl. the `.md` system message) | identical | identical | | `validate ev-dev` | identical | identical | | `promote --pipeline release --from ev-dev --to ev-prod` (plan) | identical | identical | | **Combined stdout + stderr + stub request log** | sha256 `3f3b609f…1d3b`, 14,927 bytes | sha256 `3f3b609f…1d3b`, 14,927 bytes | - Dry-run placeholder IDs (`dry-run-post-<Date.now()>`) and the stub's port are normalised before hashing. - `--allow-new-files` is passed only because every fixture file is new by construction. **Tests:** `npm test` goes from 377 to 398 passing (21 new), and `npm run build` (src + tests) is clean. ## Testing plan - **`tests/resource-parse.test.ts`** (temp-dir fixtures) covers: - `.md` body replacing a frontmatter system message, and an empty body; - missing frontmatter; - `.md` parsing being the same for `parseResourceDataFromFile` and the loader; - sorted order across `.yml`/`.yaml`/`.ts` and nested dirs; - hidden and `.bkp` files skipped; - duplicate IDs refused, with the exact YAML-not-an-object message; - missing directory; - `.vapi-ignore` comments, blanks and `!`; - `*` vs `**` vs `?` matching; - `orgResourcesRead` reading one org and applying its ignore file, and the caller's override. - **`tests/org-connection.test.ts`** covers: - plain and quoted `.env` values; - token-map parsing, with the variable named in every error; - token precedence (map, then `.env.<org>`) and base URL precedence (configured, then `.env.<org>`); - the missing-token error; - `childRun` passing the org and key to a real child, and dropping an inherited `VAPI_BASE_URL`; - a failing child. - The existing push dry-run, `.vapi-ignore` push, promotion, audit and cleanup-safety suites pass unchanged. - **Not tested:** - a push against a real org (the comparison uses a stub API that returns empty lists, so update/PATCH paths for existing resources aren't exercised; the code they call is unchanged); - `promote --apply` (the moved `connectionLoad`/`childRun` are covered by unit tests, but no live child `pull`/`apply` was run); - Node 20 locally (left to CI). Stacked on #58. Review with `git diff --color-moved=dimmed-zebra fix/sim-false-green...` to see that most of the diff is moved lines. Refs TEST-141 🤖 Generated with [Claude Code](https://claude.com/claude-code)

scott-lowe-vapi mentioned this pull request Oct 1, 2026

refactor: move resource parsing and org connection helpers into config-free modules #59

Merged

scott-lowe-vapi force-pushed the fix/sim-false-green branch from 8a438bc to 06a1492 Compare October 1, 2026 23:11

scott-lowe-vapi marked this pull request as ready for review October 1, 2026 23:52

scott-lowe-vapi assigned dhruva-vapi, vtkovapi and scott-lowe-vapi and unassigned vtkovapi and dhruva-vapi Oct 1, 2026

scott-lowe-vapi requested review from dhruva-vapi and vtkovapi October 1, 2026 23:54

vtkovapi approved these changes Oct 3, 2026

View reviewed changes

scott-lowe-vapi changed the base branch from ci/test-workflow to graphite-base/58 October 3, 2026 05:59

scott-lowe-vapi changed the base branch from graphite-base/58 to main October 3, 2026 05:59

scott-lowe-vapi force-pushed the fix/sim-false-green branch from 06a1492 to 663612e Compare October 3, 2026 06:00

scott-lowe-vapi merged commit e66f1e7 into main Oct 3, 2026
2 checks passed

Navigation Menu

fix(sim): report failed, canceled and incomplete runs instead of a false pass - #58

fix(sim): report failed, canceled and incomplete runs instead of a false pass#58
scott-lowe-vapi merged 1 commit into
mainfrom
fix/sim-false-green

scott-lowe-vapi commented Oct 1, 2026

Uh oh!

scott-lowe-vapi commented Oct 1, 2026 •

edited

Loading

Uh oh!

scott-lowe-vapi commented Oct 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

scott-lowe-vapi commented Oct 1, 2026

Value

Evidence of value

Testing plan

Uh oh!

scott-lowe-vapi commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scott-lowe-vapi commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

scott-lowe-vapi commented Oct 1, 2026 •

edited

Loading

scott-lowe-vapi commented Oct 3, 2026 •

edited

Loading