Skip to content

feat: identify every gitops API request with a User-Agent - #78

Open
scott-lowe-vapi wants to merge 1 commit into
fix/validate-referencesfrom
feat/user-agent-everywhere
Open

scott-lowe-vapi wants to merge 1 commit into
fix/validate-referencesfrom
feat/user-agent-everywhere

Conversation

@scott-lowe-vapi

@scott-lowe-vapi scott-lowe-vapi commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Value

V.A.L.U.E. tier: small — a behavior change: every API request now carries a new header. No blast-radius path.

  • Problem: we can't measure how gitops is used beyond simulations. Only npm run sim and npm run check identify themselves. Setup, pull, push, apply, promote, cleanup, rollback, call and audit send Node's default User-Agent, node, which was 126k of 728k api.vapi.ai requests in one hour this morning. So "how many deploys come from gitops, from CI or from laptops" has no answer.
  • Who it affects: the Vapi team measuring adoption after the public launch. Customers see no change in behavior.
  • What changes:
    • src/user-agent.ts builds vapi-gitops-<command>/<version>, plus (ci) when CI or GITHUB_ACTIONS is set:
      • <command> is the npm script that started the process. So npm run apply labels the pull and push it runs as apply, and a promotion's applies are labelled promote.
      • When a script is run directly, as the PR check workflow does, it falls back to the entry script's name.
      • sim and check keep their fixed labels, which existing simulation analytics already counts by.
    • Every fetch to the Vapi API sends it: api.ts (push, pull, apply, promote), cleanup, setup, the interactive pickers, rollback, call, and push's direct fetch.
    • Fixes two tests that called production. cleanup-safety and new-file-gate sent about 12 requests per npm test to api.vapi.ai, with a fake key, getting 401s. That breaks the repo's own rule that tests never call the real API, and with this PR it would have counted every fork's CI run as gitops usage. They now point at a dead local address.
    • Docs: how-it-works.md says exactly what the API sees (and that there is no other telemetry). AGENTS.md says every request must send the header.

Evidence of value

Live, through the Cloudflare request logs in Axiom (cloudflare-logpush): a read-only npm run setup -- ua-check --resources none against the test org, run from a scratch copy:

User-Agent Requests Status
vapi-gitops-setup/1.0.0 13 GET 200
vapi-gitops-test/1.0.0 24 GET 401 — the two leaky tests, from the two npm test runs before the fix

Before this PR, both rows would have been indistinguishable node traffic.

Tests:

  • tests/user-agent-coverage.test.ts reads every fetch( in src/ and requires a User-Agent. On the parent branch it lists 10 call sites without one; here it lists none. It also runs api.ts against a local server with npm_lifecycle_event=apply.
  • tests/user-agent.test.ts pins the format: fixed sim and check labels, npm script vs. entry script vs. cli, label cleaning, and the CI marker for GITHUB_ACTIONS=true, CI=true and CI=1 (but not false, 0 or empty).
  • No more production calls: running the full suite with a fetch trap that records any request to vapi.ai caught 12 requests before the test fix and none after.

Testing plan

  • npm test (527 tests) and npx tsc --noEmit pass.
  • Not covered:
    • Distinct-org counts for non-simulation commands. The request logs carry the User-Agent but not the org. Only simulation runs get an org ID, through PostHog. Joining the two needs an API-side change, outside this repo.
    • The version is still 1.0.0; bumping it is deliberately left out of this PR.
    • The call command's WebSocket audio connection isn't a Vapi REST request and doesn't carry the header.

Refs TEST-141

🤖 Generated with Claude Code

Only `npm run sim` and `npm run check` identified themselves. Setup,
pull, push, apply, promote, cleanup, rollback, call and audit sent Node's
default `node` User-Agent, about a sixth of all api.vapi.ai traffic, so
gitops usage beyond simulations couldn't be counted.

- src/user-agent.ts: `vapi-gitops-<command>/<version>`, plus ` (ci)`
  when CI or GITHUB_ACTIONS is set. The command is the npm script that
  started the process (so `npm run apply` labels the pull and push it
  runs as apply, and a promotion's applies as promote), else the entry
  script's name. sim and check keep their fixed labels, which analytics
  already counts simulation runs by.
- Every fetch to the Vapi API sends it. tests/user-agent-coverage.test.ts
  fails on a fetch without it and checks api.ts against a local server.
- cleanup-safety and new-file-gate tests sent about a dozen requests to
  the real api.vapi.ai per `npm test` (fake key, 401s), which the new
  User-Agent made visible in the request logs, and which would have
  counted every fork's CI run as usage. They now point at a dead local
  address.
- how-it-works.md says what the API sees; AGENTS.md says every request
  sends the header.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant