From fd7ded599e0d3bb1873f4097b945033fa813d391 Mon Sep 17 00:00:00 2001 From: Jordan Ritter Date: Tue, 20 Oct 2026 14:30:15 -0700 Subject: [PATCH 1/5] docs: cover unreleased model catalogs, cache usage, and client disconnects - README features: required-header policy, native model catalogs and token counts - fixtures usage override: Anthropic cache fields, linked to Claude Messages Cache Usage - Claude Messages and Error Injection: a client disconnect journals "client aborted" - write-fixtures skill: embeddingRequest match field, count, catalog and embed routes, /v1/models catalog selection, Anthropic cache usage --- README.md | 2 ++ docs/claude-messages/index.html | 9 ++++++++- docs/error-injection/index.html | 4 ++++ docs/fixtures/index.html | 7 +++++-- skills/write-fixtures/SKILL.md | 10 +++++++++- 5 files changed, 28 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index deabec5f..4d77f9e6 100644 --- a/README.md +++ b/README.md @@ -88,6 +88,8 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or - **HTTP route catalog** — `GET /__aimock/openapi.json` returns OpenAPI 3.1; `GET /__aimock/routes` returns `{ count, routes }`. For programmatic inspection, import `buildOpenApiDocument`, `CATALOG_ROUTES` and `ROUTE_DEFINITIONS` (with `CatalogRoute`/`RouteDefinition` types) from `@copilotkit/aimock`. The catalog describes HTTP operations; streaming responses include media and framing metadata, not a complete-stream JSON body. - **Request IDs** — every response carries `X-Request-Id`, echoing the caller's header or minting one; the journal stores it on each entry and `GET /__aimock/journal?requestId=` returns that request. A minted id is never forwarded upstream in record/proxy mode - **Per-Request Strict Mode** — `X-AIMock-Strict` header overrides the server-level `--strict` flag per request (`true`/`1` = strict, `false`/`0` = lenient) +- **[Required Header Policy](https://aimock.copilotkit.dev/aimock-cli#required-header-policy)** — Opt-in `requiredHeaders: "enforce"` (programmatic `policy`, config `llm.policy`, repeatable `--policy` on both CLIs, or the `X-AIMock-Policy` request header) rejects Anthropic HTTP requests (`/v1/messages`, `/v1/messages/count_tokens`, Anthropic-flavored `/v1/models`) that omit `anthropic-version` with a native 400. Off by default, and independent of strict mode +- **[Native model catalogs and token counts](https://aimock.copilotkit.dev/claude-messages#model-catalog)** — Anthropic `GET /v1/models` (selected by `anthropic-version`, or `x-api-key` without `Authorization`) and Gemini `GET /v1beta/models` list and detail routes with provider paging, built from fixture model IDs with synthetic metadata; `POST /v1/messages/count_tokens` and Gemini `:countTokens` return deterministic `ceil(chars / 4)` estimates. These routes never proxy or record - **Context-Based Fixture Routing** — `X-AIMock-Context` header scopes fixtures per integration; fixtures with `match.context` only match requests carrying that context, fixtures without it remain shared - **[Drift Detection](https://aimock.copilotkit.dev/drift-detection)** — Daily CI validation against real APIs - **[Streaming Physics](https://aimock.copilotkit.dev/streaming-physics)** — Configurable `ttft`, `tps`, and `jitter` diff --git a/docs/claude-messages/index.html b/docs/claude-messages/index.html index 7994ad04..fc03227b 100644 --- a/docs/claude-messages/index.html +++ b/docs/claude-messages/index.html @@ -340,8 +340,15 @@

SSE Event Sequence

  • message_delta — stop_reason, usage
  • message_stop
    • +

    + If the client disconnects mid-stream, aimock stops sending events and skips any remaining + per-chunk latency. The journal entry records response.interrupted: true and + response.interruptReason: "client aborted". Fixture interruptions + (truncateAfterChunks, disconnectAfterMs) keep their own reasons. + See Error Injection. +

    -

    Cache Usage

    +

    Cache Usage

    Set response.usage.cache_read_input_tokens and cache_creation_input_tokens in a fixture to emit cache counts on Messages and diff --git a/docs/error-injection/index.html b/docs/error-injection/index.html index cee19867..340b299a 100644 --- a/docs/error-injection/index.html +++ b/docs/error-injection/index.html @@ -187,6 +187,10 @@

    Interruption Behavior

    Interrupted requests are recorded in the journal with response.interrupted: true and response.interruptReason +
  • + A client that disconnects from a Claude Messages stream is recorded the same way, with + response.interruptReason: "client aborted" +
    • diff --git a/docs/fixtures/index.html b/docs/fixtures/index.html index 424fca1b..0868a101 100644 --- a/docs/fixtures/index.html +++ b/docs/fixtures/index.html @@ -935,8 +935,11 @@

    Response Override Fields

    Gemini field names (promptTokenCount, candidatesTokenCount, totalTokenCount). OpenAI Chat Completions includes usage in the response body; the Responses API uses a separate - response.usage object. When omitted, token counts are auto-computed - from content length + response.usage object. Claude Messages and Bedrock Claude invoke also + accept cache_read_input_tokens, + cache_creation_input_tokens and a cache_creation object + (see Cache Usage). When omitted, token + counts are auto-computed from content length diff --git a/skills/write-fixtures/SKILL.md b/skills/write-fixtures/SKILL.md index 57177849..3a34be4d 100644 --- a/skills/write-fixtures/SKILL.md +++ b/skills/write-fixtures/SKILL.md @@ -28,6 +28,7 @@ aimock is a zero-dependency mock infrastructure for AI apps. Fixture-driven. Mul | `systemMessage` | `RegExp` | Pattern test on the concatenated system-message text | | `inputText` | `string` | Substring of embedding input text (concatenated if multiple inputs) | | `inputText` | `RegExp` | Pattern test on embedding input text | +| `embeddingRequest` | `{ operation, input, dimensions? }` | Exact operation (`/api/embed`, `/api/embeddings` or `/v1/embeddings`), input shape and requested dimensions. Recordings set it alongside `inputText` and `model` so batches and widths stay distinct; omit it for broad embedding fixtures | | `toolName` | `string` | Exact match on any tool in request's `tools[]` array (by `function.name`). On the OpenAI Responses API, with `responsesTools: "extended"`, this also includes tools inside a `namespace` tool, `custom` tools (by `name`, in `req.customTools`), and tools in an `additional_tools` or `tool_search_output` input item; by default only top-level function tools, as in 1.44.0. Other Responses tool types (`web_search`, `mcp`, ...) are not visible | | `toolNamespace` | `string` | Exact OpenAI Responses tool namespace; needs `responsesTools: "extended"` (ignored otherwise). Alone: any offered tool sits in it. With `toolName`: one offered tool must carry both (Codex routes by this exact pair). No default namespace is assumed. Chat Completions passes request tools through, so a client-sent top-level `namespace` there also matches. With `"extended"` (`aimock validate --responses-tools extended`), `aimock validate` and `--validate-on-load` warn about a file fixture with `toolNamespace` and an `endpoint` other than `"chat"`. A Responses `namespace` tool with an empty or missing `name` is dropped. Codex names an MCP server's namespace `mcp__` (no trailing `__`). aimock-pytest needs aimock 1.45.0+, its default pin from 0.7.0 | | `toolCallId` | `string` | Exact match on `tool_call_id` of last `role: "tool"` message | @@ -533,6 +534,7 @@ All providers share the same fixture pool — write fixtures once, they work for | `POST /v1/chat/completions` | OpenAI | HTTP | | `POST /v1/responses` | OpenAI | HTTP + WS | | `POST /v1/messages` | Anthropic | HTTP | +| `POST /v1/messages/count_tokens` | Anthropic | HTTP | | `POST /v1/embeddings` | OpenAI | HTTP | | `POST /v1beta/models/{model}:{method}` | Google Gemini | HTTP | | `POST /model/{modelId}/invoke` | AWS Bedrock | HTTP | @@ -548,11 +550,13 @@ All providers share the same fixture pool — write fixtures once, they work for | `POST /api/chat` | Ollama | HTTP | | `POST /api/generate` | Ollama | HTTP | | `GET /api/tags` | Ollama | HTTP | +| `POST /api/embed` · `/api/embeddings` | Ollama | HTTP | | `POST /v2/chat` | Cohere | HTTP | | `POST /api/v1/chat/completions` | OpenRouter | HTTP | | `GET /api/v1/models` · `/api/v1/key` · `/api/v1/credits` | OpenRouter | HTTP | | `GET /metrics` | — | HTTP | -| `GET /v1/models` | OpenAI-compat | HTTP | +| `GET /v1/models` · `/v1/models/{id}` | OpenAI-compat | HTTP | +| `GET /v1beta/models` · `/v1beta/models/{model}` | Google Gemini | HTTP | | `WS /v1/responses` | OpenAI | WebSocket | | `WS /v1/realtime` | OpenAI | WebSocket | | `WS /ws/google.ai...BidiGenerateContent` | Gemini Live | WebSocket | @@ -563,6 +567,8 @@ All providers share the same fixture pool — write fixtures once, they work for | `POST /v1/videos` | OpenAI | HTTP | | `GET /v1/videos/{id}` | OpenAI | HTTP | +`GET /v1/models` returns the Anthropic catalog (native paging, `after_id` / `before_id`) when the request has `anthropic-version`, or `x-api-key` without `Authorization`; other callers get the OpenAI format. Catalogs list the fixture model IDs. Catalog and count routes (`count_tokens`, Gemini `:countTokens`) never proxy or record, and counts ignore fixture `usage`. + ## Response Template Overrides Fixture responses can include optional override fields to control auto-generated envelope values. These are merged into the provider-specific response format (OpenAI, Claude, Gemini, Responses API). @@ -582,6 +588,8 @@ Fixture responses can include optional override fields to control auto-generated | `usage.cost` | number | (omitted) | OpenRouter only: per-request `usage.cost` (scriptable — powers budget-guard tests). When set, `usage.cost_details` is emitted too. Never fabricated when omitted | | `usage.is_byok` | bool | (omitted) | OpenRouter only: emit `usage.is_byok`. Also `usage.prompt_tokens_details`, `usage.completion_tokens_details` — emitted only when set | +Claude Messages and Bedrock Claude invoke also emit `usage.cache_read_input_tokens`, `usage.cache_creation_input_tokens` and a `usage.cache_creation` object (`ephemeral_5m_input_tokens`, `ephemeral_1h_input_tokens`) when a fixture supplies them. + ### Example ```typescript From 8ce014d72a7f25b5bbe2b7abc0931c9930ac950f Mon Sep 17 00:00:00 2001 From: Jordan Ritter Date: Tue, 20 Oct 2026 14:30:15 -0700 Subject: [PATCH 2/5] docs: document streamingProfile timing on non-streaming chat completions - Streaming Physics: a new Non-Streaming Responses section (OpenAI Chat Completions only, summed per-chunk delays) and a latency note that bare latency and --latency never delay stream:false replies - write-fixtures skill: gotchas 3 and 4 carry the same rule --- docs/streaming-physics/index.html | 19 +++++++++++++++++++ skills/write-fixtures/SKILL.md | 4 ++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/docs/streaming-physics/index.html b/docs/streaming-physics/index.html index 6ade3d62..012cf430 100644 --- a/docs/streaming-physics/index.html +++ b/docs/streaming-physics/index.html @@ -158,6 +158,20 @@

    JSON Fixture File

    }
    +

    Non-Streaming Responses

    +

    + A fixture with streamingProfile or recordedTimings also delays + stream: false OpenAI Chat Completions responses, for both text and tool + calls. aimock waits for the sum of the per-chunk delays that a streamed reply at the same + chunkSize would get (ttft, then a tps gap for each + further content chunk), then writes the JSON body. A streamed reply also pays for its role + and finish chunks, so the two totals are close but not identical. +

    +

    + This applies only to OpenAI Chat Completions. Non-streaming responses from other providers + are not delayed. +

    +

    Interaction with latency

    Realistic Profiles

    diff --git a/skills/write-fixtures/SKILL.md b/skills/write-fixtures/SKILL.md index 3a34be4d..b06c5687 100644 --- a/skills/write-fixtures/SKILL.md +++ b/skills/write-fixtures/SKILL.md @@ -666,9 +666,9 @@ Custom tool calls (`customToolCalls` / `customToolCall` blocks in `responsesBloc 2. **`arguments` accepts both objects and strings** — `"arguments": {"key":"value"}` (preferred, auto-stringified) or `"arguments": "{\"key\":\"value\"}"` (legacy). The same applies to `content` fields that contain JSON. The fixture loader detects `typeof === "object"` and calls `JSON.stringify()` automatically. -3. **Latency is per-chunk, not total** — `latency: 100` means 100ms between each SSE chunk, not 100ms total response time. Similarly, `truncateAfterChunks` and `disconnectAfterMs` are for simulating stream interruptions (added in v1.3.0). +3. **Latency is per-chunk, not total** — `latency: 100` means 100ms between each SSE chunk, not 100ms total response time. It never delays a non-streaming (`stream: false`) response. Similarly, `truncateAfterChunks` and `disconnectAfterMs` are for simulating stream interruptions (added in v1.3.0). -4. **`streamingProfile` takes precedence over `latency`** — when both are set on a fixture, `streamingProfile` controls timing. Use one or the other. +4. **`streamingProfile` takes precedence over `latency`** — when both are set on a fixture, `streamingProfile` controls timing. Use one or the other. A `streamingProfile` (or `recordedTimings`) also delays `stream: false` OpenAI Chat Completions replies, text and tool calls, by the sum of the per-chunk delays a streamed reply at the same `chunkSize` would get. Other providers' non-streaming replies are not delayed. 5. **Tool result messages don't change the user message** — after a tool call, the client sends the same conversation + tool result. Matching on `userMessage` will hit the SAME fixture again → infinite loop. Always use `predicate` checking `role === "tool"` for tool results. Note: a whole-conversation `role === "tool"` check (e.g. `req.messages.some((m) => m.role === "tool")`) diverges from `hasToolResult`'s current-turn scoping in multi-turn flows — the built-in `hasToolResult` matcher only looks after the last user message, so a later turn whose history carries an earlier tool result still reads `false`. From c6b56eba07b54529b6ec458f56f56acb7e9e03e3 Mon Sep 17 00:00:00 2001 From: Jordan Ritter Date: Tue, 20 Oct 2026 14:30:16 -0700 Subject: [PATCH 3/5] docs: describe the token-count estimate accurately Counts are ceil(message chars / 4), plus ceil(tool JSON / 4) when tools are present, each at least 1 (src/count-tokens.ts and src/response/usage/estimate.ts). The docs said a bare ceil(chars / 4). --- README.md | 2 +- skills/write-fixtures/SKILL.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 4d77f9e6..dd09b54b 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,7 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or - **Request IDs** — every response carries `X-Request-Id`, echoing the caller's header or minting one; the journal stores it on each entry and `GET /__aimock/journal?requestId=` returns that request. A minted id is never forwarded upstream in record/proxy mode - **Per-Request Strict Mode** — `X-AIMock-Strict` header overrides the server-level `--strict` flag per request (`true`/`1` = strict, `false`/`0` = lenient) - **[Required Header Policy](https://aimock.copilotkit.dev/aimock-cli#required-header-policy)** — Opt-in `requiredHeaders: "enforce"` (programmatic `policy`, config `llm.policy`, repeatable `--policy` on both CLIs, or the `X-AIMock-Policy` request header) rejects Anthropic HTTP requests (`/v1/messages`, `/v1/messages/count_tokens`, Anthropic-flavored `/v1/models`) that omit `anthropic-version` with a native 400. Off by default, and independent of strict mode -- **[Native model catalogs and token counts](https://aimock.copilotkit.dev/claude-messages#model-catalog)** — Anthropic `GET /v1/models` (selected by `anthropic-version`, or `x-api-key` without `Authorization`) and Gemini `GET /v1beta/models` list and detail routes with provider paging, built from fixture model IDs with synthetic metadata; `POST /v1/messages/count_tokens` and Gemini `:countTokens` return deterministic `ceil(chars / 4)` estimates. These routes never proxy or record +- **[Native model catalogs and token counts](https://aimock.copilotkit.dev/claude-messages#model-catalog)** — Anthropic `GET /v1/models` (selected by `anthropic-version`, or `x-api-key` without `Authorization`) and Gemini `GET /v1beta/models` list and detail routes with provider paging, built from fixture model IDs with synthetic metadata; `POST /v1/messages/count_tokens` and Gemini `:countTokens` return a deterministic estimate of about 1 token per 4 characters of messages (plus tool definitions), minimum 1. These routes never proxy or record - **Context-Based Fixture Routing** — `X-AIMock-Context` header scopes fixtures per integration; fixtures with `match.context` only match requests carrying that context, fixtures without it remain shared - **[Drift Detection](https://aimock.copilotkit.dev/drift-detection)** — Daily CI validation against real APIs - **[Streaming Physics](https://aimock.copilotkit.dev/streaming-physics)** — Configurable `ttft`, `tps`, and `jitter` diff --git a/skills/write-fixtures/SKILL.md b/skills/write-fixtures/SKILL.md index b06c5687..1ad16331 100644 --- a/skills/write-fixtures/SKILL.md +++ b/skills/write-fixtures/SKILL.md @@ -567,7 +567,7 @@ All providers share the same fixture pool — write fixtures once, they work for | `POST /v1/videos` | OpenAI | HTTP | | `GET /v1/videos/{id}` | OpenAI | HTTP | -`GET /v1/models` returns the Anthropic catalog (native paging, `after_id` / `before_id`) when the request has `anthropic-version`, or `x-api-key` without `Authorization`; other callers get the OpenAI format. Catalogs list the fixture model IDs. Catalog and count routes (`count_tokens`, Gemini `:countTokens`) never proxy or record, and counts ignore fixture `usage`. +`GET /v1/models` returns the Anthropic catalog (native paging, `after_id` / `before_id`) when the request has `anthropic-version`, or `x-api-key` without `Authorization`; other callers get the OpenAI format. Catalogs list the fixture model IDs. Catalog and count routes (`count_tokens`, Gemini `:countTokens`) never proxy or record. Counts are a deterministic estimate of about 1 token per 4 characters of messages (plus tool definitions), minimum 1, and ignore fixture `usage`. ## Response Template Overrides From bf2f2a59bea58f5d144c033d44ebb6fcb598ca13 Mon Sep 17 00:00:00 2001 From: Jordan Ritter Date: Tue, 20 Oct 2026 14:30:16 -0700 Subject: [PATCH 4/5] docs: match the token-count estimate wording on Claude and Gemini pages --- docs/claude-messages/index.html | 5 +++-- docs/gemini/index.html | 9 +++++---- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/claude-messages/index.html b/docs/claude-messages/index.html index fc03227b..0a452b0d 100644 --- a/docs/claude-messages/index.html +++ b/docs/claude-messages/index.html @@ -153,8 +153,9 @@

    Approximate Token Counts

    POST /v1/messages/count_tokens accepts a Messages-style body and returns {"input_tokens": N}. It converts messages, system text, and tools through the - existing request translator, then estimates tokens with ceil(chars / 4). This - deterministic estimate is not a tokenizer or an upstream count. + existing request translator, then estimates about 1 token per 4 characters of messages + (plus tool definitions), minimum 1. This deterministic estimate is not a tokenizer or an + upstream count.

    Count a prompt shell
    diff --git a/docs/gemini/index.html b/docs/gemini/index.html index c77cbb63..e49283e5 100644 --- a/docs/gemini/index.html +++ b/docs/gemini/index.html @@ -149,10 +149,11 @@

    Approximate Token Counts

    including system text and tools.

    - aimock converts the request through its existing translator, then estimates tokens with - ceil(chars / 4). The estimate includes converted messages, system text, and - serialized converted tools. This deterministic estimate is not a tokenizer or an upstream - count. Optional cache counts and token-detail arrays are omitted. + aimock converts the request through its existing translator, then estimates about 1 token + per 4 characters of messages (plus tool definitions), minimum 1. The estimate includes + converted messages, system text, and serialized converted tools. This deterministic + estimate is not a tokenizer or an upstream count. Optional cache counts and token-detail + arrays are omitted.

    Count a prompt shell
    From c70ca9dd4a2cfa7893b8d28253cf2ac5b39757d8 Mon Sep 17 00:00:00 2001 From: Jordan Ritter Date: Tue, 20 Oct 2026 14:30:16 -0700 Subject: [PATCH 5/5] docs: list combined and blocks replies in non-streaming timing stream:false OpenAI Chat Completions delays now also cover combined content and tool-call replies and blocks replies. --- docs/streaming-physics/index.html | 11 ++++++----- skills/write-fixtures/SKILL.md | 2 +- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/streaming-physics/index.html b/docs/streaming-physics/index.html index 012cf430..be5fb199 100644 --- a/docs/streaming-physics/index.html +++ b/docs/streaming-physics/index.html @@ -161,11 +161,12 @@

    JSON Fixture File

    Non-Streaming Responses

    A fixture with streamingProfile or recordedTimings also delays - stream: false OpenAI Chat Completions responses, for both text and tool - calls. aimock waits for the sum of the per-chunk delays that a streamed reply at the same - chunkSize would get (ttft, then a tps gap for each - further content chunk), then writes the JSON body. A streamed reply also pays for its role - and finish chunks, so the two totals are close but not identical. + stream: false OpenAI Chat Completions responses: text, tool calls, combined + content and tool calls, and blocks replies. aimock waits for the sum of the + per-chunk delays that a streamed reply at the same chunkSize would get + (ttft, then a tps gap for each further content chunk), then + writes the JSON body. A streamed reply also pays for its role and finish chunks, so the + two totals are close but not identical.

    This applies only to OpenAI Chat Completions. Non-streaming responses from other providers diff --git a/skills/write-fixtures/SKILL.md b/skills/write-fixtures/SKILL.md index 1ad16331..b68ee4f2 100644 --- a/skills/write-fixtures/SKILL.md +++ b/skills/write-fixtures/SKILL.md @@ -668,7 +668,7 @@ Custom tool calls (`customToolCalls` / `customToolCall` blocks in `responsesBloc 3. **Latency is per-chunk, not total** — `latency: 100` means 100ms between each SSE chunk, not 100ms total response time. It never delays a non-streaming (`stream: false`) response. Similarly, `truncateAfterChunks` and `disconnectAfterMs` are for simulating stream interruptions (added in v1.3.0). -4. **`streamingProfile` takes precedence over `latency`** — when both are set on a fixture, `streamingProfile` controls timing. Use one or the other. A `streamingProfile` (or `recordedTimings`) also delays `stream: false` OpenAI Chat Completions replies, text and tool calls, by the sum of the per-chunk delays a streamed reply at the same `chunkSize` would get. Other providers' non-streaming replies are not delayed. +4. **`streamingProfile` takes precedence over `latency`** — when both are set on a fixture, `streamingProfile` controls timing. Use one or the other. A `streamingProfile` (or `recordedTimings`) also delays `stream: false` OpenAI Chat Completions replies (text, tool calls, combined content and tool calls, and `blocks`) by the sum of the per-chunk delays a streamed reply at the same `chunkSize` would get. Other providers' non-streaming replies are not delayed. 5. **Tool result messages don't change the user message** — after a tool call, the client sends the same conversation + tool result. Matching on `userMessage` will hit the SAME fixture again → infinite loop. Always use `predicate` checking `role === "tool"` for tool results. Note: a whole-conversation `role === "tool"` check (e.g. `req.messages.some((m) => m.role === "tool")`) diverges from `hasToolResult`'s current-turn scoping in multi-turn flows — the built-in `hasToolResult` matcher only looks after the last user message, so a later turn whose history carries an earlier tool result still reads `false`.