diff --git a/README.md b/README.md index deabec5f..dd09b54b 100644 --- a/README.md +++ b/README.md @@ -88,6 +88,8 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or - **HTTP route catalog** — `GET /__aimock/openapi.json` returns OpenAPI 3.1; `GET /__aimock/routes` returns `{ count, routes }`. For programmatic inspection, import `buildOpenApiDocument`, `CATALOG_ROUTES` and `ROUTE_DEFINITIONS` (with `CatalogRoute`/`RouteDefinition` types) from `@copilotkit/aimock`. The catalog describes HTTP operations; streaming responses include media and framing metadata, not a complete-stream JSON body. - **Request IDs** — every response carries `X-Request-Id`, echoing the caller's header or minting one; the journal stores it on each entry and `GET /__aimock/journal?requestId=` returns that request. A minted id is never forwarded upstream in record/proxy mode - **Per-Request Strict Mode** — `X-AIMock-Strict` header overrides the server-level `--strict` flag per request (`true`/`1` = strict, `false`/`0` = lenient) +- **[Required Header Policy](https://aimock.copilotkit.dev/aimock-cli#required-header-policy)** — Opt-in `requiredHeaders: "enforce"` (programmatic `policy`, config `llm.policy`, repeatable `--policy` on both CLIs, or the `X-AIMock-Policy` request header) rejects Anthropic HTTP requests (`/v1/messages`, `/v1/messages/count_tokens`, Anthropic-flavored `/v1/models`) that omit `anthropic-version` with a native 400. Off by default, and independent of strict mode +- **[Native model catalogs and token counts](https://aimock.copilotkit.dev/claude-messages#model-catalog)** — Anthropic `GET /v1/models` (selected by `anthropic-version`, or `x-api-key` without `Authorization`) and Gemini `GET /v1beta/models` list and detail routes with provider paging, built from fixture model IDs with synthetic metadata; `POST /v1/messages/count_tokens` and Gemini `:countTokens` return a deterministic estimate of about 1 token per 4 characters of messages (plus tool definitions), minimum 1. These routes never proxy or record - **Context-Based Fixture Routing** — `X-AIMock-Context` header scopes fixtures per integration; fixtures with `match.context` only match requests carrying that context, fixtures without it remain shared - **[Drift Detection](https://aimock.copilotkit.dev/drift-detection)** — Daily CI validation against real APIs - **[Streaming Physics](https://aimock.copilotkit.dev/streaming-physics)** — Configurable `ttft`, `tps`, and `jitter` diff --git a/docs/claude-messages/index.html b/docs/claude-messages/index.html index 7994ad04..0a452b0d 100644 --- a/docs/claude-messages/index.html +++ b/docs/claude-messages/index.html @@ -153,8 +153,9 @@

Approximate Token Counts

POST /v1/messages/count_tokens accepts a Messages-style body and returns {"input_tokens": N}. It converts messages, system text, and tools through the - existing request translator, then estimates tokens with ceil(chars / 4). This - deterministic estimate is not a tokenizer or an upstream count. + existing request translator, then estimates about 1 token per 4 characters of messages + (plus tool definitions), minimum 1. This deterministic estimate is not a tokenizer or an + upstream count.

Count a prompt shell
@@ -340,8 +341,15 @@

SSE Event Sequence

  • message_delta — stop_reason, usage
  • message_stop
    • +

    + If the client disconnects mid-stream, aimock stops sending events and skips any remaining + per-chunk latency. The journal entry records response.interrupted: true and + response.interruptReason: "client aborted". Fixture interruptions + (truncateAfterChunks, disconnectAfterMs) keep their own reasons. + See Error Injection. +

    -

    Cache Usage

    +

    Cache Usage

    Set response.usage.cache_read_input_tokens and cache_creation_input_tokens in a fixture to emit cache counts on Messages and diff --git a/docs/error-injection/index.html b/docs/error-injection/index.html index cee19867..340b299a 100644 --- a/docs/error-injection/index.html +++ b/docs/error-injection/index.html @@ -187,6 +187,10 @@

    Interruption Behavior

    Interrupted requests are recorded in the journal with response.interrupted: true and response.interruptReason +
  • + A client that disconnects from a Claude Messages stream is recorded the same way, with + response.interruptReason: "client aborted" +
    • diff --git a/docs/fixtures/index.html b/docs/fixtures/index.html index 424fca1b..0868a101 100644 --- a/docs/fixtures/index.html +++ b/docs/fixtures/index.html @@ -935,8 +935,11 @@

    Response Override Fields

    Gemini field names (promptTokenCount, candidatesTokenCount, totalTokenCount). OpenAI Chat Completions includes usage in the response body; the Responses API uses a separate - response.usage object. When omitted, token counts are auto-computed - from content length + response.usage object. Claude Messages and Bedrock Claude invoke also + accept cache_read_input_tokens, + cache_creation_input_tokens and a cache_creation object + (see Cache Usage). When omitted, token + counts are auto-computed from content length diff --git a/docs/gemini/index.html b/docs/gemini/index.html index c77cbb63..e49283e5 100644 --- a/docs/gemini/index.html +++ b/docs/gemini/index.html @@ -149,10 +149,11 @@

    Approximate Token Counts

    including system text and tools.

    - aimock converts the request through its existing translator, then estimates tokens with - ceil(chars / 4). The estimate includes converted messages, system text, and - serialized converted tools. This deterministic estimate is not a tokenizer or an upstream - count. Optional cache counts and token-detail arrays are omitted. + aimock converts the request through its existing translator, then estimates about 1 token + per 4 characters of messages (plus tool definitions), minimum 1. The estimate includes + converted messages, system text, and serialized converted tools. This deterministic + estimate is not a tokenizer or an upstream count. Optional cache counts and token-detail + arrays are omitted.

    Count a prompt shell
    diff --git a/docs/streaming-physics/index.html b/docs/streaming-physics/index.html index 6ade3d62..be5fb199 100644 --- a/docs/streaming-physics/index.html +++ b/docs/streaming-physics/index.html @@ -158,6 +158,21 @@

    JSON Fixture File

    }
    +

    Non-Streaming Responses

    +

    + A fixture with streamingProfile or recordedTimings also delays + stream: false OpenAI Chat Completions responses: text, tool calls, combined + content and tool calls, and blocks replies. aimock waits for the sum of the + per-chunk delays that a streamed reply at the same chunkSize would get + (ttft, then a tps gap for each further content chunk), then + writes the JSON body. A streamed reply also pays for its role and finish chunks, so the + two totals are close but not identical. +

    +

    + This applies only to OpenAI Chat Completions. Non-streaming responses from other providers + are not delayed. +

    +

    Interaction with latency

    • @@ -172,6 +187,11 @@

      Interaction with latency

      If streamingProfile is set but has neither ttft nor tps, it falls back to latency.
      • +
    • + latency and --latency are gaps between SSE chunks only. They + never delay a non-streaming response, including the fallback case above. Use + streamingProfile to delay stream: false replies. +

    Realistic Profiles

    diff --git a/skills/write-fixtures/SKILL.md b/skills/write-fixtures/SKILL.md index 57177849..b68ee4f2 100644 --- a/skills/write-fixtures/SKILL.md +++ b/skills/write-fixtures/SKILL.md @@ -28,6 +28,7 @@ aimock is a zero-dependency mock infrastructure for AI apps. Fixture-driven. Mul | `systemMessage` | `RegExp` | Pattern test on the concatenated system-message text | | `inputText` | `string` | Substring of embedding input text (concatenated if multiple inputs) | | `inputText` | `RegExp` | Pattern test on embedding input text | +| `embeddingRequest` | `{ operation, input, dimensions? }` | Exact operation (`/api/embed`, `/api/embeddings` or `/v1/embeddings`), input shape and requested dimensions. Recordings set it alongside `inputText` and `model` so batches and widths stay distinct; omit it for broad embedding fixtures | | `toolName` | `string` | Exact match on any tool in request's `tools[]` array (by `function.name`). On the OpenAI Responses API, with `responsesTools: "extended"`, this also includes tools inside a `namespace` tool, `custom` tools (by `name`, in `req.customTools`), and tools in an `additional_tools` or `tool_search_output` input item; by default only top-level function tools, as in 1.44.0. Other Responses tool types (`web_search`, `mcp`, ...) are not visible | | `toolNamespace` | `string` | Exact OpenAI Responses tool namespace; needs `responsesTools: "extended"` (ignored otherwise). Alone: any offered tool sits in it. With `toolName`: one offered tool must carry both (Codex routes by this exact pair). No default namespace is assumed. Chat Completions passes request tools through, so a client-sent top-level `namespace` there also matches. With `"extended"` (`aimock validate --responses-tools extended`), `aimock validate` and `--validate-on-load` warn about a file fixture with `toolNamespace` and an `endpoint` other than `"chat"`. A Responses `namespace` tool with an empty or missing `name` is dropped. Codex names an MCP server's namespace `mcp__` (no trailing `__`). aimock-pytest needs aimock 1.45.0+, its default pin from 0.7.0 | | `toolCallId` | `string` | Exact match on `tool_call_id` of last `role: "tool"` message | @@ -533,6 +534,7 @@ All providers share the same fixture pool — write fixtures once, they work for | `POST /v1/chat/completions` | OpenAI | HTTP | | `POST /v1/responses` | OpenAI | HTTP + WS | | `POST /v1/messages` | Anthropic | HTTP | +| `POST /v1/messages/count_tokens` | Anthropic | HTTP | | `POST /v1/embeddings` | OpenAI | HTTP | | `POST /v1beta/models/{model}:{method}` | Google Gemini | HTTP | | `POST /model/{modelId}/invoke` | AWS Bedrock | HTTP | @@ -548,11 +550,13 @@ All providers share the same fixture pool — write fixtures once, they work for | `POST /api/chat` | Ollama | HTTP | | `POST /api/generate` | Ollama | HTTP | | `GET /api/tags` | Ollama | HTTP | +| `POST /api/embed` · `/api/embeddings` | Ollama | HTTP | | `POST /v2/chat` | Cohere | HTTP | | `POST /api/v1/chat/completions` | OpenRouter | HTTP | | `GET /api/v1/models` · `/api/v1/key` · `/api/v1/credits` | OpenRouter | HTTP | | `GET /metrics` | — | HTTP | -| `GET /v1/models` | OpenAI-compat | HTTP | +| `GET /v1/models` · `/v1/models/{id}` | OpenAI-compat | HTTP | +| `GET /v1beta/models` · `/v1beta/models/{model}` | Google Gemini | HTTP | | `WS /v1/responses` | OpenAI | WebSocket | | `WS /v1/realtime` | OpenAI | WebSocket | | `WS /ws/google.ai...BidiGenerateContent` | Gemini Live | WebSocket | @@ -563,6 +567,8 @@ All providers share the same fixture pool — write fixtures once, they work for | `POST /v1/videos` | OpenAI | HTTP | | `GET /v1/videos/{id}` | OpenAI | HTTP | +`GET /v1/models` returns the Anthropic catalog (native paging, `after_id` / `before_id`) when the request has `anthropic-version`, or `x-api-key` without `Authorization`; other callers get the OpenAI format. Catalogs list the fixture model IDs. Catalog and count routes (`count_tokens`, Gemini `:countTokens`) never proxy or record. Counts are a deterministic estimate of about 1 token per 4 characters of messages (plus tool definitions), minimum 1, and ignore fixture `usage`. + ## Response Template Overrides Fixture responses can include optional override fields to control auto-generated envelope values. These are merged into the provider-specific response format (OpenAI, Claude, Gemini, Responses API). @@ -582,6 +588,8 @@ Fixture responses can include optional override fields to control auto-generated | `usage.cost` | number | (omitted) | OpenRouter only: per-request `usage.cost` (scriptable — powers budget-guard tests). When set, `usage.cost_details` is emitted too. Never fabricated when omitted | | `usage.is_byok` | bool | (omitted) | OpenRouter only: emit `usage.is_byok`. Also `usage.prompt_tokens_details`, `usage.completion_tokens_details` — emitted only when set | +Claude Messages and Bedrock Claude invoke also emit `usage.cache_read_input_tokens`, `usage.cache_creation_input_tokens` and a `usage.cache_creation` object (`ephemeral_5m_input_tokens`, `ephemeral_1h_input_tokens`) when a fixture supplies them. + ### Example ```typescript @@ -658,9 +666,9 @@ Custom tool calls (`customToolCalls` / `customToolCall` blocks in `responsesBloc 2. **`arguments` accepts both objects and strings** — `"arguments": {"key":"value"}` (preferred, auto-stringified) or `"arguments": "{\"key\":\"value\"}"` (legacy). The same applies to `content` fields that contain JSON. The fixture loader detects `typeof === "object"` and calls `JSON.stringify()` automatically. -3. **Latency is per-chunk, not total** — `latency: 100` means 100ms between each SSE chunk, not 100ms total response time. Similarly, `truncateAfterChunks` and `disconnectAfterMs` are for simulating stream interruptions (added in v1.3.0). +3. **Latency is per-chunk, not total** — `latency: 100` means 100ms between each SSE chunk, not 100ms total response time. It never delays a non-streaming (`stream: false`) response. Similarly, `truncateAfterChunks` and `disconnectAfterMs` are for simulating stream interruptions (added in v1.3.0). -4. **`streamingProfile` takes precedence over `latency`** — when both are set on a fixture, `streamingProfile` controls timing. Use one or the other. +4. **`streamingProfile` takes precedence over `latency`** — when both are set on a fixture, `streamingProfile` controls timing. Use one or the other. A `streamingProfile` (or `recordedTimings`) also delays `stream: false` OpenAI Chat Completions replies (text, tool calls, combined content and tool calls, and `blocks`) by the sum of the per-chunk delays a streamed reply at the same `chunkSize` would get. Other providers' non-streaming replies are not delayed. 5. **Tool result messages don't change the user message** — after a tool call, the client sends the same conversation + tool result. Matching on `userMessage` will hit the SAME fixture again → infinite loop. Always use `predicate` checking `role === "tool"` for tool results. Note: a whole-conversation `role === "tool"` check (e.g. `req.messages.some((m) => m.role === "tool")`) diverges from `hasToolResult`'s current-turn scoping in multi-turn flows — the built-in `hasToolResult` matcher only looks after the last user message, so a later turn whose history carries an earlier tool result still reads `false`.