diff --git a/docs/helix/02-design/adr/ADR-013-the-databricks-ai-gateway-is-a-first-class-backend-delivered-as-a-profile-and-never-the-only-backend.md b/docs/helix/02-design/adr/ADR-013-the-databricks-ai-gateway-is-a-first-class-backend-delivered-as-a-profile-and-never-the-only-backend.md index bb370c9..76dda2c 100644 --- a/docs/helix/02-design/adr/ADR-013-the-databricks-ai-gateway-is-a-first-class-backend-delivered-as-a-profile-and-never-the-only-backend.md +++ b/docs/helix/02-design/adr/ADR-013-the-databricks-ai-gateway-is-a-first-class-backend-delivered-as-a-profile-and-never-the-only-backend.md @@ -35,12 +35,13 @@ We will support the Databricks AI Gateway as a first-class backend, delivered as - **Attribution and correlation.** Required opaque session/turn/model-call/attempt/purpose tags use the documented `Databricks-Ai-Gateway-Request-Tags` JSON header. Optional bounded User-Agent attribution and real `traceparent` remain secondary signals. Keep header request id, completion body id and provider invocation id distinct; exact row/tag/header joins require profile-specific live qualification (CONTRACT-005). Reuse the inspected downstream capture/read pattern with async I/O and stronger multi-invocation/conflict handling. - **Identity.** A request's actor credential is used when present (ADR-011). Otherwise the application's own identity comes from the Databricks SDK's unified authentication (a service principal, or a developer's command-line profile), which this module alone imports and only for host discovery and authentication. A `require_user` mode refuses a call that has no user credential instead of falling back to the application's identity, as the first downstream platform's implementation does. - **Reference launcher.** `--databricks-profile NAME` derives the workspace host from the explicitly selected installed CLI profile; no URL is required. The CLI bridge owns bounded model-service discovery and OAuth token refresh, while the existing compatible adapter owns inference. Discovery initially covers `system.ai` services advertising `mlflow/v1/chat/completions`, with a picker before session creation and an explicit `--model` bypass. Catalog metadata is not invocation permission or context-capacity evidence. The bridge pins the discovered host and refuses later changes; it never persists credentials. This launcher qualifies existing CLI OAuth user profiles only; SDK application identity and full gateway attribution/history qualification remain separate profiles. -- **Declared wire capabilities.** The reference profile omits the unsupported `parallel_tool_calls` option in native inference and offline estimation, uses the gateway request-tag header with string-valued opaque tags, and permits validated partial usage on intermediate SSE chunks while requiring complete terminal input/output counts. These choices do not add an inference client or relax tool approval. Broader cache accounting and history/row joins require separate qualification. +- **Declared wire capabilities.** The reference profile omits the unsupported `parallel_tool_calls` option in native inference and offline estimation, uses the gateway request-tag header with string-valued opaque tags, and permits validated partial usage on intermediate SSE chunks while requiring complete terminal input/output counts. These choices do not add an inference client or relax tool approval. History/row joins require separate qualification. - **Gateway stream dialect.** Gateway backends stream frames outside the strict `openai-chat-text-tools-v1` chunk shape: a prompt-filter preamble (`object: ""`, empty `choices`, `prompt_filter_results`), content-filter results and filter errors on choices, null `logprobs` and `refusal`, `obfuscation`, `latency_checkpoint` and `routing` metadata, and reasoning-summary content parts before answer text. We will not widen the strict profile. A separate, explicitly selected stream profile, `databricks-ai-gateway-stream-v1`, accepts exactly those fields against closed allow-lists and forwards only normalized frames that the strict validator then re-checks; the SDK never sees a gateway field. The reference launcher selects it with its Databricks profile; `OpenAICompatModel` and `CompatibleModelSettings` default to the strict profile. - *Reasoning-summary parts are validated and dropped.* A list-valued delta `content` must hold only `{"type": "reasoning", "summary": [{"type": "summary_text", "text", "signature"?}]}` parts. They never become answer text, deltas, tool input or telemetry. Text parts in list form, other part types and other keys refuse. Preserving reasoning as typed non-text output needs a model-port change and its own decision. - *Filter metadata is shape-checked, not kept.* Category maps allow only `filtered`/`detected` booleans and a `severity` string; a filter error allows only a string `code` and `message`. None of it enters `ModelResponse`, receipts or audit. A prompt category reporting `filtered: true` refuses as `prompt_filtered`; an output category reporting `filtered: true` requires a `content_filter` terminal, which normalizes to `filtered`. Tether is not the content-safety layer; the gateway's terminal reason governs. - *Unknown fields still refuse* under both profiles. The raw stream bytes remain the attempt's wire observation, so dropped fields stay in the original exchange record under existing payload authorization. - - Nonstreaming responses still use the strict body validator; this dialect is qualified for streaming only. + - Nonstreaming bodies, including the stream fallback when a server ignores `stream: true`, still use the strict body validator; this dialect is qualified for streaming only. Structure-only checks show gateway JSON bodies carry the same filter, annotation and reasoning extras, which the strict validator refuses. + - *Gateway cache counts map to the port's cache subcounts.* Two backends report Anthropic-style `cache_read_input_tokens` and `cache_creation_input_tokens` at the top level of `usage`, sometimes with a `cache_creation` breakdown by retention; the content-filtered backend reports OpenAI-style `prompt_tokens_details.cached_tokens` and `cache_write_tokens`. Live checks showed both styles are subsets of `prompt_tokens`: an identical 10,420-token prompt reported `prompt_tokens` 10,420 with 10,403 tokens written, then 10,403 read, then neither, and `total_tokens` stayed prompt plus completion. That matches `TokenUsage`, whose cache counts are subcounts never added to totals again. In streamed frames the profile therefore moves the top-level counts into `prompt_tokens_details` before validation and forwarding, so they become `cache_read_tokens` and `cache_write_tokens`; the `cache_creation` breakdown must use only the two known retention keys and sum to the creation count, and is then dropped. A nested count that conflicts with its top-level twin, a negative count or a cache sum above `prompt_tokens` refuses as `invalid_response`. The strict profile, and the strict nonstreaming fallback under either profile, leave top-level cache counts missing rather than guessing their meaning. Tether's request mapping sends no cache markers, so these counts are nonzero only when a deployment caches on its own. - **Configuration is explicit.** The adapter reads no environment variable. The application builds its settings object and passes the tier map and the identity mode in. - **Never the only backend.** Every feature of the core runs against the scripted model and against the default adapter on a plain OpenAI-compatible server. The adapter conformance matrix includes at least one backend that is not the Databricks AI Gateway. No contract field, port method, or default exists only to serve the Databricks backend. - **Gateway authoritative history.** CONTRACT-005 makes inference-table model exchanges a first-class selected HistoryProvider source, composed with a writable supplement. The execution journal holds facts and content refs; a gateway request is the sent model projection, not the full original conversation. Reader discovery/query authentication uses the explicit calling actor, bounded SQL and configured locators; no schema mutation or denied-user fallback. The profile may use the platform SDK for these adapter-owned reads as well as authentication. Delayed/missing/partial/ambiguous observations are explicit. diff --git a/docs/helix/02-design/contracts/CONTRACT-001-model-port-and-test-doubles.md b/docs/helix/02-design/contracts/CONTRACT-001-model-port-and-test-doubles.md index 2b6f310..d833956 100644 --- a/docs/helix/02-design/contracts/CONTRACT-001-model-port-and-test-doubles.md +++ b/docs/helix/02-design/contracts/CONTRACT-001-model-port-and-test-doubles.md @@ -114,7 +114,7 @@ async def checked_stream(port: ModelPort, request: ModelRequest, *, max_text_byt | Concurrency | An adapter MUST be safe to call from many tasks at once. Caller mode uses only that request's credential; application mode uses only the explicitly configured application identity; anonymous mode sends no authorization. No mode inherits identity from another task. | | Credential | Caller mode reads credential.authorization() once per physical attempt, never caches it across attempts, and refuses missing/empty/expired credentials before egress without application fallback. Application/anonymous modes are explicit request selections; passing a caller credential in either is identity_mode_conflict rather than silent credential substitution. Adapter construction validates the selected application credential mechanism. No credential is logged, persisted or included in an error reason. | | Retries | An adapter MUST NOT retry in more than one layer; the layer that retries owns the policy. | -| Stream profile | The compatible adapter's stream profile is an explicit selection; the default `openai-chat-text-tools-v1` refuses any field outside its chunk shape and forwards original bytes. A dialect profile (`databricks-ai-gateway-stream-v1`, ADR-013) MUST check each extra field against a closed allow-list, MUST forward only normalized frames that the strict validator re-checks, and MUST NOT turn dropped non-text parts into text, deltas or usage. Unknown fields refuse under every profile. The receipt's `schema_profile` names the selected profile. A stream-validation failure carries the observed header request id and receipt. | +| Stream profile | The compatible adapter's stream profile is an explicit selection; the default `openai-chat-text-tools-v1` refuses any field outside its chunk shape and forwards original bytes. A dialect profile (`databricks-ai-gateway-stream-v1`, ADR-013) MUST check each extra field against a closed allow-list, MUST forward only normalized frames that the strict validator re-checks, and MUST NOT turn dropped non-text parts into text, deltas or usage. It MAY map a provider's cache counts to `cache_read_tokens`/`cache_write_tokens` only after their semantics are qualified as subcounts of the input total, and MUST refuse counts that exceed it or conflict. Unknown fields refuse under every profile. The receipt's `schema_profile` names the selected profile. A stream-validation failure carries the observed header request id and receipt. | | Terminal outcome | The adapter normalizes only documented wire terminal reasons to `finish_reason`; missing/unrecognized reasons are `unknown`. A stream without a terminal response is `ModelUnavailable("incomplete_stream")`. Only `stop` without calls or `tool_calls` with complete valid calls permits continuation/final success. Truncated, filtered, unknown or structurally inconsistent outcomes MUST NOT dispatch a call or be labelled final success. A structurally valid bounded `length`, `filtered` or `unknown` response remains an observed terminal for CONTRACT-002 incomplete-text persistence; it is not a transport outage. Structural inconsistency, invalid arguments and byte-limit failures remain explicit failures. A native SDK default cannot promote a missing/unrecognized wire reason from `unknown` to `stop`; exact SDK text/call projection parity is required when SDK normalization succeeds. If the public SDK rejects an unrecognized reason, a complete bounded response that passed Tether wire validation and successful recorder observation MAY be returned unchanged only as non-executable `unknown` evidence, after every local transport/recorder error has propagated. This validated unknown-terminal fallback makes no SDK projection-parity claim and MUST NOT rewrite captured bytes, retry, dispatch calls or expose SDK diagnostics. | | Wire integrity | Invalid JSON arguments, duplicate/ambiguous call ids in one reply, invalid names or incomplete calls MUST be rejected, never repaired into executable arguments. Provider-specific history formatting may add clearly synthetic non-executable separators; it MUST preserve the original log, invocation identity and effect status. | | Bounds | `max_output_tokens` is mapped to the configured endpoint output-limit field. A conflicting reserved value in `extra` is `ModelUnavailable("parameter_conflict")`. Oversized/unmeasurable input is rejected by CONTRACT-004 before transport; the adapter caps response bytes and tool argument bytes by explicit settings before materialization. | diff --git a/docs/helix/03-test/evidence/databricks-gateway-cache-counts-20261108.json b/docs/helix/03-test/evidence/databricks-gateway-cache-counts-20261108.json new file mode 100644 index 0000000..21b4c02 --- /dev/null +++ b/docs/helix/03-test/evidence/databricks-gateway-cache-counts-20261108.json @@ -0,0 +1,260 @@ +{ + "request": "Gateway profile cache counts", + "recorded": "2026-11-08", + "tether_commit": "993a06604164bbdd70526fe0f50c13eee13a0d7b", + "tether_commit_note": "Rows were run on an earlier working tree (parent b1a7d57) whose stream cache-count mapping, gateway_usage applied to streamed chunks, is identical to this commit; that tree also normalized nonstreaming bodies, which this commit does not, and no recorded row used that path.", + "python": "3.11.17", + "workspace": "development workspace", + "route": "POST /ai-gateway/mlflow/v1/chat/completions", + "service_name_pattern": "ai.gateway.-", + "identity_mode": "caller", + "stream_profile": "databricks-ai-gateway-stream-v1", + "adapter": "OpenAICompatModel through complete() (streaming)", + "transport": "real HTTP (httpx AsyncHTTPTransport, retries=0) wrapped by a call counter", + "max_attempts": 1, + "cache_marker": "Probe-only: for cache rows the test transport marks the system prompt with an ephemeral cache_control after Tether records the request; Tether's request mapping sends no cache markers.", + "recorded_fields": "statuses, content types, outcomes, finish reasons, booleans and integer token counts only; no token, host, service name, request id or text", + "rows": [ + { + "tier": "quick", + "mode": "stream", + "outcome": "response", + "finish": "stop", + "content_nonempty": true, + "schema_profile_is_gateway": true, + "usage": { + "input_tokens": 37, + "output_tokens": 29, + "cache_read_tokens": 0, + "cache_write_tokens": 0, + "reasoning_tokens": 19 + }, + "physical_calls": 1, + "statuses": [ + 200 + ], + "content_types": [ + "text/event-stream" + ] + }, + { + "tier": "balanced", + "mode": "stream", + "outcome": "response", + "finish": "stop", + "content_nonempty": true, + "schema_profile_is_gateway": true, + "usage": { + "input_tokens": 48, + "output_tokens": 34, + "cache_read_tokens": 0, + "cache_write_tokens": 0, + "reasoning_tokens": null + }, + "physical_calls": 1, + "statuses": [ + 200 + ], + "content_types": [ + "text/event-stream" + ] + }, + { + "tier": "deep", + "mode": "stream", + "outcome": "response", + "finish": "stop", + "content_nonempty": true, + "schema_profile_is_gateway": true, + "usage": { + "input_tokens": 48, + "output_tokens": 27, + "cache_read_tokens": 0, + "cache_write_tokens": 0, + "reasoning_tokens": null + }, + "physical_calls": 1, + "statuses": [ + 200 + ], + "content_types": [ + "text/event-stream" + ] + }, + { + "tier": "deep", + "mode": "cache-0", + "outcome": "response", + "finish": "stop", + "content_nonempty": true, + "schema_profile_is_gateway": true, + "usage": { + "input_tokens": 10420, + "output_tokens": 5, + "cache_read_tokens": 0, + "cache_write_tokens": 10403, + "reasoning_tokens": null + }, + "physical_calls": 1, + "statuses": [ + 200 + ], + "content_types": [ + "text/event-stream" + ] + }, + { + "tier": "deep", + "mode": "cache-1", + "outcome": "response", + "finish": "stop", + "content_nonempty": true, + "schema_profile_is_gateway": true, + "usage": { + "input_tokens": 10420, + "output_tokens": 5, + "cache_read_tokens": 10403, + "cache_write_tokens": 0, + "reasoning_tokens": null + }, + "physical_calls": 1, + "statuses": [ + 200 + ], + "content_types": [ + "text/event-stream" + ] + }, + { + "tier": "balanced", + "mode": "cache-0", + "outcome": "response", + "finish": "stop", + "content_nonempty": true, + "schema_profile_is_gateway": true, + "usage": { + "input_tokens": 10420, + "output_tokens": 6, + "cache_read_tokens": 0, + "cache_write_tokens": 10403, + "reasoning_tokens": null + }, + "physical_calls": 1, + "statuses": [ + 200 + ], + "content_types": [ + "text/event-stream" + ] + }, + { + "tier": "balanced", + "mode": "cache-1", + "outcome": "response", + "finish": "stop", + "content_nonempty": true, + "schema_profile_is_gateway": true, + "usage": { + "input_tokens": 10420, + "output_tokens": 6, + "cache_read_tokens": 10403, + "cache_write_tokens": 0, + "reasoning_tokens": null + }, + "physical_calls": 1, + "statuses": [ + 200 + ], + "content_types": [ + "text/event-stream" + ] + } + ], + "structure_probe": { + "method": "Direct HTTP posts outside Tether; keys, types and enum labels only, plus integer usage counts", + "nonstreaming": { + "note": "Observed for scoping only; nonstreaming bodies remain on the strict validator and are not covered by this profile.", + "quick": "object chat.completion; top-level prompt_filter_results (category map sometimes empty), routing{serving_pipereplica:str}, service_tier:str, system_fingerprint:null; choice content_filter_results, logprobs:null; message annotations:[], refusal:null, content:str; usage.latency_checkpoint{8 int fields}, prompt_tokens_details{audio_tokens,cached_tokens,cache_write_tokens}, completion_tokens_details{accepted_prediction_tokens,audio_tokens,reasoning_tokens,rejected_prediction_tokens}", + "deep": "object chat.completion; choice {index,message,finish_reason}; message content is str, or with a reasoning-inviting prompt and max_tokens 2048 a list [{type:reasoning, summary:[{type:summary_text,text,signature}]}, {type:text,text}] (3 of 3 calls); usage {prompt_tokens, completion_tokens, total_tokens, cache_read_input_tokens, cache_creation_input_tokens}", + "balanced": "same body and usage keys as deep, string content" + }, + "cache_semantics": [ + { + "tier": "quick", + "marker": "none (automatic)", + "calls": [ + { + "prompt_tokens": 6020, + "cached_tokens": 0, + "cache_write_tokens": 6017, + "total_tokens": 6024 + }, + { + "prompt_tokens": 6020, + "cached_tokens": 6017, + "cache_write_tokens": 0, + "total_tokens": 6024 + } + ] + }, + { + "tier": "deep", + "marker": "none", + "calls": [ + { + "prompt_tokens": 10420, + "cache_read_input_tokens": 0, + "cache_creation_input_tokens": 0, + "total_tokens": 10425 + } + ] + }, + { + "tier": "deep", + "marker": "ephemeral cache_control on system part", + "calls": [ + { + "prompt_tokens": 10420, + "cache_read_input_tokens": 0, + "cache_creation_input_tokens": 10403, + "cache_creation": { + "ephemeral_5m_input_tokens": 10403, + "ephemeral_1h_input_tokens": 0 + }, + "total_tokens": 10425 + }, + { + "prompt_tokens": 10420, + "cache_read_input_tokens": 10403, + "cache_creation_input_tokens": 0, + "total_tokens": 10425 + } + ] + }, + { + "tier": "balanced", + "marker": "ephemeral cache_control on system part", + "calls": [ + { + "prompt_tokens": 10420, + "cache_read_input_tokens": 0, + "cache_creation_input_tokens": 10403, + "total_tokens": 10426 + }, + { + "prompt_tokens": 10420, + "cache_read_input_tokens": 10403, + "cache_creation_input_tokens": 0, + "total_tokens": 10426 + } + ] + } + ], + "conclusion": "prompt_tokens is unchanged whether tokens are written, read or neither, and total_tokens is prompt plus completion; both cache styles are subsets of prompt_tokens" + }, + "limitations": [ + "One signed-in human user; identity checks are covered by the earlier caller-identity receipt.", + "Cache rows use a probe-only cache marker added below Tether.", + "Nonstreaming bodies, including the strict stream fallback, still drop top-level cache counts." + ] +} diff --git a/docs/helix/03-test/evidence/databricks-gateway-cache-counts-20261108.md b/docs/helix/03-test/evidence/databricks-gateway-cache-counts-20261108.md new file mode 100644 index 0000000..2f65543 --- /dev/null +++ b/docs/helix/03-test/evidence/databricks-gateway-cache-counts-20261108.md @@ -0,0 +1,58 @@ +# Gateway cache-count live receipt (recorded 2026-11-08) + +Machine-readable form: +[databricks-gateway-cache-counts-20261108.json](databricks-gateway-cache-counts-20261108.json). + +With the `databricks-ai-gateway-stream-v1` profile selected (ADR-013, CONTRACT-001), the deep +and balanced tiers' top-level cache counts now reach `TokenUsage` as `cache_read_tokens` and +`cache_write_tokens`. Before this change the profile forwarded them and the strict usage +reader dropped them, so both were missing. The quick tier's nested OpenAI counts were already +mapped and are unchanged. + +- Tether commit: see the JSON `tether_commit` and its note +- Route: `POST /ai-gateway/mlflow/v1/chat/completions`; services by tier alias +- Caller identity mode; real HTTP (`retries=0`) wrapped by a call counter; `max_attempts=1` +- Recorded: statuses, content types, outcomes, finish reasons, booleans and token counts only + +| Tier | Call | Outcome | Input | Output | Cache read | Cache write | +|---|---|---|---|---|---|---| +| quick | plain | response, `stop` | 37 | 29 | 0 | 0 | +| balanced | plain | response, `stop` | 48 | 34 | 0 | 0 | +| deep | plain | response, `stop` | 48 | 27 | 0 | 0 | +| deep | cache marker, 1st | response, `stop` | 10420 | 5 | 0 | 10403 | +| deep | cache marker, 2nd | response, `stop` | 10420 | 5 | 10403 | 0 | +| balanced | cache marker, 1st | response, `stop` | 10420 | 6 | 0 | 10403 | +| balanced | cache marker, 2nd | response, `stop` | 10420 | 6 | 10403 | 0 | + +Every row is one streaming physical call with status 200, `text/event-stream`, nonempty +content and a receipt whose `schema_profile` is the gateway profile. + +## Cache-count semantics + +Direct HTTP posts outside Tether (structure and integer counts only) settled what the counts +mean before the mapping was chosen: + +- deep and balanced report `cache_read_input_tokens` and `cache_creation_input_tokens` at the + top level of `usage`, plus a `cache_creation` breakdown + (`ephemeral_5m_input_tokens`, `ephemeral_1h_input_tokens`) on a write. Without a cache marker + a 10,420-token prompt reports zero for both, so they do not cache automatically. +- With a cache marker on the system part, the same prompt reports `prompt_tokens` 10,420 on the + write (10,403 created), on the read (10,403 read) and uncached; `total_tokens` stays prompt + plus completion. The counts are therefore subsets of `prompt_tokens`, the same meaning as + OpenAI's nested `cached_tokens` and the port's cache subcounts. +- quick caches automatically and reports the nested OpenAI fields: `prompt_tokens` 6,020 with + 6,017 written, then 6,017 `cached_tokens`. + +Fabricated-fixture tests for the mapping and its negative controls live in +`tests/test_gateway_cache_counts.py`. + +## Limitations + +- Cache rows add the cache marker in the test transport, below Tether, after Tether records + the request. Tether's request mapping sends no cache markers, so in ordinary use deep and + balanced report zero unless a deployment caches on its own. +- Nonstreaming bodies, including the strict fallback when a server ignores `stream: true`, are + not covered: they keep the strict validator and still drop top-level cache counts. Their + observed structure is recorded in the JSON for scoping only. +- One signed-in human user; caller-identity checks are covered by the earlier + [stream dialect receipt](databricks-gateway-stream-dialect-20261108.md). diff --git a/src/tether/adapters/pydantic_ai/gateway.py b/src/tether/adapters/pydantic_ai/gateway.py index c08ffd1..2bfb4af 100644 --- a/src/tether/adapters/pydantic_ai/gateway.py +++ b/src/tether/adapters/pydantic_ai/gateway.py @@ -2,11 +2,14 @@ Every gateway-specific field is checked against a closed allow-list and then removed, so the strict chunk validator and the SDK only ever see the `openai-chat-text-tools-v1` chunk shape. -Unknown fields, unknown filter structure and unknown content parts still refuse. +Unknown fields, unknown filter structure and unknown content parts still refuse. Top-level +gateway cache counts move to the nested OpenAI usage fields. """ from __future__ import annotations +from typing import TypeGuard + from pydantic import JsonValue from tether.types import ModelUnavailable @@ -21,13 +24,18 @@ DELTA_KEYS = frozenset({"role", "content", "tool_calls"}) FILTER_ENTRY_KEYS = frozenset({"filtered", "severity", "detected"}) SUMMARY_KEYS = frozenset({"type", "text", "signature"}) +CACHE_COUNTS = ( + ("cache_read_input_tokens", "cached_tokens"), + ("cache_creation_input_tokens", "cache_write_tokens"), +) +CACHE_CREATION_KEYS = frozenset({"ephemeral_5m_input_tokens", "ephemeral_1h_input_tokens"}) def _unsupported() -> ModelUnavailable: return ModelUnavailable("unsupported_content") -def _count(value: JsonValue) -> bool: +def _count(value: JsonValue) -> TypeGuard[int]: return isinstance(value, int) and not isinstance(value, bool) and value >= 0 @@ -95,6 +103,52 @@ def gateway_metadata(value: dict[str, JsonValue]) -> None: raise _unsupported() +def _cache_creation(usage: dict[str, JsonValue]) -> None: + detail = usage.pop("cache_creation", None) + if detail is None: + return + if ( + not isinstance(detail, dict) + or set(detail) - CACHE_CREATION_KEYS + or not all(_count(item) for item in detail.values()) + ): + raise _unsupported() + total = usage.get("cache_creation_input_tokens") + if total is not None and total != sum(item for item in detail.values() if _count(item)): + raise ModelUnavailable("invalid_response") + + +def gateway_usage(value: JsonValue) -> JsonValue: + """Move top-level cache counts to the nested OpenAI fields; drop the retention breakdown. + + Live evidence shows both cache counts are subsets of `prompt_tokens`, matching the nested + `cached_tokens` semantics (ADR-013), so their sum may not exceed it. + """ + if not isinstance(value, dict): + return value + usage = dict(value) + _cache_creation(usage) + details = usage.get("prompt_tokens_details") + nested = dict(details) if isinstance(details, dict) else None + for flat, field in CACHE_COUNTS: + count = usage.pop(flat, None) + if count is None: + continue + if nested is None and details is not None: + raise ModelUnavailable("invalid_response") + nested = {} if nested is None else nested + if not _count(count) or nested.get(field, count) != count: + raise ModelUnavailable("invalid_response") + nested[field] = count + if nested is not None: + usage["prompt_tokens_details"] = nested + cached = [item for item in (nested.get(f) for _, f in CACHE_COUNTS) if _count(item)] + prompt = usage.get("prompt_tokens") + if _count(prompt) and sum(cached) > prompt: + raise ModelUnavailable("invalid_response") + return usage + + class GatewayStreamDialect: """Normalize one stream's gateway chunks to the strict chunk shape, or refuse.""" @@ -116,6 +170,8 @@ def chunk(self, value: JsonValue) -> dict[str, JsonValue] | None: choices = value.get("choices") if isinstance(choices, list): normalized["choices"] = [self._choice(choice) for choice in choices] + if "usage" in value: + normalized["usage"] = gateway_usage(value["usage"]) return normalized def finish(self, finish_reason: str | None) -> None: diff --git a/tests/test_gateway_cache_counts.py b/tests/test_gateway_cache_counts.py new file mode 100644 index 0000000..d370915 --- /dev/null +++ b/tests/test_gateway_cache_counts.py @@ -0,0 +1,224 @@ +"""Gateway cache counts under the explicitly selected stream dialect, on fabricated frames only. + +The usage shapes mirror the field structure observed from gateway backends: top-level +`cache_read_input_tokens`/`cache_creation_input_tokens` with an optional `cache_creation` +retention breakdown, and the nested OpenAI `prompt_tokens_details` fields. Every value is +fabricated. +""" + +from __future__ import annotations + +import json +from typing import Any + +import httpx2 as httpx +import pytest + +from tether.adapters.openai_compat import OpenAICompatModel +from tether.adapters.pydantic_ai.gateway import gateway_usage +from tether.adapters.pydantic_ai.wire import ( + GATEWAY_STREAM_PROFILE, + STRICT_STREAM_PROFILE, + StreamingWireValidator, + usage_from_wire, +) +from tether.models import complete +from tether.types import ModelExecutionLimits, ModelRequest, ModelUnavailable, TokenUsage + +pytestmark = pytest.mark.anyio + +LIMITS = ModelExecutionLimits( + max_response_bytes=16384, + max_tool_argument_bytes=64, + max_attempts=1, + request_timeout_seconds=5, + max_retry_delay_seconds=0.001, +) +FLAT = {"cache_read_input_tokens", "cache_creation_input_tokens", "cache_creation"} + + +@pytest.fixture +def anyio_backend(): + return "asyncio" + + +def flat_usage(read: int = 90, write: int = 0, **extra: Any) -> dict[str, Any]: + return { + "prompt_tokens": 100, + "completion_tokens": 5, + "total_tokens": 105, + "cache_read_input_tokens": read, + "cache_creation_input_tokens": write, + } | extra + + +def write_usage() -> dict[str, Any]: + return flat_usage( + 0, + 90, + cache_creation={"ephemeral_5m_input_tokens": 90, "ephemeral_1h_input_tokens": 0}, + ) + + +def _chunk(delta: dict[str, Any], usage: dict[str, Any] | None, finish=None) -> dict[str, Any]: + return { + "id": "fabricated-completion", + "object": "chat.completion.chunk", + "created": 1, + "model": "fabricated-model", + "choices": [{"index": 0, "delta": delta, "finish_reason": finish}], + "usage": usage, + } + + +def frames(final: dict[str, Any]) -> list[dict[str, Any]]: + interim = final | {"completion_tokens": None, "total_tokens": None} + return [ + _chunk({"role": "assistant", "content": "fabricated "}, interim), + _chunk({"content": "answer"}, final, "stop"), + ] + + +def sse(values: list[dict[str, Any]]) -> bytes: + events = b"".join(b"data: " + json.dumps(value).encode() + b"\n\n" for value in values) + return events + b"data: [DONE]\n\n" + + +def run(final: dict[str, Any], profile: str = GATEWAY_STREAM_PROFILE): + validator = StreamingWireValidator((), LIMITS, profile=profile) + forwarded = validator.feed(sse(frames(final))) + return validator.finish(), forwarded + + +def forwarded_usage(data: bytes) -> list[Any]: + events = [line.removeprefix(b"data: ") for line in data.split(b"\n\n") if line] + return [json.loads(event)["usage"] for event in events[:-1]] + + +@pytest.mark.parametrize(("usage", "cache"), [(flat_usage(), (90, 0)), (write_usage(), (0, 90))]) +def test_gateway_cache_counts_map_to_the_port_subcounts(usage, cache): + response, forwarded = run(usage) + assert response.usage is not None + assert (response.usage.input_tokens, response.usage.output_tokens) == (100, 5) + assert (response.usage.cache_read_tokens, response.usage.cache_write_tokens) == cache + for item in forwarded_usage(forwarded): + assert not set(item) & FLAT + assert item["prompt_tokens_details"] == { + "cached_tokens": cache[0], + "cache_write_tokens": cache[1], + } + # The forwarded frames carry the nested shape, which the strict profile maps identically. + strict = StreamingWireValidator((), LIMITS) + strict.feed(forwarded) + assert strict.finish() == response + + +def test_strict_profile_leaves_top_level_cache_counts_missing(): + response, forwarded = run(flat_usage(), STRICT_STREAM_PROFILE) + assert response.usage == TokenUsage(input_tokens=100, output_tokens=5) + assert forwarded == sse(frames(flat_usage())) + assert usage_from_wire(flat_usage()) == TokenUsage(input_tokens=100, output_tokens=5) + + +def test_nested_counts_are_kept_and_matching_twins_accepted(): + nested = {"audio_tokens": 0, "cached_tokens": 90, "cache_write_tokens": 0} + usage = flat_usage(prompt_tokens_details=nested) + assert gateway_usage(usage) == { + "prompt_tokens": 100, + "completion_tokens": 5, + "total_tokens": 105, + "prompt_tokens_details": nested, + } + only_nested = {"prompt_tokens": 100, "completion_tokens": 5, "prompt_tokens_details": nested} + assert gateway_usage(only_nested) == only_nested + + +def test_usage_without_cache_counts_is_unchanged(): + usage: dict[str, Any] = {"prompt_tokens": 7, "completion_tokens": 2, "total_tokens": 9} + assert gateway_usage(usage) == usage + assert gateway_usage(None) is None + response, _ = run(usage) + assert response.usage == TokenUsage(input_tokens=7, output_tokens=2) + + +def test_interim_usage_with_null_counts_still_maps_cache_counts(): + interim = flat_usage() | {"completion_tokens": None, "total_tokens": None} + mapped = gateway_usage(interim) + assert isinstance(mapped, dict) + assert mapped["prompt_tokens_details"] == {"cached_tokens": 90, "cache_write_tokens": 0} + + +def _with(usage: dict[str, Any], **changes: Any) -> dict[str, Any]: + return usage | changes + + +@pytest.mark.parametrize( + ("usage", "reason"), + [ + (flat_usage(read=101), "invalid_response"), + (flat_usage(read=60, write=41), "invalid_response"), + (flat_usage(read=-1), "invalid_response"), + (flat_usage(write=True), "invalid_response"), + (_with(flat_usage(), cache_read_input_tokens="90"), "invalid_response"), + (_with(flat_usage(), prompt_tokens_details={"cached_tokens": 89}), "invalid_response"), + (_with(flat_usage(), prompt_tokens_details="x"), "invalid_response"), + ( + _with(flat_usage(read=0), prompt_tokens_details={"cached_tokens": 101}), + "invalid_response", + ), + (_with(write_usage(), cache_creation=[]), "unsupported_content"), + ( + _with(write_usage(), cache_creation={"ephemeral_5m_input_tokens": 90, "x": 0}), + "unsupported_content", + ), + ( + _with(write_usage(), cache_creation={"ephemeral_5m_input_tokens": -1}), + "unsupported_content", + ), + ( + _with(write_usage(), cache_creation={"ephemeral_5m_input_tokens": 89}), + "invalid_response", + ), + ], +) +def test_malformed_or_inconsistent_cache_counts_refuse(usage, reason): + with pytest.raises(ModelUnavailable) as caught: + run(usage) + assert caught.value.reason == reason + + +def test_cache_creation_without_a_total_is_dropped(): + usage = write_usage() + del usage["cache_creation_input_tokens"] + mapped = gateway_usage(usage) + assert isinstance(mapped, dict) + assert "cache_creation" not in mapped + assert mapped["prompt_tokens_details"] == {"cached_tokens": 0} + + +async def test_adapter_reports_gateway_cache_counts(model_request: ModelRequest): + def handler(request: httpx.Request) -> httpx.Response: + assert json.loads(request.content)["stream"] is True + return httpx.Response( + 200, + content=sse(frames(write_usage())), + headers={"content-type": "text/event-stream", "x-request-id": "fabricated-gw-id"}, + ) + + model = OpenAICompatModel( + base_url="https://fabricated-workspace.invalid/ai-gateway/mlflow/v1", + tiers={"demo": "fabricated-service"}, + limits=LIMITS, + output_limit_field="max_tokens", + supports_parallel_tool_calls=False, + request_tags_header="Databricks-Ai-Gateway-Request-Tags", + stream_profile=GATEWAY_STREAM_PROFILE, + transport=httpx.MockTransport(handler), + ) + try: + response = await complete(model, model_request) + finally: + await model.aclose() + assert response.content == "fabricated answer" + assert response.usage is not None + assert (response.usage.cache_read_tokens, response.usage.cache_write_tokens) == (0, 90)