Skip to content

Map gateway cache counts in the gateway stream profile - #93

Merged
easel merged 2 commits into
mainfrom
claude/goofy-mcnulty-c274f2
Nov 9, 2026
Merged

easel merged 2 commits into
mainfrom
claude/goofy-mcnulty-c274f2

Conversation

@easel

@easel easel commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Summary

With databricks-ai-gateway-stream-v1 selected, the deep and balanced gateway backends' top-level cache_read_input_tokens / cache_creation_input_tokens now reach TokenUsage as cache_read_tokens / cache_write_tokens. Before, the profile forwarded them and the strict usage reader dropped them.

  • gateway_usage (src/tether/adapters/pydantic_ai/gateway.py) moves the top-level counts into prompt_tokens_details before validation and forwarding, so the SDK and the strict re-check see only the nested OpenAI shape.
  • Refuses: cache sum above prompt_tokens, a nested count conflicting with its top-level twin, negative/non-integer counts, and a cache_creation retention breakdown with unknown keys or a sum that differs from the creation count. The breakdown is otherwise dropped.
  • The strict profile and the nonstreaming fallback (_capture → validate_wire → synthetic_stream) are unchanged and still leave top-level cache counts missing.

Evidence for the semantics

Structure-and-counts-only live checks: an identical 10,420-token prompt reported prompt_tokens 10,420 whether 10,403 tokens were written, read or neither, and total_tokens stayed prompt plus completion. The counts are therefore subsets of prompt_tokens, matching OpenAI's nested cached_tokens and the port's cache subcounts. These backends report nonzero counts only when the request carries a cache marker; Tether sends none, so the live cache rows add one in the test transport below Tether (stated in the receipt).

Recorded in ADR-013 and CONTRACT-001; redacted receipt at docs/helix/03-test/evidence/databricks-gateway-cache-counts-20261108.{md,json} (no token, host, service name or request id).

Scope

The nonstreaming half of the original task was dropped at the coordinating session's request: nonstreaming bodies stay on the strict validator, and this PR does not touch _retry, _once, request_native or validate_wire.

Test plan

  • tests/test_gateway_cache_counts.py: fabricated-fixture mapping, strict-profile parity, adapter end to end, and negative controls.
  • make check passes locally (87% branch coverage). Two earlier runs hit unrelated environment issues: a terminal-probe test that passed 3/3 alone, and the embed example returning 503 because this fresh worktree lacked the gitignored built web-ui assets.

🤖 Generated with Claude Code

easel and others added 2 commits November 8, 2026 17:26
With databricks-ai-gateway-stream-v1 selected, the gateway's top-level
cache_read_input_tokens and cache_creation_input_tokens move into the nested
OpenAI prompt_tokens_details fields before validation and forwarding, so they
reach TokenUsage as cache_read_tokens and cache_write_tokens. Live evidence
shows both are subsets of prompt_tokens. A cache sum above prompt_tokens, a
conflicting nested count or a malformed cache_creation retention breakdown
refuses; the breakdown is otherwise dropped. The strict profile and the
nonstreaming fallback are unchanged.

Co-authored-by: Erik LaBianca <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
ADR-013 and CONTRACT-001 now map the gateway's top-level cache counts to the
port's cache subcounts in streamed frames, after live checks showed they are
subsets of prompt_tokens, and state that nonstreaming bodies stay on the strict
validator. The redacted receipt records the deep and balanced cache counts
reaching TokenUsage.

Co-authored-by: Erik LaBianca <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@easel
easel merged commit 6a08db1 into main Nov 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant