Repository navigation
Map gateway cache counts in the gateway stream profile - #93
Merged
Merged
Conversation
With databricks-ai-gateway-stream-v1 selected, the gateway's top-level cache_read_input_tokens and cache_creation_input_tokens move into the nested OpenAI prompt_tokens_details fields before validation and forwarding, so they reach TokenUsage as cache_read_tokens and cache_write_tokens. Live evidence shows both are subsets of prompt_tokens. A cache sum above prompt_tokens, a conflicting nested count or a malformed cache_creation retention breakdown refuses; the breakdown is otherwise dropped. The strict profile and the nonstreaming fallback are unchanged. Co-authored-by: Erik LaBianca <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
ADR-013 and CONTRACT-001 now map the gateway's top-level cache counts to the port's cache subcounts in streamed frames, after live checks showed they are subsets of prompt_tokens, and state that nonstreaming bodies stay on the strict validator. The redacted receipt records the deep and balanced cache counts reaching TokenUsage. Co-authored-by: Erik LaBianca <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With
databricks-ai-gateway-stream-v1selected, the deep and balanced gateway backends' top-levelcache_read_input_tokens/cache_creation_input_tokensnow reachTokenUsageascache_read_tokens/cache_write_tokens. Before, the profile forwarded them and the strict usage reader dropped them.gateway_usage(src/tether/adapters/pydantic_ai/gateway.py) moves the top-level counts intoprompt_tokens_detailsbefore validation and forwarding, so the SDK and the strict re-check see only the nested OpenAI shape.prompt_tokens, a nested count conflicting with its top-level twin, negative/non-integer counts, and acache_creationretention breakdown with unknown keys or a sum that differs from the creation count. The breakdown is otherwise dropped._capture→validate_wire→synthetic_stream) are unchanged and still leave top-level cache counts missing.Evidence for the semantics
Structure-and-counts-only live checks: an identical 10,420-token prompt reported
prompt_tokens10,420 whether 10,403 tokens were written, read or neither, andtotal_tokensstayed prompt plus completion. The counts are therefore subsets ofprompt_tokens, matching OpenAI's nestedcached_tokensand the port's cache subcounts. These backends report nonzero counts only when the request carries a cache marker; Tether sends none, so the live cache rows add one in the test transport below Tether (stated in the receipt).Recorded in ADR-013 and CONTRACT-001; redacted receipt at
docs/helix/03-test/evidence/databricks-gateway-cache-counts-20261108.{md,json}(no token, host, service name or request id).Scope
The nonstreaming half of the original task was dropped at the coordinating session's request: nonstreaming bodies stay on the strict validator, and this PR does not touch
_retry,_once,request_nativeorvalidate_wire.Test plan
tests/test_gateway_cache_counts.py: fabricated-fixture mapping, strict-profile parity, adapter end to end, and negative controls.make checkpasses locally (87% branch coverage). Two earlier runs hit unrelated environment issues: a terminal-probe test that passed 3/3 alone, and the embed example returning 503 because this fresh worktree lacked the gitignored built web-uiassets.🤖 Generated with Claude Code