Repository navigation
openai-chat: keep the client's cache breakpoints on system and tool_result - #148
Merged
Merged
Conversation
…esult A /v1/messages request served by an openai-chat model flattened its system blocks into a string system message and each tool_result into a string tool message, so their cache_control markers never reached the vendor. Through OpenRouter to a Claude model, an agent loop (system breakpoint plus a breakpoint on the trailing tool_result) never read its cache: turns ending on a user text block paid the 1.25x write, and tool_result turns read nothing. The system message now carries the native blocks as content parts, and a marked tool_result becomes a one-part text array that keeps its marker. OpenRouter honors both shapes; OpenAI, DeepSeek, Qwen, Moonshot, xAI, Gemini's compatible endpoint and MiniMax accept them. Tool-level markers stay dropped: Gemini's compatible endpoint rejects the field, and a later breakpoint covers the tools prefix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A
/v1/messagesrequest served by anopenai-chatmodel lost its prompt-cache breakpoints:systemblocks were flattened into a string system message, which dropped theircache_control.tool_resultwas flattened into a stringrole: toolmessage, which dropped itscache_control.An agent loop through OpenRouter to a Claude model puts a breakpoint on the system prompt and another on the trailing
tool_result, and it never read its cache. Turns that ended on a user text block, whose marker did pass through, paid the 1.25× cache-write price. Turns that ended on atool_resultread nothing. The billedtotal_tokensrunning above prompt + completion was that write premium, not an overcount: cost matches OpenRouter's reportedcost.Fix
tool_resultwithcache_controlbecomes a one-part text array that keeps its marker. An unmarked one stays a string.cache_controlfield on a tool (400), and any later breakpoint already covers the tools prefix.Evidence
Gates (macOS):
cargo fmt --checkandcargo clippy --all-targets -- -D warningsare clean.cargo testpasses 705, fails 0, ignores 4. The PG/Redis env-gated suites were not run because no storage code changed.Fail-first:
native_system_blocks_stay_inside_the_openai_wire_shapesandtool_result_cache_control_survives_the_openai_wireboth fail on the unfixed source.Live test: a release build sent requests through a recording proxy to OpenRouter, model
anthropic/claude-sonnet-4.6. Each shape went out twice; the table shows the second call's cache read.tool_result(+ system)In a 4-turn growing agent loop, cache reads are 0 / 3885 / 3968 / 4051 tokens after the fix. On every ledger row,
cost_microsis within 1 micro of the vendor-reported cost.Vendor acceptance (direct calls): a system content-part array (with and without
cache_control) and a tool message content array withcache_controlboth return 200 on OpenAI, DeepSeek, Qwen (compatible mode), Moonshot, xAI, MiniMax, OpenRouter and Gemini's OpenAI-compatible endpoint.