Skip to content

OpenAI: sort request body keys so prompt caches can hit - #432

Merged
mattt merged 1 commit into
huggingface:mainfrom
gtokman:sort-request-body-keys
Nov 20, 2026
Merged

mattt merged 1 commit into
huggingface:mainfrom
gtokman:sort-request-body-keys

Conversation

@gtokman

@gtokman gtokman commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Requests are encoded with a plain JSONEncoder, so the same tools and messages can come out with their keys in a different order on each request. OpenAI's prompt cache, and Azure's, only match an exact byte prefix. When the order changes, no request ever reads from the cache, and each one is billed as a fresh cache write.

This sorts the keys (.sortedKeys) when encoding request bodies, in all three places they're encoded: Chat Completions, Responses, and streaming. It also adds a test.

What we measured: we ran a 15-turn tool-using benchmark on gpt-6.2-sol through Azure:

  • Before: about 1.5M input tokens were written to the cache and none were read, at $4.05 a run.
  • After: 71–93% of input tokens were read from the cache, at about $0.50 a run.

The key-order differences were easy to see in a diff of two consecutive raw requests: one showed up at character 16 of the system message, and another at character 190 of the tools.

@gtokman
gtokman force-pushed the sort-request-body-keys branch from b133b51 to ca63241 Compare November 19, 2026 12:23
Requests were encoded with a plain JSONEncoder, so the same tools and messages came out
in a different key order each time. Providers whose prompt cache matches an exact prefix
(OpenAI's, Azure's) never hit: every call was billed as a cache write.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@gtokman
gtokman force-pushed the sort-request-body-keys branch from ca63241 to 8117244 Compare November 19, 2026 12:24
@mattt
mattt merged commit eb9b055 into huggingface:main Nov 20, 2026
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants