Repository navigation
Serve System One decisions (TypeSafe Jev) on /v1/decisions and /v1/systemone - #149
Merged
Merged
Conversation
…/v1/systemone Jev is a typed decision model, not a chat model: OpenRouter serves it on /api/alpha/decisions and /api/v1/systemone and reports each call's charge in usage.cost. A decisions model is configured, entitled, budgeted and billed like any other; the reply passes through whole, and the ledger charges usage.cost as reported.
Collaborator
Author
Live verificationSetup:
Smoke: a chat completion through the box egress proxy answers 200. Its ledger row is priced from the list as before ( Real Jev call: a Save As accessibility-tree excerpt, one
Per-call reconciliation: each gw ledger row was found by
The ledger stores integer micro-dollars, so 20.916 µ$ rounds to 21. That is at most 0.5 µ$ per call. |
Count decision JSON bytes without buffering the payload so daily quota and TPM reservations include state and questions. Move score criteria into the mock legend and include probabilities for every level. Cover both decision routes with in-flight quota and TPM regressions and validate the complete mock score response.
…or input line POST /v1/batches indexed the client's JSON mutably (body["items"], req["body"] on each input-file line); serde_json panics on IndexMut of a non-object, so a body of [] or a file line of 5 killed the request task.
…Redis recorded; degrade within a protocol - The user-budget store raised SYSTEM_ERROR 503, which is_upstream_fault reads as a gateway-raised upstream failure: a Postgres outage walked the model's fallback chain, and CacheLookup ahead of BudgetGate could answer a healthy model's request with a cached fallback reply. The store now raises DB_READ, which is never a fallback trigger. - A daily reserve the Redis script never recorded (a failed round trip that admits) was still settled or refunded at full size, subtracting from the shared day counter other requests had reserved into. quota_reserve now reports the amount it recorded, 0 for the fail-open admit, the way #99 made the TPM window settle only what it held. - The tenant fallback_model replaced the model of any surface on a quota or moderation degrade; an embeddings, rerank or decisions request degraded into a chat model and failed upstream. A typed family now degrades only to a model whose protocol serves its surface, else it is denied as before.
…verride price it, echo the requested name - /v1/decisions joined alpha/decisions onto the endpoint as written, so a provider endpoint of https://openrouter.ai/api/v1 (OpenRouter's documented base, which versioned_url supports) sent decisions to /api/v1/alpha/decisions: a vendor 404 (verified live; /api/alpha/decisions and /api/v1/systemone answer 200). - A decisions call charged usage.cost even when the tenant set a model_prices override for the model; config accepted the override and billing ignored it. An explicit tenant override now wins; usage.cost is charged when there is none, the list price when the reply has no cost. - The typed families pass the vendor body through whole, so a variant- or fallback-served call showed the served model in the reply's model field. family_response now writes the requested name back, as the Responses engine already does; one site covers embeddings, images, moderations, search, rerank and decisions.
…ils open A tool-id anchor keeps the last four fingerprints because compatible upstreams reuse ids like call_1 across conversations; a fifth conversation on the same key, model and id evicted the oldest, whose next tool turn then reviewed as Mismatch and was refused with 400 on every retry (the refresh on each remember kept the anchor alive). A full anchor no longer holds complete evidence, so its no-match is now Miss, the module's fail-open answer.
…em breakpoints and names across wires - chat/Messages to a Responses-protocol model: image parts became nothing (the input item carried only the flattened text) and response_format was never mapped. Image parts now go as input_image items next to input_text, and response_format becomes text.format (json_schema flattens its schema). - /v1/messages served by an openai-chat upstream (OpenRouter anthropic/*): the vendor's signed reasoning_details were dropped, so thinking blocks reached the client with signature "" and the next tool turn replayed prose only. Signed details now become thinking/redacted_thinking blocks (a signature_delta when streaming), and a native assistant turn replays its signed blocks as reasoning_details. Verified live through OpenRouter: the signature round-trips and the continuation answers 200. - /v1/chat/completions to a Claude model: a system message whose content parts carry cache_control was flattened to a string system. Marked system parts now become the native system blocks (Converse gets its cachePoint); unmarked ones stay a string. - A streamed chat reply that carried tool calls but finished "stop" (a named tool_choice) reported stop while the buffered reply reported tool_calls; both now say tool_calls. - Block-form text (Messages blocks, chat content parts, system blocks) was walked twice, once as the derived flat copy, so the moderator received and retention stored every such request twice and scan hits doubled. Read walks now skip the derived copy; a moderation mask re-derives it; DLP still rewrites both. - The chat message name was parsed and dropped; it now reaches the OpenAI wire. - The response-cache key ignored anthropic-beta, so a beta-gated request could be answered from an entry cached without the beta.
…duplicate A tenant name with ':' and an access key with ':' outside the sha256 id form were rejected through ConfigError::DuplicateName, so the operator read "duplicate tenant (...) name" for a name that duplicates nothing.
…alid YAML as ErrInvalid validateConfig answered 400 for every gateway error, so a gateway outage read as "invalid config"; it now goes through mapError (a gateway 400 is ErrInvalid -> 400, an outage 502), like publishConfig. The mock returned a plain error, so a mock-mode publish of invalid YAML answered 502 where the real client answers 400. Also correct the auth package doc (sessions live in kv) and the case of one own-line comment.
…in; check fallbacks before scanning for replayed reasoning - check_budgets awaited one governance read per configured scope, and with monthly_cost_rollover budgets() read each previous-month counter in turn: a tenant with k budget scopes paid k sequential Redis round trips at admission and again at settlement. The reads, the rollover reads and the charges now run together (one round trip per stage); results are read in scope order, so the first exhausted scope still names the denial and alerts keep their order. - consume_budgets ran after settle_and_bill's join, ledger write included, so settlement took the ledger commit plus the budget round trips. The charge is now a fifth branch of that join (SettleInput carries the key and the resolved override); the three call sites drop their trailing call. - fallback_ahead scanned every message and block for replayed reasoning on every request before the fallback-chain lookup, which is a hash miss for models without fallback_models; the lookup now runs first.
…als and scans, dead fields - views: audio_transcribe's tail is family_response; family_response's engine label is the protocol name; the realtime text bridge's lossy fallback (and the Bytes clone feeding it) cannot run on validated UTF-8; the two relay branches merge. - engines: one mock ok(body) for the eight 200 literals; one Base::post_form for the two multipart uploads; the pump's feed-error branch is the stream fault branch; realtime audio tokens reuse engine::tok; with_status and the Claude tool_calls option take the shorter forms. - protocol: MessageContent's unused Default and ChunkDelta's never-set role. - config: StabilityConf/AlertsConf use a struct-level serde default (seven default_* fns go); AccountConf derives Default for the preset account. - state: the thinking capture keeps completeness beside the block, not in each variant; governance's two GET-or-zero reads share one helper; epoch_secs derives from epoch_millis. - handler: the private Moderation enum was Verdict plus Unavailable, now Option<Verdict>; the fallback condition is a let-chain; the drain stops bulk-loading a claimed batch's items it re-reads one by one before dispatch (one full-batch Postgres read per claimed batch). - dag: reserve_estimate drops a clamp the final min already applies.
Both date from the initial import ("placeholder protocol, byte-level
alignment deferred") and post a {model, raw|payload} envelope to
{base}/v1/audio/other and {base}/v1/passthrough, endpoints no vendor serves;
only the mock transport answered them. No shipped config, control-plane view
or doc example uses them. A config naming either protocol now fails to load
as an unknown protocol instead of failing at the vendor.
… to their facts Wrong: rt_error claimed in-band errors never end a session (an access denial does); write_rt_event listed inbound DLP only; ledger_max_rows and the in-memory cap said oldest rows are pruned (only rolled rows are); billing_record now states the decisions precedence; CallEngine fails over on 401-403 too; the raw media-aware walk is Responses-only; the cache key names what it hashes; the video units fallback is per dialect; the transcription seconds note sits on its block. Restating or duplicated comments go, and the dag node and context docs keep one line per fact. Comment lines -81/+43.
Contradicted by code: the key-guard cache is no longer two seconds (keys invalidate by NOTIFY since #145); an MCP session must be routed sticky by Mcp-Session-Id, not per key (the sample nginx.conf gains /mcp/ with that hash); account failover and the fallback chain also fire on 401-403 (and the chain on an exhausted pool); decisions charges usage.cost unless a tenant overrides the price; image parts and response_format now cross onto the Responses wire. Omitted: the Postgres config store's precedence, video and search models pinning a provider, search's per-query unit, durable records on Postgres, the batch result's user, the /metrics scrape not counting itself, a decisions model's protocol on the openrouter preset, the gw release tarballs, the fleet guide link. The api.md sections Rerank, Search and Decisions move below the OpenAI-compatible subsections they had split from their heading; the crate graph matches Cargo.toml; the release-tag example is a placeholder.
Pure moves (sorted lines identical per file): impl AppState sits under its type and pub TRACE_TARGET heads the const block, app above its private helpers (views); free fns leave the type/impl run and PerEntryTtl follows the pub types (state/lib); Settled and BillingLedger precede the private budget types (admission); the in-memory store's private ledger/content types follow MemoryStore (store); PromptBytes sits on its single consumer and settle_deferred_stream above its private siblings (nodes); the pub(crate) helpers precede the private ones (openai_engine); every pub fn precedes the private walkers (plugins).
The id-less initialize can land on another instance than the requests that carry the id, so the hash keeps every id-carrying request on the instance holding the binding; it does not make the binding fleet-wide.
…ho gaps, finish the cross-wire fixes A fresh full re-read of every file the round touched: - Revert the block-form walk de-duplication from c43a78b: ChatMsg.content (and ChatParams.system) is the only joined copy of adjacent text parts, so skipping it let a blocklisted term split across two parts through the scan and gave the moderator only the split view. Both copies are walked again; a test pins the split term. - swap_to_fallback treated typed: None as chat, so a native /v1/responses request still degraded into a chat model; only a Chat-typed request now degrades across protocols. - The requested-name echo moves to the handler's fallback rewrite, which now also writes response_v2's model field: one site covers every typed family and the video submit; video polls echo the requested name too. - ChatMsg.name (forwarded since c43a78b) is scanned and redacted like every other forwarded client string. - Marked chat system parts become native blocks only on Anthropic wires: a non-Claude Converse model (Grok rejects cachePoint) keeps a plain system. - response_format merges into a client's raw text object instead of replacing it; Responses input items are built only for a turn that carries an image; image blocks inside a tool_result reach the Responses wire as input_image items (the API accepts an array output; verified live). - The quota decision trail reports the recorded amount; consume_budgets is private now that settle_and_bill is its only caller; a renamed test and two corrected docs.
…ed thinking over openai-chat, typed-family fallback, layer table
… in the cache, tool-message images on Responses - Converse turned every cache_control (system, message blocks, tools) into a cachePoint for any model, so a /v1/messages client that marks breakpoints (Claude Code always does) got a Bedrock 400 on Grok, which refuses the field. cachePoint now goes only to the Claude and Nova families; the Claude engine's narrower gate from pass 2 is no longer needed. - The Responses engine wrote the requested name into its reply before CacheStore, so a fallback-served reply was cached under the served model's key with the requested name, and a later direct request for the served model read the wrong name back. The handler's post-cache rewrite is the only one now; the engine keeps the vendor's model. - A chat-surface role: tool message's image parts reach the Responses wire as input_image items, through the same tool_output helper as tool_result.
…message checkpoints Bedrock lists system and messages as Nova's checkpoint fields, so a marked tool sent to Nova as toolConfig.tools[].cachePoint is rejected.
…ly where Bedrock takes it Bedrock gives Nova only the default checkpoint TTL (a 1h ttl from a Messages client is refused), and documents toolResult.status for the Claude and Nova families only. The family the model id names now decides cache points, their TTL and the error status.
…ock refuses them in Bedrock documents ToolResultContentBlock.image for the Nova and Claude families only; for the others the images now follow the toolResult as sibling blocks of the same user message.
…e it Everything from the first image on follows the toolResult in order, so interleaved labels and images keep their sequence.
…ve out A message-level document needs a text block beside it, and the nested ToolResultContentBlock.document carries no family restriction, so documents stay in the toolResult; only the images and the text after the first image move.
Live Bedrock: Llama 4 Maverick and Scout answer a tool result with a nested
image (200) and refuse sibling image blocks in that turn ("Conversation blocks
and tool result blocks cannot be provided in the same turn"), while
openai.gpt-5.6/6, xai.grok and moonshotai.kimi refuse the nested image and
take the siblings. Families outside that list keep the nested layout again.
Live Bedrock takes toolResult.status "error" from Llama 3.3, Llama 4, Pixtral, gpt-oss, gpt-6 and Grok as well as Claude and Nova (200, same input tokens), so the family gate only dropped the error signal the model reads.
us.cohere.embed-v4:0 failed the "cohere." prefix test and was sent Titan's
{inputText} body (424), and Cohere Embed v4 answers {"embeddings":{"float":
[...]}}, which came back as an empty data list while the call was still
billed. Both confirmed live on Bedrock.
…sion Cohere Embed v4 takes output_dimension (256/512/1024/1536); the Cohere branch dropped the client's dimensions and answered 1536-wide vectors as success. Like the Titan branch, the value is now forwarded and a model that refuses it (v3: "extraneous key [output_dimension]") answers the vendor's 400.
…ed model ModelQuotaGate has never denied: with no usable tenant fallback_model it lets the request through on the requested model (live: an embeddings request over quota whose fallback is a chat model is served by the embeddings model). The pass-2 docs said such a request "is denied" and called the quotas caps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
TypeSafe's Jev (
typesafe/jev-1.13) is a System One decision model: it takes astateand typedquestions(choice / noul / score) and returns typed answers with probabilities. OpenRouter serves it on two non-chat endpoints,POST /api/alpha/decisionsandPOST /api/v1/systemone, bills input tokens only, and reports each call's charge inusage.cost. The gateway had no surface for it, so a caller could not reach Jev with its gateway key, and nothing enforced entitlement or budgets or recorded billing for it.Change
decisions, served on two routes that follow the/v1/<surface>convention:POST /v1/decisionsforwards to{endpoint}/alpha/decisions(the Decisions API).POST /v1/systemoneforwards to{endpoint}/v1/systemone, so a TypeSafe SDK pointed at the gateway works unchanged.modelreaches the vendor as sent (state,questions,session_id,user,provider,trace). The vendor reply comes back whole, itsidincluded. That id is OpenRouter's generation id, so one call can be looked up in OpenRouter's generation history. Chat surfaces rewrite ids tochatcmpl-*; this surface does not.usage.input_tokensandusage.output_tokens.decisionscall,cost_microsisusage.cost, which also lands invendor_cost_micros. The model's price list applies only to a reply withoutusage.cost.openrouterprovider preset now servesdecisions.The ledger has no upstream-id column, so the row joins a call by
request_id, and the client sees the vendorid. This PR adds no schema change.Hot-path cost
The new work runs only on the two new routes. Existing routes gain one change in
billing_record, which runs once per billed call: when the vendor reported a cost, a short string compare of the protocol name. The request path itself does no new work.Evidence
Fail-first: the new suite
crates/server/tests/decisions.rswas written first and failed 9/9 before the change, because the config parser rejectedprotocol: decisions. Mutation check: with the vendor-cost charge disabled,a_decision_bills_the_vendor_cost_to_the_userfails withleft: 428400, right: 20, which is the list price instead ofusage.cost.Gates on 1.98.1 (rust-toolchain pin), macOS arm64. The PG/Redis-gated suites ran against throwaway postgres:16 and redis:7 containers. The run created the
gwtest_*databases and 13 tables ingw, and wrote 7 keys to Redis.New tests (all ok):
Whole-repo review round (commits after 8cb32a6)
A full read of the repo (every file the hygiene ledger listed as due, plus the 12 cross-file flows),
/simplify, a whole-repo LOC audit, a docs reconciliation, and then five more fresh re-reads of everything the round touched until one came back clean, with an independent review that ran 11 rounds to "no blocking findings". Every behavior fix has a test that fails without it.Fixes
/v1/batches: a non-object body or JSONL line panicked the request task (serdeIndexMut); now 400./v1endpoint (404, verified live); a tenantmodel_pricesoverride now wins overusage.cost; the requested model name is echoed in passthrough replies (one rewrite in the handler, after the cache, for every typed family and video submit/poll).tool_result, inrole: tool) andresponse_formatreach a Responses model, verified live.reasoning_detailsfrom an openai-chat upstream reach/v1/messagesas signed thinking and are replayed, verified live through OpenRouter.cache_controlparts keep their breakpoints on Anthropic wires, verified live.finish_reasonmatches the buffered reply for a forced tool.nameis forwarded (and scanned).anthropic-betais part of the cache key.toolResult.statusgoes only where Bedrock documents it.Performance. Budget reads, rollover reads and charges run concurrently, the charge inside the settle join. The fallback lookup runs before the replay scan. Interleaved A/B against 8cb32a6 on the same mac, both orders, 4 runs per arm, mock upstream:
Cuts. Duplicated literals, scans and branches, derivable parameters, and dead fields are gone, along with the placeholder
audioandpassthroughprotocols, which no vendor serves (dropped in their own commit). Comments the code contradicts are corrected and multi-line docs trimmed. A layout-only commit moves items (sorted lines identical per file). Prod lines 28,778 → 28,727 with the fixes included; comments 2,294 → 2,251; tests +634 lines.Docs. README and docs reconciled with the code;
deploy/nginx.confgains an/mcp/location hashed onMcp-Session-Id.Gates at f56c87e.
cargo fmt --all -- --checkclean,cargo clippy --workspace --all-targets -- -D warningsclean,cargo test --workspace736 passed / 0 failed / 4 ignored with the Postgres 16 and Redis 7 suites enabled.rust:1.98.1): the same three gates, same result.go test -raceok,make lint0 issues on darwin and linux.Open, not changed. Bedrock access for the test account is currently refused, so the Converse family rules follow the AWS API reference rather than a live run.
Live E2E follow-up (commits after f56c87e)
BASE 8cb32a6 vs HEAD, real vendors through two gateways side by side.
Bedrock (Converse, InvokeModel, embeddings, Llama): ~420 calls. HEAD regressed one case, an image inside a tool result on Llama 4. That case and two other Bedrock bugs (one older than this round) are fixed:
a328411b: onlyopenai.gpt-*,xai.grok-*andmoonshotai.kimi-*move a tool result's images out into sibling blocks; they refuse a nested image. Llama 4 Maverick and Scout take the nested image and refuse the siblings ("Conversation blocks and tool result blocks cannot be provided in the same turn"), so every other family keeps the nested layout.253b433a:toolResult.status: "error"goes to every family again. Llama 3.3/4, Pixtral, gpt-oss, gpt-6 and Grok all return 200 with identical input tokens.9c9ff5db,75532bf1: Cohere on aus./global.inference profile no longer receives Titan's body. v4's{"embeddings":{"float":…}}no longer comes back as an empty list.dimensionsis forwarded asoutput_dimension.statusand a nested tool-result image ("Red").ttl: 1h, Nova toolcachePoint, andcache_controlon Llama, Grok and gpt. The Claude chat-surface system cache now reads (cached=5275).Other vendors: OpenAI, Anthropic, OpenRouter, Gemini, xAI, DeepSeek, Moonshot, MiniMax, DashScope, Qianfan, SiliconFlow, Cohere, Jina and Brave, about 650 calls.
scripts/live-matrix: BASE 177/187, HEAD 177/187, with no case differing between them. The 10 failures are vendor-side: Qianfan model access, SiliconFlow balance, a revoked Brave key, and the AWS rows that were run separately.role: toolimages,json_schemanameforwarded,anthropic-betacache key/v1base, tenant price override, model echob303c78d(docs only): a per-model quota with no usable fallback serves the requested model. ModelQuotaGate has always done this, and the pass-2 docs wrongly said "denied".Gates at 75532bf (b303c78 is docs only): mac and Linux arm64 (rust 1.98.1), fmt and clippy clean, 34 suites, 737 passed / 0 failed / 4 ignored, including PG16 and Redis7. Codex converged on each commit; the last verdict was "No blocking findings" at b303c78.
Fleet and Kling (b303c78)
Two-instance fleet. Two Debian hosts run gateways that share one Redis and one Postgres. The upstream is the built-in mock plus a delayed-status stub. BASE is 8cb32a6.
PEXPIRETIMEis identical on A and B.Kling video. Runs
kling-v2-6directly and through the aliasvid-kling-pub→kling-v2-6, on both BASE and HEAD.succeed, and the clip downloads as 206video/mp4.modelfield, so there is no echo to rewrite.