diff --git a/README.md b/README.md index 5220e74..3f56450 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,7 @@ Each check ends PASS, FAIL, UNPROVEN (not enough evidence) or NOT RUN. A missing ## Features -- **9 skills, 4 commands and 5 read-only subagents**, backed by [47 executable CLI/MCP operations](https://github.com/KittenML/inferencekit/tree/main/src/inferencekit/ops/). +- **9 skills, 4 commands and 5 read-only subagents**, backed by [49 executable CLI/MCP operations](https://github.com/KittenML/inferencekit/tree/main/src/inferencekit/ops/). - **Reusable run tools:** [ikstep](https://github.com/KittenML/inferencekit/blob/main/plugin/skills/optimizing-inference/references/remote-host.md) manages jobs on local or remote Linux hosts, readiness checks and concise status output. - **Compact output:** CLI summaries and bounded MCP responses keep full evidence records on disk, with paging for large lists. - **Speed-of-light ceilings:** measured memory bandwidth (`ceiling bw`) and exact bytes per token read from the checkpoint's safetensors headers (`roofline`), including NVFP4, FP8 and mixture-of-experts models. diff --git a/docs/audio-quality.md b/docs/audio-quality.md index dfe2527..032be9f 100644 --- a/docs/audio-quality.md +++ b/docs/audio-quality.md @@ -7,6 +7,7 @@ The acceptance policy is explicit: | Policy | Required evidence | Meaning of PASS | |---|---|---| | `exact` | Identical PCM, frame counts, rate and channels | Preservation of the named baseline outputs | +| `asr_noninferiority` | Paired WER and/or CER | Noninferiority under the registered ASR metrics | | `task_noninferiority` | Paired WER/CER and predicted naturalness; optional speaker metric | Noninferiority under the registered task/proxy metrics | | `perceptual_and_task_noninferiority` | Task metrics plus blinded, paired listening ratings | Noninferiority under the registered task and listening protocol | @@ -124,3 +125,9 @@ For WER/CER the worse-direction difference is candidate minus baseline; for larg SNR is `10*log10(sum(baseline²)/sum((baseline-candidate)²))`; maxabs is the largest signed-sample difference divided by 32768. Both use unmodified, aligned PCM. Unequal lengths have no SNR/maxabs value. Identical PCM has `snr_status: "identical"` and a null numeric SNR, avoiding a fabricated finite or nonfinite JSON value. The retained `audio_score.json` lists per-pair parity, source IDs, policy, coverage and checks; `imported_scores.json` retains external observations. Report comparison quality can use only evidence bound to its exact baseline/candidate IDs. Proxy acceptance remains labelled as such; missing/changed observations become UNPROVEN, previous FAIL records remain visible, and lifecycle/reliability requirements are not satisfied by quality evidence. + +For an independent exact PCM control, pass the repeat baseline capture with `--baseline-repeat` to `quality audio`. An unstable or incomplete A/A comparison leaves the candidate’s exactness unproven. + +## Import native PCM + +Run `inferencekit import audio --help` for the saved WAV or PCM import. Match the synthesis settings and seed base across arms, then pass the evidence IDs to `quality audio`. The import retains PCM and declared provenance for quality and carries no timing measurements. Use a policy registered before the declared capture start for an acceptance verdict. diff --git a/docs/audio-scorers.md b/docs/audio-scorers.md index 66f4cf3..e8c2af4 100644 --- a/docs/audio-scorers.md +++ b/docs/audio-scorers.md @@ -85,6 +85,8 @@ inferencekit quality audio --baseline BASELINE_ID --candidate CANDIDATE_ID \ inferencekit report COMPARISON_ID QUALITY_ID --format md ``` +Set UTMOSv2 `min_tail_seconds` to merge a shorter trailing window into its preceding window. An out-of-range score marks its utterance incomplete for naturalness and retains the other observations. + The score file binds each observation to its source PCM, text, seed, fixture and round. Models are reused within a scoring invocation. Missing or failed observations cannot become a quality PASS. Keep the observations with the evidence; they contain transcripts and are not automatically safe to publish. ## Optional checks diff --git a/docs/install.md b/docs/install.md index 9130a88..9e7ec5b 100644 --- a/docs/install.md +++ b/docs/install.md @@ -44,6 +44,7 @@ python3 ~/.local/share/inferencekit/plugin/opencode/install.py ## The CLI or the MCP server alone - **CLI:** `uvx --from 'inferencekit-cli==0.1.1' inferencekit doctor`, or `uv tool install 'inferencekit-cli==0.1.1'`. +- **Hosts without a build environment:** install the published wheel in an existing virtual environment with `python -m pip install --only-binary=:all: inferencekit-cli==0.1.1`. For unreleased builds, use the wheel from the matching source revision. - **MCP server, for any MCP client:** `uvx --from 'inferencekit-cli[mcp]==0.1.1' inferencekit mcp serve` (stdio). Every CLI command is also a tool with the same arguments. - **Unreleased source changes:** the plugin's MCP configuration uses the published CLI version above, not the current checkout. New operations require a release or an MCP server installed from the matching source; check the advertised tool list before relying on them. - **If the MCP server does not start** (no `uvx`, no network, a failed install): install the CLI pinned to the plugin's version and let the agent call it from the shell. The skills say how. diff --git a/docs/media-benchmarks.md b/docs/media-benchmarks.md index 662cc0c..0f4eb0b 100644 --- a/docs/media-benchmarks.md +++ b/docs/media-benchmarks.md @@ -58,7 +58,7 @@ Repeat audio events as needed. `sample_count` counts frames per channel. The fin Failed request rows have a null `sample_count` and no valid speed metrics. Their `received_pcm_frames` and `chunks` preserve how much decoded PCM arrived and when, including partial delivery before a timeout. These diagnostics do not establish a complete or usable speech output. -The client reports first PCM arrival, first nonzero signal arrival, simulated nonzero playback onset, complete generation time, real-time factor (wall time / generated audio duration), gaps and underruns with zero extra prebuffer. `first_audio_p95_ms` in round comparisons uses simulated nonzero onset, accounting for leading zero PCM and stalls. A nonzero sample is not proof of audible speech. Default timing limits are p95 onset <=2 seconds, maximum RTF <=1 and zero underruns; use the SLO flags to register different limits before measuring. +The client reports first PCM arrival (`first_pcm_p95_ms` for round comparisons), first nonzero signal arrival, simulated nonzero playback onset, complete generation time, real-time factor (wall time / generated audio duration), gaps and underruns with zero extra prebuffer. `first_audio_p95_ms` in round comparisons uses simulated nonzero onset, accounting for leading zero PCM and stalls. A nonzero sample is not proof of audible speech. Default timing limits are p95 onset <=2 seconds, maximum RTF <=1 and zero underruns; use the SLO flags to register different limits before measuring. `inferencekit review build RUN_DIR` displays saved PNG images and speech outputs. Raw PCM is wrapped in a WAV under `review/media/` only after its hash, byte count, encoding, sample rate and channel count match `outputs/LABEL/index.json`; the source PCM stays unchanged. The review data lists each playback copy and its source hash. For speech captions, pass `--prompts FIXTURES.jsonl`: unbucketed fixture texts fill every observed round of the matching id. Attach the original evidence records to display per-request timing and failures. The page keeps simulated nonzero onset distinct from first PCM arrival and semantic quality remains UNPROVEN until separate quality evidence is supplied. diff --git a/plugin/skills/benchmarking/SKILL.md b/plugin/skills/benchmarking/SKILL.md index b9c16b7..a6f92e3 100644 --- a/plugin/skills/benchmarking/SKILL.md +++ b/plugin/skills/benchmarking/SKILL.md @@ -13,6 +13,8 @@ Choose the workload before the metric: complete-request latency, sustained throu `bench rounds --mode audio|image|clip` runs fresh-process media comparisons; CPU manifests use `hardware.backend: cpu` for a CPU lock and process-group telemetry. Compare the final evidence ID for each engine in the run plan; [media contracts and round scheduling](https://github.com/KittenML/inferencekit/blob/main/docs/media-benchmarks.md). +For native phone speech, use [on-device speech](../inference-triage/references/phones.md) to import first audio, RTF, underruns and throughput under concurrent open-loop load. + Speed numbers that survive review: a timer outside the engine, one precision for every engine, settings proven in effect, warm engines, fresh-process rounds in a balanced order, confidence intervals, a speed-of-light ceiling and anti-cheat gates. [optimizing-inference](../optimizing-inference/SKILL.md) calls this skill. **Use** for latency and tok/s per input bucket, capacity (`N*` within an SLO, goodput, $/1M tokens), comparisons of engines, configs or builds, and any reported number. A change-loop step needs only a screen; the full protocol is for milestones (fixed rounds: [references/statistics.md](references/statistics.md)) and anything published ([optimizing-kernels](../optimizing-kernels/SKILL.md), "Checks"). **Not** for *why* it is slow ([profiling](../profiling/SKILL.md)), nor while builds' outputs differ unchecked ([proving-correctness](../proving-correctness/SKILL.md) first). diff --git a/plugin/skills/inference-triage/SKILL.md b/plugin/skills/inference-triage/SKILL.md index 6dca0e8..1c6c850 100644 --- a/plugin/skills/inference-triage/SKILL.md +++ b/plugin/skills/inference-triage/SKILL.md @@ -50,6 +50,7 @@ Explicit session authorization persists. An approved finite-deadline run may spe NVIDIA is fully supported, and the only hardware inferencekit has been run on so far. On other hardware, the skills guide the same flow, and the CLI does less: [references/hardware.md](references/hardware.md) has the table. - **Hardware-neutral, not yet run off NVIDIA:** `bench rounds`, `bench serve`, `gates run` and quality (they talk HTTP to any OpenAI-compatible server), `roofline` with a numeric `--gpu-bw`, and the intake, manifest, ledger and report tools. Treat a first run there as a test. - **NOT RUN off NVIDIA:** `ceiling bw` and nsys/ncu captures. `doctor` records selected AMD DRM/rocminfo and Apple sysctl/system_profiler facts with a vendor profiling route; unknown details remain unknown. On AMD, `rocprof capture` and `rocprof analyze` create trace evidence with rocprofv3; device/version support must be checked. Use current official docs for unfamiliar tools and a vendor or STREAM-style bandwidth test. Captures made outside the CLI are vendor evidence files. +- **Phone workloads:** [on-device speech](references/phones.md). Capture request timings on the device. - **Platform guides:** [AMD ROCm](references/amd-rocm.md), [x86 CPU](references/cpu.md), [Apple Silicon](references/apple-silicon.md), [Intel GPU and TPU](references/other-accelerators.md). ## Preflight checklist diff --git a/plugin/skills/inference-triage/references/phones.md b/plugin/skills/inference-triage/references/phones.md new file mode 100644 index 0000000..5336aa3 --- /dev/null +++ b/plugin/skills/inference-triage/references/phones.md @@ -0,0 +1,27 @@ +# On-device speech and concurrency + +Identify the phone model, SoC, OS build, runtime and actual CPU, GPU or NPU placement. For Android gather `adb shell getprop`, process affinity and `adb shell dumpsys thermalservice` before and after each cell. Read VmHWM or PSS separately with the process and measurement method identified. For iOS use the selected device's OS and runtime facts plus Instruments or native timers when available. Report inaccessible counters as NOT RUN. + +Run the client and engine on the phone. With Android device labs, push a native arm64 test or start an app with ADB, collect its artifacts, and use the same phone for the baseline and candidate. A forwarded HTTP endpoint adds transport time. Label native measurements by where the timer starts and ends, and verify settings, operator placement and process isolation from runtime logs or vendor tools. + +## Capacity under speech SLOs + +- Pin the model, precision, voice, seed policy, text and sentence boundaries. Start with sentence streaming for interactive output. Benchmark whole-utterance delivery on the same text and report its first audio and complete latency. Native smaller chunks can be a separate workload. +- Set the RTF limit before testing. Register a first-audio limit if the user requires one. Schedule numbered waves of distinct fixtures at fixed times regardless of previous completion. At each level dispatch the stated number of requests together. Use the same ordered fixture set and period across configurations. Count time from scheduled release, including queueing. Warm up each fresh process outside timing. +- In each measured round all stream slots must meet the registered p95 limits. Simulate playback starting at first PCM with no extra buffer and require zero underruns. Track first PCM separately from the first nonzero playback sample. A response running beyond its wave period fails that level. +- Start at one stream and measure consecutive levels, including the first failing level above a contiguous passing prefix. A screen can identify a candidate; repeat with balanced fresh-process rounds, enough requests per stream to characterize the tail and a sustained thermal check. If the top measured level passes, report a lower bound. Report the wave period and output audio seconds per second of scheduled wall time. +- Establish completion and speech quality on saved output from the exact configurations under test. Native timing import does not attest to fixture contents, correctness, actual audio hardware playback, or the capture implementation. + +Use `hardware.device`, `workload.client: audio_capacity` and `goals.lead: capacity` in the intake. Set `goals.slo.rtf_p95` and `max_underruns: 0`. Optional `first_audio_p95_ms` sets a separate gate. + +Register the plan before timing. Fields are `levels` (consecutive from 1), `rounds`, `waves`, `period_ms`, `sample_rate`, `mode` (`stream` or `full`), `chunks_per_request` (1 for full), `fixture_ids` and `fixture_sha256` (digest of the fixture file). List one distinct ID per wave and stream at the largest level. For each run use fixture index `wave * concurrency + stream`. + +Export one JSONL row per request with zero-based `round`, `wave`, `stream`, and fields `concurrency`, `id`, `complete_ms`, and `chunks`. Each chunk is an array of arrival milliseconds since scheduled release, PCM frame count and leading silent frame count. Use null completion for failed requests. Retain a separate record of the process launches, resolved settings, thermal state and capture code. + +```bash +inferencekit import audio-capacity --timings requests.jsonl --plan plan.json --fixtures fixtures.tsv --label candidate \ + --model MODEL --model-revision REV --engine-version VER --hardware-name DEVICE \ + --first-audio-as-metric --slo-rtf-p95 1 +``` + +For a latency-gated run, use `--slo-first-audio-p95-ms` in place of `--first-audio-as-metric`. diff --git a/plugin/skills/optimizing-inference/SKILL.md b/plugin/skills/optimizing-inference/SKILL.md index 583af62..69cd5fa 100644 --- a/plugin/skills/optimizing-inference/SKILL.md +++ b/plugin/skills/optimizing-inference/SKILL.md @@ -49,7 +49,7 @@ Use the derived run id; `--run-dir` pins the intake and starts the clock. SSH ru Validate [intake](references/intake.md); pin revision, component precisions, processing, dataset, shapes and output work. Derive workload details from examples/logs ([workload.md](references/workload.md)); record assumptions. -Choose response latency, sustained throughput, peak memory or affordable hardware under a latency/quality limit ([affordability.md](references/affordability.md)). Token metrics and GSM8K belong to text workloads. Speech needs content, completion and speaker checks; images need semantic/visual checks; video needs temporal checks. Other architectures need a registered task-specific adapter and protocol; unsupported CLI modalities remain UNPROVEN rather than being relabelled as text. +Choose response latency, sustained throughput, peak memory or affordable hardware under a latency/quality limit ([affordability.md](references/affordability.md)). Apply a requested [compute resource policy](references/resource-policy.md). Token metrics and GSM8K belong to text workloads. Speech needs content, completion and speaker checks; images need semantic/visual checks; video needs temporal checks. Other architectures need a registered task-specific adapter and protocol; unsupported CLI modalities remain UNPROVEN rather than being relabelled as text. Use [researching-optimizations](../researching-optimizations/SKILL.md) now, from the pinned model, shapes, hardware and runtime. It returns a short experiment queue, including existing engine paths to try before custom code. @@ -57,7 +57,7 @@ Use [researching-optimizations](../researching-optimizations/SKILL.md) now, from Run `inferencekit doctor`. Check competing work, memory, swap, thermals and profiler permissions before occupying the device. Stop only the run's own jobs; do not kill unrelated services to obtain a clean measurement. -On NVIDIA, measured `ceiling bw` and `roofline` inform supported text models. Elsewhere use the [hardware route](../inference-triage/references/hardware.md) and vendor measurements; a PyTorch CUDA namespace does not establish NVIDIA-tool support. For non-token workloads derive bytes/FLOPs per request or step with assumptions. Unsupported ceilings stay NOT RUN. +On NVIDIA, measured `ceiling bw` and `roofline` inform supported text models. Elsewhere use the [hardware route](../inference-triage/references/hardware.md) and vendor measurements; a PyTorch CUDA namespace does not establish NVIDIA-tool support. For [phones](../inference-triage/references/phones.md), capture the SoC, OS, runtime placement and thermal state on-device and use its supported timers. For non-token workloads derive bytes/FLOPs per request or step with assumptions. Unsupported ceilings stay NOT RUN. Estimate critical-path share, attainable efficiency, exact byte savings and permitted algorithmic savings ([ceilings.md](references/ceilings.md), [stop-point.md](references/stop-point.md)). Recompute after changing precision or work. An inferred ceiling orders experiments; it cannot establish a speedup. @@ -71,7 +71,7 @@ For text, a fixed baseline with route build may skip the survey and use the [eng ## 4. Screen and measure -Use [harness-commands.md](references/harness-commands.md) for text, [affordability.md](references/affordability.md) for image/audio and [video.md](references/video.md) for video. Screen representative cases before full rounds. Text screens include `short` and `long` lengths. Promote secondary wins the user values without hiding the lead metric. +For phone speech concurrency use the [on-device speech capacity protocol](../inference-triage/references/phones.md). Use [harness-commands.md](references/harness-commands.md) for text, [affordability.md](references/affordability.md) for image/audio and [video.md](references/video.md) for video. Screen representative cases before full rounds. Text screens include `short` and `long` lengths. Promote secondary wins the user values without hiding the lead metric. Final comparisons use balanced fresh-process rounds, excluded warmup, registered stopping and confidence intervals. Include complete output work and transfers/synchronization/decoding. [Capacity](references/capacity.md) needs its own tuning and measured concurrency. diff --git a/plugin/skills/optimizing-inference/references/resource-policy.md b/plugin/skills/optimizing-inference/references/resource-policy.md new file mode 100644 index 0000000..46cc2e2 --- /dev/null +++ b/plugin/skills/optimizing-inference/references/resource-policy.md @@ -0,0 +1,24 @@ +# Compute resource policy + +Set `goals.resource_policy` when the user specifies how the inference workload may use the device. Ask which outcome matters when throughput, latency, energy and interactive responsiveness compete. Keep the registered workload, quality limits, resource authorization and target hardware in either mode. + +## Maximum performance + +Set `maximum_performance` to maximize the stated metric under the stated limits. Inventory authorized CPU cores and accelerators. Verify which runtimes can place the pinned model or its stages on each unit. Benchmark the strongest tuned single-unit placement in addition to promising combinations. + +Try parallel requests on separate units, overlapping independent stages across requests, and supported in-process batching or pipelining. Measure useful stage overlap and wall time including transfers, synchronization, queueing and output. Reuse read-only weights or share tensor buffers where the runtime supports it and verify buffer placement, copies and lifetime. Measure sustained bandwidth, thermals and memory pressure where possible. Select the fastest configuration that meets the registered limits. Record actual placement, fallback and any units that could not run the model. + +## Shared-device efficiency + +Set `shared_device_efficiency` to meet the stated workload targets while leaving capacity for other applications. Agree on what should remain responsive, such as a game, a call, media playback or a desktop application. Register a representative foreground workload, its performance with inference idle and any latency or energy limits the user specifies. + +Screen fewer inference workers and CPU threads, bounded queues, capped concurrency, OS scheduling hints, or a supported accelerator that leaves resources available to the foreground work. Compare on the same device with the foreground workload present in each measured run. Record inference latency/throughput and foreground latency or missed deadlines, plus CPU usage, memory, thermal state and energy when those can be measured. Prefer the configuration that consumes fewer of the constrained resources while meeting both workloads' targets. If foreground contention was never measured, report device responsiveness as UNPROVEN. + +## Platform implications + +The objectives apply to phones, laptops and desktops. Actual placement depends on the model, runtime, memory topology and device access. A shared memory pool can still have competing bandwidth, copies or synchronization. Discrete GPUs introduce a separate device memory budget and transfers. Test the real path on the deployed hardware. + +- Qualcomm Android can expose NPU or GPU backends through an appropriate QNN build. Verify graph placement and fallback. [ONNX Runtime QNN documentation](https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html) +- Android scheduling and thermal hints can help bound concurrent application impact. [Android Dynamic Performance Framework](https://developer.android.com/games/optimize/adpf) +- Apple Core ML allows different compute-unit policies and provides a compute plan. [Apple Core ML performance analysis](https://developer.apple.com/documentation/coreml/analyzing-a-core-ml-model-s-performance-in-xcode) +- On discrete NVIDIA systems, measure transfers and supported copy/compute overlap. [CUDA memory optimization guide](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#memory-optimizations) diff --git a/plugin/skills/reporting-perf-claims/references/claims-lint.md b/plugin/skills/reporting-perf-claims/references/claims-lint.md index dd4e9e6..60c5ab4 100644 --- a/plugin/skills/reporting-perf-claims/references/claims-lint.md +++ b/plugin/skills/reporting-perf-claims/references/claims-lint.md @@ -1,6 +1,6 @@ # Claims lint: line by line -Run this over the final text of any report, PR, README table or agent summary: what to check, a failing example, the fix (placeholder values). A line that cannot be fixed downgrades the claim per the verdict precedence (say which line failed). Early-engine numbers quoted in these skills are engine-timed (rule 18) and carry that label (line 6) wherever repeated. +This checklist is applied manually. Run it over the final text of any report, PR, README table or agent summary: what to check, a failing example, the fix (placeholder values). A line that cannot be fixed downgrades the claim per the verdict precedence (say which line failed). Early-engine numbers quoted in these skills are engine-timed (rule 18) and carry that label (line 6) wherever repeated. | # | Check | Fails | Passes | |---|---|---|---| diff --git a/schemas/evidence.v2.json b/schemas/evidence.v2.json index 6d3be1a..f89e04e 100644 --- a/schemas/evidence.v2.json +++ b/schemas/evidence.v2.json @@ -51,6 +51,7 @@ "clip_steps", "image_benchmark", "audio_benchmark", + "audio_capacity", "quality_audio", "rocprof_capture", "rocprof_summary" @@ -78,6 +79,15 @@ "workload" ], "properties": { + "device": { + "type": "object", + "additionalProperties": false, + "required": ["class", "name"], + "properties": { + "class": {"enum": ["mobile"]}, + "name": {"type": "string", "minLength": 1} + } + }, "cpu": { "type": "object", "additionalProperties": false, "properties": { diff --git a/schemas/intake.v1.json b/schemas/intake.v1.json index 0909155..0bb89a6 100644 --- a/schemas/intake.v1.json +++ b/schemas/intake.v1.json @@ -35,13 +35,14 @@ "modality": {"enum": ["llm", "tts", "stt", "video", "image"]}, "hardware": { "type": "object", - "required": ["gpu"], + "anyOf": [{"required": ["gpu"]}, {"required": ["device"]}], "additionalProperties": false, "properties": { "gpu": {"type": "string", "minLength": 1, "description": "GPU model, e.g. RTX 5090."}, + "device": {"type": "string", "minLength": 1, "description": "On-device hardware label including the phone or SoC."}, "count": {"type": "integer", "minimum": 1, "default": 1}, "price_usd_h": {"type": "number", "exclusiveMinimum": 0, - "description": "Hourly price of the whole host (all `count` GPUs) in USD."}, + "description": "Hourly price of the whole host or phone lease in USD."}, "access": { "type": "object", "required": ["kind"], @@ -63,6 +64,10 @@ "additionalProperties": false, "properties": { "lead": {"enum": ["single_user", "capacity", "affordability"], "default": "single_user"}, + "resource_policy": { + "enum": ["maximum_performance", "shared_device_efficiency"], + "description": "Optional placement objective: maximize the stated performance metric across supported units, or meet it with headroom for other workloads." + }, "affordability": { "type": "object", "additionalProperties": false, "required": ["memory_gb_max", "memory_kind"], @@ -83,6 +88,7 @@ "decode_tok_s": {"type": "number", "exclusiveMinimum": 0}, "ttft_ms": {"type": "number", "exclusiveMinimum": 0}, "n_star": {"type": "integer", "minimum": 1}, + "audio_s_per_s": {"type": "number", "exclusiveMinimum": 0}, "decode_ceiling_fraction": {"type": "number", "exclusiveMinimum": 0, "maximum": 1, "description": "Decode tok/s as a fraction of the measured-bandwidth roofline ceiling."}, "ttfa_ms": {"type": "number", "exclusiveMinimum": 0, "description": "TTS: time to first audio."}, @@ -104,6 +110,9 @@ "properties": { "ttft_p95_ms": {"type": "number", "exclusiveMinimum": 0}, "tpot_p95_ms": {"type": "number", "exclusiveMinimum": 0}, + "first_audio_p95_ms": {"type": "number", "exclusiveMinimum": 0}, + "rtf_p95": {"type": "number", "exclusiveMinimum": 0}, + "max_underruns": {"type": "integer", "minimum": 0}, "max_fail": {"type": "integer", "minimum": 0, "default": 0} } }, @@ -141,13 +150,13 @@ "rounds": {"type": "integer", "minimum": 3, "default": 5}, "concurrency": {"type": "array", "minItems": 1, "items": {"type": "integer", "minimum": 1}}, "prompt_set": {"type": "string", "description": "LLM prompt set (default dev)."}, - "client": {"enum": ["bench_serve", "bench_audio", "bench_image", "bench_clip", "external"], "description": "Configured timing client; selecting one declares the route, not successful measurement or quality proof."}, + "client": {"enum": ["bench_serve", "bench_audio", "audio_capacity", "bench_image", "bench_clip", "external"], "description": "Configured timing client; selecting one declares the route, not successful measurement or quality proof."}, "tts": { "type": "object", "additionalProperties": false, "properties": { "voice": {"type": "string", "minLength": 1}, "sample_rate": {"type": "integer", "minimum": 1}, - "lead_metric": {"enum": ["rtf", "first_audio_ms", "complete_audio_ms"]}, + "lead_metric": {"enum": ["rtf", "first_audio_ms", "complete_audio_ms", "audio_s_per_s"]}, "text_length_buckets": { "type": "array", "minItems": 1, "items": {"type": "object", "additionalProperties": false, diff --git a/src/inferencekit/evidence.py b/src/inferencekit/evidence.py index cacb6cb..f96c74a 100644 --- a/src/inferencekit/evidence.py +++ b/src/inferencekit/evidence.py @@ -53,7 +53,7 @@ KINDS += ("clip_benchmark", "clip_gates", "clip_quality", "clip_steps") # Client-timed images and speech. Structural validity never establishes semantic quality. KINDS += ("image_benchmark", "audio_benchmark", "rocprof_capture", "rocprof_summary") -KINDS += ("quality_audio",) +KINDS += ("quality_audio", "audio_capacity") # identity.workload fields: the v1 four are always present (null when unknown); the v2 serving fields # are written only when the operation knows them. WORKLOAD_V1 = ("prompt_len", "gen_len", "batch", "seed") @@ -91,7 +91,7 @@ def metric(name: str, value: float | int | None, unit: str, stat: str, n: int | def identity(gpu: dict | None = None, engine: dict | None = None, model: dict | None = None, - workload: dict | None = None, cpu: dict | None = None) -> dict: + workload: dict | None = None, cpu: dict | None = None, device: dict | None = None) -> dict: """The four things a number depends on. Unknown fields are null, never guessed.""" g = {"name": None, "count": None, "driver": None, "cuda": None, "clocks": None} e = {"name": None, "version": None, "commit": None, "flags": {}} @@ -108,6 +108,8 @@ def identity(gpu: dict | None = None, engine: dict | None = None, model: dict | if cpu is not None: result["cpu"] = {k: cpu.get(k) for k in ("model", "architecture", "logical_cores", "physical_cores", "allowed_cpus", "topology_sha256")} + if device is not None: + result["device"] = {k: device.get(k) for k in ("class", "name")} return result diff --git a/src/inferencekit/ops/__init__.py b/src/inferencekit/ops/__init__.py index 8296420..fb1a07c 100644 --- a/src/inferencekit/ops/__init__.py +++ b/src/inferencekit/ops/__init__.py @@ -29,6 +29,8 @@ from inferencekit.ops import gates_clip # noqa: F401,E402 (video: gates clip; needs bench_clip) from inferencekit.ops import bench_image # noqa: F401,E402 from inferencekit.ops import bench_audio # noqa: F401,E402 +from inferencekit.ops import audio_capacity # noqa: F401,E402 +from inferencekit.ops import import_audio # noqa: F401,E402 from inferencekit.ops import quality_audio # noqa: F401,E402 from inferencekit.ops import audio_scorers # noqa: F401,E402 from inferencekit.ops import cpu # noqa: F401,E402 diff --git a/src/inferencekit/ops/audio_capacity.py b/src/inferencekit/ops/audio_capacity.py new file mode 100644 index 0000000..1cebfa9 --- /dev/null +++ b/src/inferencekit/ops/audio_capacity.py @@ -0,0 +1,226 @@ +"""Validate an open-loop speech capture and derive its capacity under the registered SLOs.""" + +from collections import defaultdict +import json +from pathlib import Path +import re + +from inferencekit import evidence as ev, stats +from inferencekit.canonical import canonical_json, file_sha256, sha256_hex +from inferencekit.registry import Arg, OpError, operation +from inferencekit.serve.audio_client import playback_metrics +from inferencekit.serve.metrics import pct +from inferencekit.serve.round_evidence import finite + +KIND = "audio_capacity" +PLAN_KEYS = {"levels", "rounds", "waves", "period_ms", "sample_rate", "mode", + "fixture_ids", "chunks_per_request", "fixture_sha256"} +ROW_KEYS = {"round", "concurrency", "wave", "stream", "id", "complete_ms", "chunks"} +ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}") +SHA = re.compile(r"[0-9a-f]{64}") + +ARGS = ( + Arg("timings", "string", "JSONL timing rows produced by the on-device client.", required=True, sensitive=True), + Arg("plan", "string", "Preregistered load plan JSON.", required=True, sensitive=True), + Arg("fixtures", "string", "Optional fixture file whose SHA-256 must match the plan.", sensitive=True), + Arg("label", "string", "Runtime configuration label.", required=True), + Arg("model", "string", "Pinned speech model identifier.", required=True), + Arg("model_revision", "string", "Checkpoint revision.", required=True), + Arg("engine_version", "string", "Runtime build or revision.", required=True), + Arg("hardware_name", "string", "Measured device model, SoC and OS.", required=True), + Arg("model_dtype", "string", "Weights and compute precision."), + Arg("slo_first_audio_p95_ms", "number", "Simulated first nonzero playback onset p95 per stream.", default=400.0), + Arg("first_audio_as_metric", "boolean", "Measure first audio without applying a first-audio SLO.", default=False), + Arg("slo_rtf_p95", "number", "Complete-response real-time factor p95 per stream.", default=1.0), +) + + +def _integer(value, low, high): + return type(value) is int and low <= value <= high + + +def read_plan(path): + try: + plan = json.loads(Path(path).read_text(encoding="utf8")) + except (OSError, ValueError): + raise OpError("audio capacity plan is not readable JSON") from None + if not isinstance(plan, dict) or set(plan) - PLAN_KEYS or PLAN_KEYS - {"fixture_sha256"} - set(plan): + raise OpError("invalid audio capacity plan fields") + levels = plan["levels"] + if (not isinstance(levels, list) or not levels or len(levels) > 256 + or any(type(level) is not int for level in levels) or levels != list(range(1, len(levels) + 1))): + raise OpError("audio capacity levels must be consecutive integers starting at 1") + if (not _integer(plan["waves"], 1, 10000) or not _integer(plan["rounds"], 1, 1000) + or not _integer(plan["sample_rate"], 8000, 192000) or not _integer(plan["period_ms"], 1, 3600000) + or not _integer(plan["chunks_per_request"], 1, 1000) or plan["mode"] not in ("stream", "full") + or (plan["mode"] == "full" and plan["chunks_per_request"] != 1) + or len(levels) * (len(levels) + 1) // 2 * plan["waves"] * plan["rounds"] > 1000000): + raise OpError("invalid audio capacity plan bounds or delivery mode") + ids = plan["fixture_ids"] + if (not isinstance(ids, list) or len(ids) != plan["waves"] * levels[-1] + or any(not isinstance(x, str) or not ID.fullmatch(x) for x in ids) or len(set(ids)) != len(ids) or ev.redact(ids) != ids): + raise OpError("one distinct fixture ID is required per wave and stream at the largest level") + fixture_sha = plan.get("fixture_sha256") + if fixture_sha is not None and (not isinstance(fixture_sha, str) or not SHA.fullmatch(fixture_sha)): + raise OpError("fixture_sha256 must be a lowercase SHA-256 digest") + if ev.redact(plan) != plan: + raise OpError("audio capacity plan fields must be safe to retain") + return plan + + +def request_metrics(row, plan): + complete, chunks = row["complete_ms"], row["chunks"] + if not isinstance(chunks, list): + raise ValueError + if (complete is not None and (not finite(complete) or complete <= 0) + or len(chunks) > plan["chunks_per_request"]): + raise ValueError + latest = complete if complete is not None else float("inf") + rate = plan["sample_rate"] + previous, end, onset, frames = 0.0, 0.0, None, 0 + playback = [] + for value in chunks: + if not isinstance(value, list) or len(value) != 3: + raise ValueError + arrival, count, silent = value + if (not finite(arrival) or not previous <= arrival <= latest + or not _integer(count, 1, rate * 3600) or not _integer(silent, 0, count)): + raise ValueError + start = max(end, arrival) + if onset is None and silent < count: + onset = start + silent * 1000 / rate + end = start + count * 1000 / rate + frames += count + previous = arrival + playback.append((round(arrival * 1e6), count)) + if complete is None or len(chunks) != plan["chunks_per_request"] or onset is None: + return None + seconds = frames / rate + return {"first_pcm_ms": chunks[0][0], "first_audio_ms": onset, "rtf": complete / 1000 / seconds, + "audio_s": seconds, "complete_ms": complete, **playback_metrics(playback, rate)} + + +def read_timings(path, plan): + cells = defaultdict(dict) + try: + with Path(path).open(encoding="utf8") as handle: + for index, line in enumerate(handle, 1): + row = json.loads(line) + if not isinstance(row, dict) or set(row) != ROW_KEYS: + raise ValueError + level, round_index, wave, stream = (row[key] for key in ("concurrency", "round", "wave", "stream")) + if (not _integer(level, 1, len(plan["levels"])) or not _integer(round_index, 0, plan["rounds"] - 1) + or not _integer(wave, 0, plan["waves"] - 1) or not _integer(stream, 0, level - 1) + or row["id"] != plan["fixture_ids"][wave * level + stream]): + raise ValueError + cell = cells[level, round_index] + if (wave, stream) in cell: + raise ValueError + cell[wave, stream] = (row, request_metrics(row, plan)) + except (OSError, ValueError, OverflowError): + raise OpError(f"invalid or unreadable audio timing row {locals().get('index', 0)}") from None + return cells + + +def summarize_cell(cell, level, plan, first_audio_ms, rtf): + count = plan["waves"] * level + valid = [(wave, stream, stats) for (wave, stream), (_, stats) in sorted(cell.items()) if stats is not None] + per_stream = [[] for _ in range(level)] + for _, stream, stats in valid: + per_stream[stream].append(stats) + streams = [{"stream": i, "requests": len(rows), + "first_pcm_p95_ms": pct([r["first_pcm_ms"] for r in rows], 95), + "first_audio_p95_ms": pct([r["first_audio_ms"] for r in rows], 95), + "rtf_p95": pct([r["rtf"] for r in rows], 95), + "underruns": sum(r["underruns"] for r in rows)} for i, rows in enumerate(per_stream)] + und = sum(r["underruns"] for r in streams) + period = plan["period_ms"] + overruns = sum(r["complete_ms"] > period for _, _, r in valid) + failed = len(cell) - len(valid) + slo_missed = any(r["requests"] and (r["rtf_p95"] > rtf or ( + first_audio_ms is not None and r["first_audio_p95_ms"] > first_audio_ms)) for r in streams) + verdict = "UNPROVEN" if len(cell) < count else ("FAIL" if failed or und or overruns or slo_missed else "PASS") + finish = max((wave * period + r["complete_ms"] for wave, _, r in valid), default=0) + observation_ms = max(plan["waves"] * period, finish) + return {"requests": len(cell), "expected_requests": count, "failed_requests": failed, "underruns": und, + "wave_overruns": overruns, "streams": streams, "verdict": verdict, + "audio_s_per_s": 1000 * sum(r["audio_s"] for _, _, r in valid) / observation_ms if valid else None} + + +@operation("import_audio_capacity", ("import", "audio-capacity"), + "Derive speech capacity and audio throughput from a client-timed open-loop capture.", ARGS) +def import_audio_capacity(call, timings, plan, fixtures, label, model, model_revision, engine_version, hardware_name, + model_dtype, slo_first_audio_p95_ms, slo_rtf_p95, first_audio_as_metric=False): + slo = {"rtf_p95": slo_rtf_p95, "underruns": 0} + if not first_audio_as_metric: + slo["first_audio_p95_ms"] = slo_first_audio_p95_ms + if any(not finite(v) or v <= 0 for v in (slo_first_audio_p95_ms, slo_rtf_p95)): + raise OpError("audio capacity SLOs must be finite and positive") + for value in (label, model, model_revision, engine_version, hardware_name, model_dtype): + if value is not None and (not value.strip() or ev.redact(value) != value): + raise OpError("audio capacity requires nonempty, public identity labels") + plan_doc = read_plan(plan) + if fixtures: + try: + fixture_digest, _ = file_sha256(Path(fixtures)) + except OSError: + raise OpError("audio capacity fixture file is not readable") from None + if fixture_digest != plan_doc.get("fixture_sha256"): + raise OpError("audio capacity fixture file differs from the registered plan") + cells = read_timings(timings, plan_doc) + levels, highest, metrics = [], None, [] + for level in plan_doc["levels"]: + rounds = [summarize_cell(cells[level, index], level, plan_doc, slo.get("first_audio_p95_ms"), slo_rtf_p95) + for index in range(plan_doc["rounds"])] + statuses = {row["verdict"] for row in rounds} + verdict = "UNPROVEN" if "UNPROVEN" in statuses else ("FAIL" if "FAIL" in statuses else "PASS") + if verdict == "PASS" and (level == 1 or highest == level - 1): + highest = level + levels.append({"concurrency": level, "verdict": verdict, "rounds": rounds}) + throughput = [row["audio_s_per_s"] for row in rounds if row["verdict"] == "PASS"] + if verdict == "PASS": + ci = stats.bootstrap_ci(throughput, ps=(50,))[50] if len(throughput) >= 3 else None + metrics.append(ev.metric("audio_s_per_s", pct(throughput, 50), "audio s/s", "median", + n=len(throughput), ci95=ci, subject=f"concurrency {level}")) + boundary = next((row["verdict"] for row in levels if row["concurrency"] == (highest or 0) + 1), None) + confirmed = plan_doc["rounds"] >= 3 and boundary == "FAIL" and bool(fixtures) + verdict = ("PASS" if highest else "FAIL") if confirmed else "UNPROVEN" + missing = ["audio quality reference proof", "independent timer and runtime configuration verification"] + if plan_doc["rounds"] < 3: + missing.append("at least three fresh-process rounds per level") + if not fixtures: + missing.append("fixture file verified against its registered digest") + if boundary != "FAIL": + missing.append("complete failing level immediately above the passing prefix") + if any(row["verdict"] == "UNPROVEN" for row in levels): + missing.append("complete timing rows for all planned cells") + metrics.append(ev.metric("highest_passing_tested", highest, "streams", "max", n=len(levels))) + metrics.append(ev.metric("n_star", (highest or 0) if confirmed else None, "streams", "capacity", n=len(levels))) + store = ev.Store() + with store.staging_dir() as staging: + timing_path = staging / "timings.jsonl" + timing_path.write_text("".join(json.dumps(row) + "\n" for _, cell in sorted(cells.items()) + for _, (row, _) in sorted(cell.items())), encoding="utf8") + plan_path = staging / "plan.json" + plan_path.write_text(json.dumps(plan_doc, indent=2) + "\n", encoding="utf8") + record = ev.build( + call, kind=KIND, claim_class="derived", + summary=f"{label}: highest passing tested speech concurrency {highest}; capacity {verdict}", + identity_=ev.identity(device={"class": "mobile", "name": hardware_name}, + engine={"name": label, "version": engine_version}, + model={"id": model, "revision": model_revision, "dtype": model_dtype}, + workload={"mode": "load", "concurrency": plan_doc["levels"], + "request_set": {"sha256": sha256_hex(canonical_json(plan_doc)), "n": len(plan_doc["fixture_ids"])}, + "task": "tts", "endpoint_kind": "on_device_audio"}), + metrics=metrics, missing=missing, + artifacts=[ev.artifact(timing_path, staging, "requests"), ev.artifact(plan_path, staging, "load_plan")], + limitations=["Native on-device client times and process isolation are supplied by the capture producer.", + "Playback uses a PCM buffer simulation with immediate start and no extra prebuffer.", + "Burst arrivals represent this registered workload only.", + "Audio throughput is decoded seconds divided by the full scheduled window plus any drain past it."], + data={"source": "import audio-capacity", "hardware_name": hardware_name, "timer": "imported on-device client", + "delivery_mode": plan_doc["mode"], "levels": levels, "slo": slo, "verdict": verdict, + "highest_passing_tested": highest, "n_star": (highest or 0) if confirmed else None, + "quality_verdict": "UNPROVEN"}, + ) + return store.save(record, staging) diff --git a/src/inferencekit/ops/audio_scorers.py b/src/inferencekit/ops/audio_scorers.py index 49a957c..c28b6a3 100644 --- a/src/inferencekit/ops/audio_scorers.py +++ b/src/inferencekit/ops/audio_scorers.py @@ -84,8 +84,15 @@ def quality_audio_score(call, baseline, candidate, fixtures, protocol, config, o if not isinstance(rows, list) or len(rows) != len(jobs): raise OpError("audio scorer returned incomplete observations") required = ({"asr"} if "asr" in roles else set()) | ({"naturalness"} if "naturalness" in roles else set()) + naturalness_errors = 0 for row in rows: - if not isinstance(row, dict) or "error" in row or "listening" in row or not required.issubset(row): + if not isinstance(row, dict) or "listening" in row: + raise OpError("audio scorer returned invalid observations") + error = row.get("error") + missing = required - set(row) + if missing == {"naturalness"} and isinstance(error, dict) and error.get("naturalness") == "nonfinite_or_out_of_scale_score": + naturalness_errors += 1 + elif missing or "error" in row: raise OpError("audio scorer returned failed or incomplete observations") if "asr_retry" in roles and (not isinstance(row.get("asr"), dict) or "retry_text" not in row["asr"]): raise OpError("audio scorer omitted registered ASR retry") @@ -113,5 +120,6 @@ def quality_audio_score(call, baseline, candidate, fixtures, protocol, config, o finally: Path(staging_name).unlink(missing_ok=True) return {"format": qa.SCORES_FORMAT, "sha256": qa.sha(raw), "rows": len(rows), - "model_scorers": sorted(roles), "listening": "NOT RUN", "quality_verdict": "NOT RUN", + "model_scorers": sorted(roles), "naturalness_error_rows": naturalness_errors, + "listening": "NOT RUN", "quality_verdict": "NOT RUN", "next": "Import this file with quality audio --scores; model proxies do not establish listening quality."} diff --git a/src/inferencekit/ops/bench_audio.py b/src/inferencekit/ops/bench_audio.py index 028717c..c5ee87d 100644 --- a/src/inferencekit/ops/bench_audio.py +++ b/src/inferencekit/ops/bench_audio.py @@ -40,6 +40,7 @@ Arg("rounds", "integer", "Timed passes; screening in one server process.", default=3), Arg("first_round", "integer", "First round index; default zero or previous maximum plus one."), Arg("warmup", "integer", "Untimed distinct-text requests before measurement.", default=1), + Arg("warmup_fixtures", "string", "Optional separate warmup fixture JSONL, for precomputed inputs."), Arg("seed_base", "integer", "Shared deterministic round-seed base.", default=0), Arg("sample_rate", "integer", "Required decoded PCM sample rate.", default=44100), Arg("channels", "integer", "Required PCM channels.", default=1), @@ -211,7 +212,7 @@ def _previous(store, reference, config_sha, call): "first-audio p95, complete audio, RTF and buffer underruns. Audio quality remains UNPROVEN.", ARGS) def bench_audio(call, base_url, model, model_revision, model_dtype, model_quant, fixtures, label, rounds, first_round, warmup, seed_base, sample_rate, channels, settings, api_key_env, engine_version, hardware_name, run_dir, timeout_s, - resamples, save_outputs, slo_first_audio_p95_ms, slo_rtf, extend, endpoint_id=None, device="gpu", gpu_count=1): + resamples, save_outputs, slo_first_audio_p95_ms, slo_rtf, extend, endpoint_id=None, device="gpu", gpu_count=1, warmup_fixtures=None): if rounds < 1 or (first_round is not None and first_round < 0) or warmup < 0 or resamples < 1: raise OpError("rounds/resamples must be positive; first_round/warmup nonnegative") if not _ID.fullmatch(label): @@ -246,6 +247,10 @@ def bench_audio(call, base_url, model, model_revision, model_dtype, model_quant, raise OpError(f"settings conflict with declared {key}") setting_doc[key] = value fixture_rows, fixture_sha = load_fixtures(fixtures) + warm_fixtures, warm_fixture_sha = load_fixtures(warmup_fixtures) if warmup_fixtures else (fixture_rows, None) + measured_text = {f.text for f in fixture_rows} + if warmup_fixtures and any(f.text in measured_text for f in warm_fixtures): + raise OpError("warmup fixtures must use texts distinct from timed fixtures") try: parsed_url = urlsplit(base_url if "://" in base_url else "http://" + base_url) if parsed_url.username is not None or parsed_url.password is not None or parsed_url.query or parsed_url.fragment: @@ -266,6 +271,7 @@ def bench_audio(call, base_url, model, model_revision, model_dtype, model_quant, **({"device": "cpu"} if device == "cpu" else {}), **({"gpu_count": gpu_count} if gpu_count != 1 else {}), "engine_version": engine_version, "effective_settings": setting_doc, + **({"warmup_fixture_sha256": warm_fixture_sha} if warm_fixture_sha else {}), "save_outputs": save_outputs, "slo_first_audio_p95_ms": slo_first_audio_p95_ms, "slo_rtf": slo_rtf}).encode()) previous, previous_rows, previous_rounds = _previous(store, extend, config_sha, record_call) @@ -303,10 +309,12 @@ def request(fixture, seed, text=None): else: _write(roots, artifact["path"], blob) for i in range(warmup): - fixture = fixture_rows[i % len(fixture_rows)] - text = f"Inferencekit untimed speech warmup {i}. " + fixture.text - while text in {f.text for f in fixture_rows}: - text = "Warmup. " + text + fixture = warm_fixtures[i % len(warm_fixtures)] + text = fixture.text + if not warmup_fixtures: + text = f"Inferencekit untimed speech warmup {i}. " + text + while text in measured_text: + text = "Warmup. " + text result = backend.generate(request(fixture, round_seed(seed_base, -i - 1), text)) warm_rows.append({"ok": result.ok, "error": result.error}) for round_index in range(first_round, first_round + rounds): @@ -329,6 +337,7 @@ def request(fixture, seed, text=None): "n_failed": len(current_rows) - len(valid), "audio_ms": sum(r["complete_audio_ms"] for r in valid) / len(valid) if valid else None, "first_audio_p95_ms": pct([r["first_audio_ms"] for r in valid], 95), + "first_pcm_p95_ms": pct([r["first_pcm_ms"] for r in valid], 95), "rtf": sum(r["rtf"] for r in valid) / len(valid) if valid else None}) metrics, table = audio_metrics(rows, resamples) failed = sum(not row["ok"] for row in rows) @@ -341,7 +350,8 @@ def request(fixture, seed, text=None): "base_url_kind": "local" if endpoint.is_local() else "remote", "fixture_set": {"sha256": fixture_sha, "fixtures": [f.identity() for f in fixture_rows]}, "configuration_sha256": config_sha, "extends": previous["evidence_id"] if previous else None, - "rounds": round_rows, "warmup": {"excluded": True, "requests": warm_rows}, + "rounds": round_rows, "warmup": {"excluded": True, "requests": warm_rows, + "fixture_sha256": warm_fixture_sha}, "per_request": table, "failed_requests": failed, "total_requests": len(rows), "structural_verdict": structure, "timing_verdict": timing, "quality_verdict": "UNPROVEN", "verdict": verdict, "claim_validity": {"valid": not failed, "checked": [ diff --git a/src/inferencekit/ops/compare.py b/src/inferencekit/ops/compare.py index 8d67e11..30b2c31 100644 --- a/src/inferencekit/ops/compare.py +++ b/src/inferencekit/ops/compare.py @@ -50,11 +50,11 @@ CLIP_METRICS = ("clip_ms", "clips_per_hour") + tuple(f"{m}_{t}" for t in CLIP_TASKS for m in ("clip_ms", "clips_per_hour")) MEDIA_METRICS = {"image_benchmark": ("image_ms", "images_per_hour"), - "audio_benchmark": ("audio_ms", "first_audio_p95_ms", "rtf")} + "audio_benchmark": ("audio_ms", "first_audio_p95_ms", "first_pcm_p95_ms", "rtf")} HIGHER_IS_BETTER = {"decode_tok_s": True, "prefill_tok_s": True, "ttft_ms": False, "tpot_ms": False, "total_ms": False, **{m: m.startswith("clips_per_hour") for m in CLIP_METRICS}, "image_ms": False, "images_per_hour": True, "audio_ms": False, - "first_audio_p95_ms": False, "rtf": False} + "first_audio_p95_ms": False, "first_pcm_p95_ms": False, "rtf": False} BENCH_KINDS = ("benchmark", "serve_benchmark", "clip_benchmark", *MEDIA_METRICS) UNITS = {**{m: "clips/h" for m in CLIP_METRICS if m.startswith("clips_per_hour")}, "images_per_hour": "images/h", "rtf": "x"} @@ -342,6 +342,8 @@ def compare(call, baseline, candidate, metric, threshold, resamples, seed, image missing.append("input integrity") if rec.get("kind") not in BENCH_KINDS: reasons_inconclusive.append(f"{label} is a '{rec.get('kind')}' record, not a benchmark") + elif rec.get("kind") == "audio_benchmark" and rec.get("data", {}).get("source") == "import audio": + reasons_inconclusive.append(f"{label} contains imported PCM without benchmark timing") elif rec.get("kind") == "serve_benchmark" and (rec.get("identity", {}).get("workload") or {}).get( "mode") == "load": reasons_inconclusive.append(f"{label} is a load-mode serve_benchmark (one level, no rounds); compare " diff --git a/src/inferencekit/ops/import_audio.py b/src/inferencekit/ops/import_audio.py new file mode 100644 index 0000000..4750df5 --- /dev/null +++ b/src/inferencekit/ops/import_audio.py @@ -0,0 +1,128 @@ +"""Wrap saved PCM in a speech evidence record for registered quality scoring.""" + +from datetime import datetime +import json +from pathlib import Path +import wave + +from inferencekit import evidence as ev +from inferencekit.canonical import canonical_json, sha256_hex +from inferencekit.ops.bench_audio import _ID, load_fixtures, round_seed +from inferencekit.registry import Arg, OpError, operation +from inferencekit.serve.audio_client import ENCODING, MAX_PCM_BYTES + + +def _read_pcm(path, sample_rate, channels): + frame_bytes = channels * 2 + try: + if path.suffix == ".wav": + with wave.open(str(path), "rb") as handle: + frames = handle.getnframes() + if ((handle.getframerate(), handle.getnchannels(), handle.getsampwidth()) != (sample_rate, channels, 2) + or frames * frame_bytes > MAX_PCM_BYTES): + raise ValueError + pcm = handle.readframes(frames) + if len(pcm) != frames * frame_bytes: + raise ValueError + else: + with path.open("rb") as handle: + pcm = handle.read(MAX_PCM_BYTES + 1) + except (OSError, EOFError, ValueError, wave.Error): + raise OpError("unreadable or invalid PCM/WAV input") from None + if not pcm or len(pcm) > MAX_PCM_BYTES or len(pcm) % frame_bytes or not any(pcm): + raise OpError("saved PCM is empty, silent, oversized or incomplete") + return pcm + + +@operation("import_audio", ("import", "audio"), "Import saved PCM/WAV for speech quality without timing claims.", ( + Arg("fixtures", "string", "Original speech fixture JSONL.", required=True, sensitive=True), + Arg("dir", "string", "Directory of ID.wav/ID.pcm, or r0/, r1/, … for repeated captures.", required=True, sensitive=True), + Arg("label", "string", "Configuration label.", required=True), + Arg("model", "string", "Model identifier.", required=True), + Arg("model_revision", "string", "Checkpoint revision.", required=True), + Arg("engine_version", "string", "Runtime version or commit.", required=True), + Arg("model_dtype", "string", "Checkpoint dtype."), + Arg("model_quant", "string", "Quantization recipe."), + Arg("settings", "string", "Declared generation settings JSON, including voice.", default="{}"), + Arg("capture_start", "string", "Declared earliest capture start in UTC, in ISO 8601 with offset."), + Arg("rounds", "integer", "Number of complete fixture passes.", default=1), + Arg("seed_base", "integer", "Declared common bench audio round seed base.", default=0), + Arg("sample_rate", "integer", "Sample rate of the imported PCM/WAV.", required=True), + Arg("channels", "integer", "Number of audio channels.", default=1), +)) +def import_audio(call, fixtures, dir, label, model, model_revision, engine_version, model_dtype, model_quant, + settings, capture_start, rounds, seed_base, sample_rate, channels): + if not _ID.fullmatch(label) or not all(value.strip() for value in (model, model_revision, engine_version)): + raise OpError("audio import needs a file-safe label and nonempty model and engine pins") + if not 1 <= rounds <= 1000 or not 8000 <= sample_rate <= 192000 or not 1 <= channels <= 8: + raise OpError("invalid audio import rounds, sample rate or channel count") + try: + settings = json.loads(settings) + if (not isinstance(settings, dict) or ev.redact(settings) != settings + or set(settings) & {"model", "seed", "input", "sample_rate", "channels", "encoding", "stream"}): + raise ValueError + canonical_json(settings) + for key, value in (("model_revision", model_revision), ("engine_version", engine_version), + ("dtype", model_dtype), ("quant", model_quant)): + if value is not None and settings.setdefault(key, value) != value: + raise ValueError + start = datetime.fromisoformat(capture_start.replace("Z", "+00:00")) if capture_start else None + if start is not None and start.utcoffset() is None: + raise ValueError + if start is not None and start > datetime.fromisoformat(call.started.replace("Z", "+00:00")): + raise ValueError + timestamp = start.timestamp() if start else None + except (ValueError, TypeError, OSError, OverflowError): + raise OpError("audio import requires finite public settings and a timezone-aware capture start") from None + items, fixture_sha = load_fixtures(fixtures) + if ev.redact([label, model, model_revision, engine_version, model_dtype, model_quant] + + [item.id for item in items]) != ( + [label, model, model_revision, engine_version, model_dtype, model_quant] + + [item.id for item in items]): + raise OpError("audio capture identity fields must be safe to retain") + root = Path(dir).resolve() + store = ev.Store() + with store.staging_dir() as staging: + rows, outputs, artifacts, round_rows = [], [], [], [] + for index in range(rounds): + source = root / f"r{index}" if rounds > 1 or (root / "r0").is_dir() else root + seed = round_seed(seed_base, index) + for fixture in items: + candidates = [source / (fixture.id + ext) for ext in (".pcm", ".wav")] + found = [path for path in candidates if path.is_file() and path.resolve().is_relative_to(root)] + if len(found) != 1: + raise OpError(f"expected one PCM/WAV file for fixture {fixture.id} in round {index}") + pcm = _read_pcm(found[0], sample_rate, channels) + relative = f"outputs/{label}/r{index}/{fixture.id}.pcm" + target = staging / relative + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(pcm) + sha = sha256_hex(pcm) + rows.append({**fixture.identity(), "round": index, "seed": seed, "ok": True, + "sample_rate": sample_rate, "channels": channels, "sample_count": len(pcm) // (2 * channels), + "pcm_sha256": sha, "pcm_bytes": len(pcm)}) + outputs.append({"path": relative, "id": fixture.id, "round": index, "encoding": ENCODING, + "sample_rate": sample_rate, "channels": channels, "sha256": sha, "bytes": len(pcm)}) + artifacts.append(ev.artifact(target, staging, "audio_pcm")) + round_rows.append({"index": index, "seed": seed, "t_start": timestamp, "n_failed": 0}) + row_path = staging / "requests.jsonl" + row_path.write_text("".join(json.dumps(row) + "\n" for row in rows), encoding="utf8") + index_path = staging / "outputs" / label / "index.json" + index_path.write_text(json.dumps(outputs, indent=2) + "\n", encoding="utf8") + artifacts += [ev.artifact(row_path, staging, "requests"), ev.artifact(index_path, staging, "outputs_index")] + data = {"source": "import audio", "label": label, "rounds": round_rows, + "round_plan": {"expected_indices": list(range(rounds))}, "total_requests": len(rows), + "structural_verdict": "PASS", "timing_verdict": "NOT RUN", "quality_verdict": "UNPROVEN", + "fixture_set": {"sha256": fixture_sha, "fixtures": [item.identity() for item in items]}, + "audio_config": {"sample_rate": sample_rate, "channels": channels, "encoding": ENCODING, + "settings": settings, "seed_base": seed_base}} + record = ev.build(call, kind="audio_benchmark", claim_class="observed", + summary=f"{label}: imported {len(rows)} PCM outputs for quality", + identity_=ev.identity(engine={"name": label, "version": engine_version}, + model={"id": model, "revision": model_revision, + "dtype": model_dtype, "quant": model_quant}, + workload={"task": "tts", "endpoint_kind": "imported_pcm"}), + artifacts=artifacts, data=data, missing=["speech timing", "capture configuration verification"], + limitations=["Model, settings, seeds and capture times are declared by the importer.", + "PCM structure is checked independently of synthesis provenance."]) + return store.save(record, staging) diff --git a/src/inferencekit/ops/intake.py b/src/inferencekit/ops/intake.py index 87a4ed7..b9717d5 100644 --- a/src/inferencekit/ops/intake.py +++ b/src/inferencekit/ops/intake.py @@ -59,6 +59,7 @@ "decode_tok_s": ("higher", "tok/s"), "ttft_ms": ("lower", "ms"), "n_star": ("higher", "streams/GPU"), + "audio_s_per_s": ("higher", "audio s/s"), "decode_ceiling_fraction": ("higher", "fraction of roofline ceiling"), "ttfa_ms": ("lower", "ms"), "ttfa_p95_ms": ("lower", "ms"), @@ -71,7 +72,7 @@ } MODALITY_TARGETS = { "llm": {"decode_tok_s", "ttft_ms", "n_star", "decode_ceiling_fraction", "usd_per_1m"}, - "tts": {"ttfa_ms", "ttfa_p95_ms", "rtf", "n_star", "usd_per_1m"}, + "tts": {"ttfa_ms", "ttfa_p95_ms", "rtf", "n_star", "audio_s_per_s", "usd_per_1m"}, "stt": {"rtf", "n_star", "usd_per_1m"}, "video": {"clip_ms", "clips_per_hour"}, "image": {"image_ms", "images_per_hour"}, @@ -296,6 +297,7 @@ def normalize(data: dict) -> tuple[dict, list[str], list[str]]: hw = d["hardware"] hw.setdefault("count", 1) + hardware_label = hw.get("device") or hw.get("gpu") access = hw.setdefault("access", {"kind": "local"}) if access.get("kind") == "ssh": access.setdefault("port", 22) @@ -323,7 +325,15 @@ def normalize(data: dict) -> tuple[dict, list[str], list[str]]: warnings.append(f"goals.targets {bad} do not apply to modality {modality} " f"(use {sorted(MODALITY_TARGETS[modality])}); they are reported NOT RUN") needs_load = goals["lead"] == "capacity" or "n_star" in targets - if needs_load and "ttft_p95_ms" not in slo: + if needs_load and modality == "tts": + audio_slo = {"first_audio_p95_ms", "rtf_p95"} + if "rtf_p95" in slo: + slo.setdefault("max_underruns", 0) + elif "ttft_p95_ms" in slo and not audio_slo.intersection(slo) and d.get("workload", {}).get("client") != "audio_capacity": + warnings.append("Legacy TTS first-token SLO retained. Speech capacity needs rtf_p95.") + else: + errors.append("TTS capacity requires goals.slo.rtf_p95") + elif needs_load and "ttft_p95_ms" not in slo: errors.append("goals.slo.ttft_p95_ms is required when the lead goal is capacity or a target is n_star " "(N* is defined by the SLO)") if needs_load and modality == "llm" and "tpot_p95_ms" not in slo: @@ -385,12 +395,19 @@ def normalize(data: dict) -> tuple[dict, list[str], list[str]]: wl.setdefault("rounds", 5) client = wl.get("client") expected_client = {"llm": "bench_serve", "tts": "bench_audio", "image": "bench_image", "video": "bench_clip"} - if client is not None and client not in ("external", expected_client.get(modality)): + allowed_clients = {None, "external", expected_client.get(modality)} + if modality == "tts": + allowed_clients.add("audio_capacity") + if client == "audio_capacity" and (slo.get("max_underruns", 0) != 0 or slo["max_fail"] != 0): + errors.append("workload.client audio_capacity requires zero failures and zero underruns") + if client not in allowed_clients: errors.append(f"workload.client {client!r} does not support modality {modality}") if needs_load: - wl.setdefault("concurrency", list(DEFAULT_CONCURRENCY)) + wl.setdefault("concurrency", list(range(1, 9)) if client == "audio_capacity" else list(DEFAULT_CONCURRENCY)) if "concurrency" in wl: wl["concurrency"] = sorted(set(wl["concurrency"])) + if client == "audio_capacity" and wl["concurrency"] != list(range(1, len(wl["concurrency"]) + 1)): + errors.append("workload.client audio_capacity requires consecutive concurrency levels starting at 1") budget = d.setdefault("budget", {}) budget.setdefault("approved", False) @@ -444,7 +461,7 @@ def normalize(data: dict) -> tuple[dict, list[str], list[str]]: warnings.append("repo.path missing: interactive runs ask which repo to use; non_interactive runs skip " "the ledger (NOT RUN)") repo.setdefault("push", "ask") - repo.setdefault("branch", f"perf/{_slug(model['id'].split('/')[-1])}-{_slug(hw['gpu'])}") + repo.setdefault("branch", f"perf/{_slug(model['id'].split('/')[-1])}-{_slug(hardware_label)}") else: d["repo"] = None warnings.append("repo missing: kept changes are not committed; the commit ledger is NOT RUN") @@ -554,7 +571,7 @@ def normalize(data: dict) -> tuple[dict, list[str], list[str]]: cands.setdefault("exclude", []) cands.setdefault("max_engines", 5) if modality in ("tts", "stt") and client is None: - hint = "configure workload.client: bench_audio for speech synthesis, or external" if modality == "tts" else "configure workload.client: external for speech recognition" + hint = "configure workload.client: bench_audio or audio_capacity for speech synthesis, or external" if modality == "tts" else "configure workload.client: external for speech recognition" warnings.append(f"modality {modality}: no audio timing client is configured; {hint}. " "Audio latency claims without a matching client are UNPROVEN") if modality == "tts" and client == "bench_audio": @@ -570,7 +587,7 @@ def normalize(data: dict) -> tuple[dict, list[str], list[str]]: d["intake_version"] = 1 if not d.get("run_id"): stamp = _dt.datetime.now(_dt.timezone.utc).strftime("%Y%m%d-%H%M") - d["run_id"] = f"{_slug(model['id'].split('/')[-1])}-{_slug(hw['gpu'])}-{stamp}" + d["run_id"] = f"{_slug(model['id'].split('/')[-1])}-{_slug(hardware_label)}-{stamp}" return d, errors, warnings @@ -753,7 +770,8 @@ def derive(intake: dict) -> dict: goals, targets = intake["goals"], intake["goals"]["targets"] named = [k for k in targets if k != "buckets"] video = intake["modality"] in ("image", "video") - target_rows = [{"name": k, "value": targets[k], "better": TARGETS[k][0], "unit": TARGETS[k][1], + target_rows = [{"name": k, "value": targets[k], "better": TARGETS[k][0], + "unit": "streams/device" if k == "n_star" and intake["hardware"].get("device") else TARGETS[k][1], "buckets": ["all"] if video else targets.get("buckets", intake["buckets"]), "applies": k in MODALITY_TARGETS[intake["modality"]]} for k in named] ni = intake["non_interactive"] @@ -789,6 +807,8 @@ def derive(intake: dict) -> dict: modes = ["image"] if intake["modality"] == "tts" and intake["workload"].get("client") == "bench_audio": modes = ["audio"] + if intake["modality"] == "tts" and intake["workload"].get("client") == "audio_capacity": + modes = ["audio_capacity"] if intake["modality"] == "video": modes = ["clip"] # bench rounds --mode clip: one clip at a time questions = [] @@ -826,6 +846,7 @@ def derive(intake: dict) -> dict: round(hours * REPORT_RESERVE, 3) if hours else None), "stage_timeboxes": intake["budget"].get("stage_timeboxes", True), "affordability": goals.get("affordability"), + **({"resource_policy": goals["resource_policy"]} if "resource_policy" in goals else {}), "precision_rule": _precision_rule(intake["model"], fidelity, intake["modality"]), "fidelity_rule": {"value": fidelity, "fidelity": fidelity, "source": rules["fidelity"]["source"], "text": ("exact: equality of the declared deterministic/seeded outputs is required; " diff --git a/src/inferencekit/ops/ledger.py b/src/inferencekit/ops/ledger.py index 0b0d277..d60dc5d 100644 --- a/src/inferencekit/ops/ledger.py +++ b/src/inferencekit/ops/ledger.py @@ -48,7 +48,9 @@ from inferencekit.registry import Arg, OpError, operation ORDER = ("Perf-Workload", "Perf-Decode-Tok-S", "Perf-TTFT-Ms", "Perf-P95-Latency-Ms", "Perf-Quality", "Perf-Cost", - "Perf-Capacity-N", "Perf-Goodput", "Perf-Evidence", "Perf-Baseline") + "Perf-Capacity-N", "Perf-Goodput", "Perf-Speech-First-Audio-Ms", "Perf-Speech-First-PCM-Ms", + "Perf-Speech-RTF", "Perf-Speech-Peak-RSS-MiB", "Perf-Speech-Capacity-N", "Perf-Speech-Audio-S-Per-S", + "Perf-Evidence", "Perf-Baseline") SPEED_KINDS = ("benchmark", "serve_benchmark") # Metric names accepted for each headline (v1 names first; v2 aliases after). DECODE = ("decode_tok_s", "decode_tok_s_median") @@ -334,6 +336,8 @@ def build_trailers(recs: list[dict], base: dict | None, baseline_commit: str | N else: b.omit("Perf-Goodput", "no goodput in capacity evidence") + speech_trailers(b, recs) + ids = [r.get("evidence_id") for r in recs] + ([base.get("evidence_id")] if base else []) b.add("Perf-Evidence", ", ".join(short(i) for i in dict.fromkeys(ids) if i), "observed", []) baseline = baseline_commit or (short(base["evidence_id"]) if base else None) @@ -347,6 +351,44 @@ def build_trailers(recs: list[dict], base: dict | None, baseline_commit: str | N return b +def speech_trailers(builder, recs): + benchmarks = [r for r in recs if kind_of(r) == "audio_benchmark" + and r.get("data", {}).get("claim_validity", {}).get("valid") is True] + if benchmarks: + for key, name, stat in ( + ("Perf-Speech-First-Audio-Ms", "first_audio_ms", "p95"), + ("Perf-Speech-First-PCM-Ms", "first_pcm_ms", "p95"), + ("Perf-Speech-RTF", "rtf", "p95"), + ("Perf-Speech-Peak-RSS-MiB", "peak_rss_mib", None), + ): + got = pick(benchmarks, (name,), stats=(stat,) if stat else None) + if got: + metrics, record = got + value = values_text(metrics) + (" p95" if stat else "") + claim_class = "observed" if name in ("first_pcm_ms", "peak_rss_mib") else "derived" + builder.add(key, value, claim_class, [record.get("evidence_id")]) + for record in recs: + if kind_of(record) != "audio_capacity": + continue + data = record["data"] + level, source = data.get("n_star"), [record.get("evidence_id")] + if level is not None: + slo = data["slo"] + limits = [] + if slo.get("first_audio_p95_ms") is not None: + limits.append(f"p95 first audio <= {num(slo['first_audio_p95_ms'])} ms") + limits.extend((f"p95 RTF <= {num(slo['rtf_p95'])}", "zero underruns")) + value = f"{num(level)} streams (per stream: {', '.join(limits)})" + builder.add("Perf-Speech-Capacity-N", value, "derived", source) + subject = f"concurrency {level or data.get('highest_passing_tested')}" + metric = next((m for m in record.get("metrics", []) if m["name"] == "audio_s_per_s" + and m.get("subject") == subject and m.get("value") is not None), None) + if metric: + builder.add("Perf-Speech-Audio-S-Per-S", f"{num(metric['value'])} audio s/s at {subject}", + "derived", source) + break + + def _subject_metric(rec: dict, names, subject: str | None, others=()) -> dict | None: """The metric named in ``names`` for ``subject`` (the engine label), else the first engine-level one. Metrics whose subject is in ``others`` (another server's, e.g. the sampler check's baseline) never stand in.""" @@ -488,7 +530,7 @@ def select_engine(recs: list[dict], label: str | None) -> list[dict]: Two engines' speed records (or a baseline's and a candidate's passed together) would otherwise put the first one's number in the commit as its headline. """ - measured = SPEED_KINDS + ("capacity",) + measured = SPEED_KINDS + ("capacity", "audio_capacity", "audio_benchmark") labels = sorted({label_of(r) for r in recs if kind_of(r) in measured and label_of(r)}) if label is not None: if label not in labels: @@ -500,7 +542,9 @@ def select_engine(recs: list[dict], label: str | None) -> list[dict]: "for the engine this commit measures, and the baseline's record as --baseline-ev (or a " "`compare` record for a delta with a CI)") for what, group in (("single-user benchmark", [r for r in recs if kind_of(r) in SPEED_KINDS and not is_load(r)]), - ("capacity", [r for r in recs if kind_of(r) == "capacity"])): + ("capacity", [r for r in recs if kind_of(r) == "capacity"]), + ("speech benchmark", [r for r in recs if kind_of(r) == "audio_benchmark"]), + ("speech capacity", [r for r in recs if kind_of(r) == "audio_capacity"])): if len(group) > 1: raise OpError(f"{len(group)} {what} records for one engine " f"({', '.join(short(r.get('evidence_id')) for r in group)}): pass only the one this commit " diff --git a/src/inferencekit/ops/nsys.py b/src/inferencekit/ops/nsys.py index b7ffffd..86fcda5 100644 --- a/src/inferencekit/ops/nsys.py +++ b/src/inferencekit/ops/nsys.py @@ -57,16 +57,20 @@ def redact_outputs(staging: Path) -> None: Arg("trace", "string", "nsys -t value.", default="cuda,nvtx"), Arg("capture_range", "string", "Limit capture to cudaProfilerStart/Stop or an NVTX range.", default="none", choices=("none", "cudaProfilerApi", "nvtx")), + Arg("capture_range_end", "string", "Optional end-of-range behavior; stop lets the target keep running.", + choices=("none", "stop", "stop-shutdown")), Arg("nvtx_capture", "string", "NVTX range name for capture_range=nvtx."), Arg("delay", "number", "Seconds to wait before collecting."), Arg("duration", "number", "Seconds to collect."), Arg("timeout", "integer", "Seconds before the run (and every process it started) is killed.", default=3600)], passthrough=True) -def capture_nsys(call, keep_env, trace, capture_range, nvtx_capture, delay, duration, timeout, argv): +def capture_nsys(call, keep_env, trace, capture_range, nvtx_capture, delay, duration, timeout, argv, capture_range_end=None): nsys = require("nsys", "Install Nsight Systems (it ships with the CUDA toolkit).") if capture_range == "nvtx" and not nvtx_capture: raise OpError("capture_range=nvtx needs nvtx_capture=") + if capture_range_end is not None and capture_range == "none": + raise OpError("capture-range-end requires an active capture-range") env, env_names = clean_env(keep_env) # raises on secret-looking names, before anything runs dropped = dropped_config_env(env) store = ev.Store() @@ -76,6 +80,8 @@ def capture_nsys(call, keep_env, trace, capture_range, nvtx_capture, delay, dura "--force-overwrite", "true", "-o", str(base)] if capture_range != "none": cmd += ["--capture-range", capture_range] + if capture_range_end is not None: + cmd += ["--capture-range-end", capture_range_end] if nvtx_capture: cmd += ["--nvtx-capture", nvtx_capture] if delay: @@ -104,11 +110,13 @@ def capture_nsys(call, keep_env, trace, capture_range, nvtx_capture, delay, dura record = ev.build(call, kind="nsys_capture", claim_class="observed", summary=summary, identity_=ev.identity(gpu=gpu_identity()), artifacts=[ev.artifact(rep, staging, "nsys_rep"), ev.artifact(sqlite, staging, "sqlite"), - ev.artifact(staging / "stderr.txt", staging, "stderr")], + ev.artifact(staging / "stderr.txt", staging, "stderr"), + ev.artifact(staging / "stdout.txt", staging, "stdout")], missing=missing, limitations=limitations, data={"nsys_version": tool_version("nsys"), "exit_code": r.returncode, "env_names": env_names, "env_dropped": dropped, "sqlite_scrubbed": scrubbed, - "trace": trace, "cuda_graph_trace": "node", "capture_range": capture_range}) + "trace": trace, "cuda_graph_trace": "node", "capture_range": capture_range, + "capture_range_end": capture_range_end}) return store.save(record, staging) diff --git a/src/inferencekit/ops/quality_audio.py b/src/inferencekit/ops/quality_audio.py index da85b49..4408394 100644 --- a/src/inferencekit/ops/quality_audio.py +++ b/src/inferencekit/ops/quality_audio.py @@ -149,11 +149,12 @@ def score(values): "Score saved speech PCM with exact parity and registered, hash-bound ASR/MOS/listening observations.", [Arg("baseline", "string", "Baseline bench audio evidence id.", required=True), Arg("candidate", "string", "Candidate bench audio evidence id.", required=True), + Arg("baseline_repeat", "string", "Optional independent A/A baseline capture for exact PCM acceptance."), Arg("fixtures", "string", "Original speech fixture JSONL.", required=True), Arg("protocol", "string", "Evidence id from quality audio-policy.", required=True), Arg("scores", "string", "Optional external observation JSON; never executes or downloads models."), Arg("run_dir", "string", "Also copy evidence and quality/audio_score.json to this run.")]) -def quality_audio(call, baseline, candidate, fixtures, protocol, scores, run_dir): +def quality_audio(call, baseline, candidate, fixtures, protocol, scores, run_dir, baseline_repeat=None): store = ev.Store() registered, policy_root = qa.load_record(store, protocol, "quality_audio") if registered["data"].get("format") != qa.POLICY_FORMAT: @@ -171,16 +172,37 @@ def quality_audio(call, baseline, candidate, fixtures, protocol, scores, run_dir if set(arows) != set(brows): raise OpError("audio source request pairs are incomplete") keys = sorted(arows) + repeat = None + repeat_check = None + if baseline_repeat: + if policy["acceptance"] != "exact": + raise OpError("baseline-repeat is available for exact PCM acceptance") + repeat, repeat_rows = qa.load_benchmark(store, baseline_repeat) + _source_compatibility(a, repeat, policy) + if repeat["evidence_id"] in {a["evidence_id"], b["evidence_id"]}: + raise OpError("baseline-repeat must identify a separate capture") + a_engine, repeat_engine = (source["identity"]["engine"] for source in (a, repeat)) + if (set(arows) != set(repeat_rows) or any(a_engine.get(k) != repeat_engine.get(k) + for k in ("version", "commit", "flags"))): + raise OpError("baseline-repeat must match the baseline configuration and requests") + complete = [key for key in keys if all(rows[key]["ok"] and rows[key]["pcm"] is not None + for rows in (arows, repeat_rows))] + differences = [f"{key[0]}/r{key[1]}" for key in complete if arows[key]["pcm"] != repeat_rows[key]["pcm"]] + repeat_check = {"evidence_id": repeat["evidence_id"], "differences": differences, "compared": len(complete), + "verdict": "FAIL" if differences else "PASS" if len(complete) == len(keys) else "UNPROVEN"} registered_at = _timestamp(registered["provenance"]["ended"]) - starts = [r.get("t_start") for rec in (a, b) for r in rec["data"]["rounds"]] + sources = {"baseline": a, "candidate": b} + if repeat is not None: + sources["baseline_repeat"] = repeat + starts = [r.get("t_start") for rec in sources.values() for r in rec["data"]["rounds"]] preregistered = bool(registered_at is not None and starts and all( qa.finite(t) and registered_at <= t for t in starts)) observations, imported = qa.load_scores(scores, registered["evidence_id"], a["evidence_id"], b["evidence_id"], policy, {"baseline": arows, "candidate": brows}, {f.id: f.text for f in fixture_rows}) missing, pairs = [], [] - source_coverage = {arm: round_coverage(rec) for arm, rec in (("baseline", a), ("candidate", b))} - for arm, rec in (("baseline", a), ("candidate", b)): + source_coverage = {arm: round_coverage(rec) for arm, rec in sources.items()} + for arm, rec in sources.items(): coverage = source_coverage[arm] if (coverage["complete"] is False or coverage["gaps"] or rec["data"].get("scheduler_final") and coverage["complete"] is not True): @@ -207,6 +229,10 @@ def quality_audio(call, baseline, candidate, fixtures, protocol, scores, run_dir "parity": {"verdict": exact, "required": policy["acceptance"] == "exact"}, "round_coverage": {"verdict": round_verdict, "arms": source_coverage}, "registration": {"verdict": "PASS" if preregistered else "UNPROVEN"}} + if repeat_check is not None: + checks["baseline_repeat"] = repeat_check + if repeat_check["verdict"] != "PASS": + missing.append("byte-identical A/A baseline PCM") if not preregistered: missing.append("policy registered before every measured round") rounds = len(a["data"]["rounds"]) @@ -236,10 +262,12 @@ def quality_audio(call, baseline, candidate, fixtures, protocol, scores, run_dir subject=b["data"]["label"])) required = [structural, checks["registration"]["verdict"], "PASS" if enough and not missing else "UNPROVEN"] if policy["acceptance"] == "exact": - required.append(exact) + stable_baseline = repeat_check is None or repeat_check["verdict"] == "PASS" + required.append(exact if stable_baseline else "UNPROVEN") required.extend(checks[name]["verdict"] for name in policy["metrics"]) verdict = stats.combine_claims(required) scope = {"exact": "bit-exact preservation of the named baseline outputs", + "asr_noninferiority": "registered WER/CER noninferiority", "task_noninferiority": "registered speech task/proxy noninferiority (predicted MOS, not human listening)", "perceptual_and_task_noninferiority": "registered speech task and blinded-listening noninferiority"}[policy["acceptance"]] limitations = ["Quality is relative to the named baseline; baseline intelligibility and suitability are not established.", diff --git a/src/inferencekit/ops/report.py b/src/inferencekit/ops/report.py index a551c5e..59c75c7 100644 --- a/src/inferencekit/ops/report.py +++ b/src/inferencekit/ops/report.py @@ -71,6 +71,11 @@ def claims_of(rec: dict, expect: str, integrity: dict) -> list[dict]: out.append({"source": eid, "claim": f"{arm} speech latency and streaming SLOs", "verdict": timing if timing in ("PASS", "FAIL") else "UNPROVEN", "reason": "Absolute first-audio, RTF and underrun limits from the source benchmark."}) + elif kind == "audio_benchmark" and data.get("source") == "import audio": + out.append({"source": eid, "claim": "complete imported PCM", "verdict": data["structural_verdict"], + "reason": "PCM format, frame alignment and fixture coverage."}) + out.append({"source": eid, "claim": "speech timing", "verdict": "NOT RUN", + "reason": "This operation imports saved output for quality evaluation."}) elif kind in ("image_benchmark", "audio_benchmark"): validity = data.get("claim_validity") or {} structure = "PASS" if validity.get("valid") is True else "FAIL" if validity.get("valid") is False else "UNPROVEN" @@ -84,6 +89,13 @@ def claims_of(rec: dict, expect: str, integrity: dict) -> list[dict]: out.append({"source": eid, "claim": "media quality and usability", "verdict": "UNPROVEN", "reason": "Timing and structural checks do not establish image quality, " "speech intelligibility, speaker fidelity or repeated-run reliability; cite task-specific quality evidence."}) + elif kind == "audio_capacity": + out.append({"source": eid, "claim": "on-device speech capacity under the registered SLOs", + "verdict": data.get("verdict", "UNPROVEN"), + "reason": f"Highest passing tested level: {data.get('highest_passing_tested')}. " + "Open-loop burst arrivals with simulated zero-prebuffer playback."}) + out.append({"source": eid, "claim": "speech quality and process isolation", "verdict": "UNPROVEN", + "reason": "The timing import requires separate verification of the capture and outputs."}) elif kind == "quality_audio": from inferencekit.ops.quality_audio import report_verdict @@ -149,6 +161,7 @@ def environment_rows(recs: list[dict]) -> list[dict]: for i, memory in identities: g, e, m, w = i.get("gpu", {}), i.get("engine", {}), i.get("model", {}), i.get("workload", {}) cpu = i.get("cpu") or {} + device = i.get("device") or {} gpu = g.get("name") or "?" gpu_label = f"{g['count']}x {gpu}" if g.get("count") else gpu tuning = (e.get("flags") or {}).get("server_env") @@ -166,8 +179,9 @@ def environment_rows(recs: list[dict]) -> list[dict]: qualifiers.append("partial" if measurement.get("complete") is False else "coverage unknown") memory_parts.append(f"{name}: {_fmt(measurement.get('value'))} MiB" + (" (" + "; ".join(qualifiers) + ")" if qualifiers else "")) - row = {"gpu": gpu_label, "hardware": "CPU: " + (cpu.get("model") or "unknown") if cpu else gpu_label, - "driver / CUDA": "n/a" if cpu else f"{g.get('driver') or '?'} / {g.get('cuda') or '?'}", + hardware = device.get("name") or ("CPU: " + (cpu.get("model") or "unknown") if cpu else gpu_label) + row = {"gpu": gpu_label, "hardware": hardware, + "driver / CUDA": "n/a" if cpu or device else f"{g.get('driver') or '?'} / {g.get('cuda') or '?'}", "clocks": g.get("clocks") or "?", "engine": " ".join(str(x) for x in (e.get("name"), e.get("version")) if x) or "-", "model": " ".join(str(x) for x in (m.get("id"), m.get("dtype"), m.get("quant")) if x) or "-", diff --git a/src/inferencekit/quality/audio.py b/src/inferencekit/quality/audio.py index 0559df4..0ac575c 100644 --- a/src/inferencekit/quality/audio.py +++ b/src/inferencekit/quality/audio.py @@ -51,8 +51,8 @@ def validate_policy(doc): out = {"min_fixtures": 2, "min_rounds": 1, "resamples": 2000, "seed": 0, "allow_numerics_change": False, "metrics": {}, "scorers": {}, "normalization": "nfkc_casefold_punctuation_v1", **doc} - if out.get("acceptance") not in ("exact", "task_noninferiority", "perceptual_and_task_noninferiority"): - raise OpError("audio acceptance must be exact, task_noninferiority or perceptual_and_task_noninferiority") + if out.get("acceptance") not in ("exact", "asr_noninferiority", "task_noninferiority", "perceptual_and_task_noninferiority"): + raise OpError("audio acceptance must be exact, asr_noninferiority, task_noninferiority or perceptual_and_task_noninferiority") for key in ("min_fixtures", "min_rounds", "resamples", "seed"): if type(out[key]) is not int or out[key] < (0 if key == "seed" else 1): raise OpError(f"audio policy {key} must be a positive integer (seed may be zero)") @@ -65,8 +65,12 @@ def validate_policy(doc): raise OpError("unknown audio quality metric") if not isinstance(scorers, dict) or set(scorers) - {"asr", "asr_retry", "naturalness", "speaker", "listening"}: raise OpError("unknown audio scorer") - if out["acceptance"] != "exact" and (not set(ASR_METRICS).intersection(metrics) or "naturalness" not in metrics): - raise OpError("noninferiority requires registered WER or CER (or both) and naturalness margins") + if out["acceptance"] != "exact" and not set(ASR_METRICS).intersection(metrics): + raise OpError("speech task noninferiority requires WER or CER") + if out["acceptance"] in ("task_noninferiority", "perceptual_and_task_noninferiority") and "naturalness" not in metrics: + raise OpError("speech task/proxy noninferiority requires a naturalness margin") + if out["acceptance"] == "asr_noninferiority" and (set(metrics) - set(ASR_METRICS) or set(scorers) - {"asr", "asr_retry"}): + raise OpError("ASR noninferiority accepts only WER/CER and their registered scorers") if out["acceptance"] == "perceptual_and_task_noninferiority" and "listening" not in metrics: raise OpError("perceptual acceptance requires a registered listening metric") for name, spec in metrics.items(): @@ -138,8 +142,8 @@ def load_record(store, ref, kind): def load_benchmark(store, ref): record, root = load_record(store, ref, "audio_benchmark") data = record["data"] - if data.get("source") != "bench audio": - raise OpError("audio quality requires bench audio source evidence") + if data.get("source") not in ("bench audio", "import audio"): + raise OpError("audio quality requires bench audio or imported PCM source evidence") try: rows = [json.loads(line) for line in artifact_bytes(root, one_artifact(record, "requests")).splitlines() if line.strip()] @@ -308,6 +312,16 @@ def load_scores(path, protocol_id, baseline_id, candidate_id, policy, sources, t raise ValueError except (KeyError, TypeError, ValueError): raise OpError("imported audio observation has a duplicate, unknown or mismatched PCM/text identity") from None + error = item.get("error") + if isinstance(error, dict) and error.get("naturalness") == "nonfinite_or_out_of_scale_score": + windows = error.get("windows") + if (set(error) != {"naturalness", "windows"} or "naturalness" in item + or "naturalness" not in policy["metrics"] or not isinstance(windows, list) or not windows + or any(not isinstance(window, dict) or set(window) != {"start_frame", "end_frame"} + or any(type(v) is not int for v in window.values()) + or not 0 <= window["start_frame"] < window["end_frame"] <= row["sample_count"] + for window in windows)): + raise OpError("invalid naturalness scorer error") values = {} if "asr" in item: asr = item["asr"] diff --git a/src/inferencekit/quality/audio_scorers.py b/src/inferencekit/quality/audio_scorers.py index 13a15a4..f4fc264 100644 --- a/src/inferencekit/quality/audio_scorers.py +++ b/src/inferencekit/quality/audio_scorers.py @@ -118,6 +118,8 @@ def local(value): "predict_dataset": "sarulab", "num_repetitions": 1} else: defaults = {} + if role == "naturalness" and "min_tail_seconds" in spec.get("settings", {}): + defaults["min_tail_seconds"] = 0 if set(spec.get("settings", {})) - set(defaults): raise ValueError("unknown_scorer_setting") settings = {**defaults, **spec.get("settings", {})} @@ -134,6 +136,9 @@ def local(value): seconds = settings["window_seconds"] if type(seconds) not in (int, float) or not math.isfinite(seconds) or not 0 < seconds <= 3: raise ValueError("utmos_window_seconds_must_be_positive_at_most_three") + min_tail = settings.get("min_tail_seconds", 0) + if type(min_tail) not in (int, float) or not math.isfinite(min_tail) or not 0 <= min_tail <= seconds: + raise ValueError("utmos_min_tail_seconds_must_be_within_window") _positive(settings["fold"], "fold", 4, zero=True) _positive(settings["checkpoint_seed"], "checkpoint_seed", zero=True) _positive(settings["num_repetitions"], "num_repetitions") @@ -268,9 +273,12 @@ def describe(config): return result -def windows(frames, rate, seconds): +def windows(frames, rate, seconds, min_tail_seconds=0): step = max(1, int(rate * seconds)) - return [(start, min(start + step, frames)) for start in range(0, frames, step)] + result = [(start, min(start + step, frames)) for start in range(0, frames, step)] + if len(result) > 1 and (frames - result[-1][0]) / rate < min_tail_seconds: + result[-2:] = [(result[-2][0], frames)] + return result def _audio(raw, rate, channels): @@ -374,7 +382,8 @@ def _naturalness(spec, config, jobs, rows): for job, row in zip(jobs, rows): raw = Path(job["pcm_path"]).read_bytes() row["naturalness"] = {"windows": []} - for start, end in windows(job["sample_count"], job["sample_rate"], settings["window_seconds"]): + for start, end in windows(job["sample_count"], job["sample_rate"], settings["window_seconds"], + settings.get("min_tail_seconds", 0)): window = {"start_frame": start, "end_frame": end} row["naturalness"]["windows"].append(window) chunk = raw[start * job["channels"] * 2:end * job["channels"] * 2] @@ -397,8 +406,18 @@ def _naturalness(spec, config, jobs, rows): if len(scores) != len(batch): raise ValueError("missing_naturalness_scores") for (targets, _), score in zip(batch, scores): + try: + value = _scalar(score, 1, 5) + except ValueError: + continue for window in targets: - window["score"] = _scalar(score, 1, 5) + window["score"] = value + for row in rows: + invalid = [{"start_frame": window["start_frame"], "end_frame": window["end_frame"]} + for window in row["naturalness"]["windows"] if "score" not in window] + if invalid: + row.pop("naturalness") + row["error"] = {"naturalness": "nonfinite_or_out_of_scale_score", "windows": invalid} def _speaker(spec, config, jobs, rows, scratch): diff --git a/src/inferencekit/serve/media_rounds.py b/src/inferencekit/serve/media_rounds.py index 60642c4..24631a1 100644 --- a/src/inferencekit/serve/media_rounds.py +++ b/src/inferencekit/serve/media_rounds.py @@ -16,7 +16,7 @@ from inferencekit.serve import resources, round_evidence, rounds as rr, servers METRICS = { - "audio": {"audio_ms": "ms", "first_audio_p95_ms": "ms", "rtf": "x"}, + "audio": {"audio_ms": "ms", "first_audio_p95_ms": "ms", "first_pcm_p95_ms": "ms", "rtf": "x"}, "image": {"image_ms": "ms", "images_per_hour": "images/h"}, } @@ -137,7 +137,9 @@ def run_cell(spec: dict, k: int, position: int, cfg: dict) -> dict: mets = {name: {"value": row[name], "unit": unit, "stat": "per_round", "n": cfg["n"]} for name, unit in METRICS[mode].items() if round_evidence.finite(row.get(name))} mets["failed_requests"] = {"value": row.get("n_failed", 0), "unit": "requests", "n": cfg["n"]} - if all(round_evidence.finite(row.get(name)) for name in METRICS[mode]) or ( + # Older compatible speech clients have the original timing columns. + required_metrics = METRICS[mode].keys() - {"first_pcm_p95_ms"} + if all(round_evidence.finite(row.get(name)) for name in required_metrics) or ( round_evidence.finite(row.get("n_failed")) and row["n_failed"] > 0): cell["metrics"] = {"all": mets} cfg["last_ev"][spec["label"]] = record["evidence_id"] diff --git a/tests/test_audio_capacity.py b/tests/test_audio_capacity.py new file mode 100644 index 0000000..fe2ed0a --- /dev/null +++ b/tests/test_audio_capacity.py @@ -0,0 +1,104 @@ +import hashlib +import json + +import pytest + +from inferencekit import evidence as ev +from inferencekit.registry import OpError, call + + +def capture(tmp_path, change=lambda rows: rows): + fixtures = tmp_path / "fixtures.tsv" + fixtures.write_text("a\tFirst.\tSecond.\nb\tThird.\tFourth.\n") + plan = tmp_path / "plan.json" + plan.write_text(json.dumps({ + "levels": [1, 2], "rounds": 3, "waves": 1, "period_ms": 2000, "sample_rate": 24000, + "mode": "stream", "chunks_per_request": 2, "fixture_ids": ["a", "b"], + "fixture_sha256": hashlib.sha256(fixtures.read_bytes()).hexdigest(), + })) + rows = [] + for index in range(3): + for level in (1, 2): + for stream in range(level): + first = 100 if stream == 0 else 450 + rows.append({"round": index, "concurrency": level, "wave": 0, "stream": stream, + "id": "ab"[stream], "complete_ms": first + 350, + "chunks": [[first, 12000, 480], [first + 300, 12000, 0]]}) + recorded = tmp_path / "timings.jsonl" + recorded.write_text("".join(json.dumps(row) + "\n" for row in change(rows))) + args = {"timings": str(recorded), "plan": str(plan), "fixtures": str(fixtures), "label": "candidate", + "model": "example", "model_revision": "abc", "engine_version": "0.1", "hardware_name": "phone"} + return args + + +def test_capacity_checks_every_stream_and_window(tmp_path, store): + record = call("import_audio_capacity", capture(tmp_path)) + assert ev.validate(record) == [] + data = record["data"] + assert data["n_star"] == 1 + assert data["quality_verdict"] == "UNPROVEN" + assert record["identity"]["device"]["name"] == "phone" + trailers = call("ledger_trailers", {"ids": [record["evidence_id"]], "format": "json"}) + assert any(item["key"] == "Perf-Speech-Capacity-N" for item in trailers["trailers"]) + passing, failing = data["levels"] + assert passing["verdict"] == "PASS" + assert passing["rounds"][0]["audio_s_per_s"] == 0.5 + assert failing["verdict"] == "FAIL" + assert failing["rounds"][0]["streams"][1]["first_audio_p95_ms"] == 470 + + +def test_first_audio_can_be_measured_without_a_capacity_gate(tmp_path, store): + args = capture(tmp_path) + record = call("import_audio_capacity", {**args, "first_audio_as_metric": True}) + assert ev.validate(record) == [] + data = record["data"] + assert data["slo"] == {"rtf_p95": 1.0, "underruns": 0} + assert data["highest_passing_tested"] == 2 + assert data["n_star"] is None + assert data["levels"][1]["rounds"][0]["streams"][1]["first_audio_p95_ms"] == 470 + + def overload(rows): + for row in rows: + if row["concurrency"] == 2: + row["complete_ms"] = 1200 + return rows + + args = capture(tmp_path, overload) + record = call("import_audio_capacity", {**args, "first_audio_as_metric": True}) + assert record["data"]["n_star"] == 1 + trailers = call("ledger_trailers", {"ids": [record["evidence_id"]], "format": "json"}) + capacity = next(item for item in trailers["trailers"] if item["key"] == "Perf-Speech-Capacity-N") + assert "first audio" not in str(capacity) + assert "p95 RTF <= 1" in str(capacity) + + +def test_missing_rows_cannot_establish_capacity(tmp_path, store): + record = call("import_audio_capacity", capture(tmp_path, lambda rows: rows[:-1])) + assert record["data"]["n_star"] is None + assert record["data"]["highest_passing_tested"] == 1 + assert record["data"]["levels"][1]["verdict"] == "UNPROVEN" + + +def test_detects_underruns_despite_good_real_time_factor(tmp_path, store): + def delay(rows): + rows[0]["chunks"][1][0] = 650 + rows[0]["complete_ms"] = 700 + return rows + + record = call("import_audio_capacity", {**capture(tmp_path, delay), "first_audio_as_metric": True}) + assert record["data"]["n_star"] == 0 + assert record["data"]["levels"][0]["rounds"][0]["underruns"] == 1 + + +def test_rejects_tampered_fixture_and_malformed_failed_row(tmp_path, store): + args = capture(tmp_path) + args["fixtures"] = args["plan"] + with pytest.raises(OpError, match="fixture file differs"): + call("import_audio_capacity", args) + + def malformed(rows): + rows[0].update(complete_ms=None, chunks=[["invalid", 100, 0]]) + return rows + + with pytest.raises(OpError, match="timing row 1"): + call("import_audio_capacity", capture(tmp_path, malformed)) diff --git a/tests/test_audio_scorers.py b/tests/test_audio_scorers.py index 58a5276..61ecfd6 100644 --- a/tests/test_audio_scorers.py +++ b/tests/test_audio_scorers.py @@ -315,14 +315,15 @@ def timm_create(name, **kwargs): assert timm.create_model is timm_create -def test_utmos_predict_api_covers_tail_reuses_identical_windows_and_loads_once(tmp_path, monkeypatch): +@pytest.mark.parametrize("score_tail", [4.0, 0.3]) +def test_utmos_predict_api_covers_tail_reuses_identical_windows_and_loads_once(tmp_path, monkeypatch, score_tail): calls = [] class Predictor: def eval(self): calls.append(("eval", {})) def predict(self, **kwargs): calls.append(("predict", kwargs)) - return [4.0] * len(kwargs["data"]) + return [score_tail if len(audio) < 10 else 4.0 for audio in kwargs["data"]] def create(**kwargs): calls.append(("create", kwargs)) return Predictor() @@ -343,8 +344,13 @@ def create(**kwargs): rows = [{}, {}] sc._naturalness(settings["scorers"]["naturalness"], settings, jobs, rows) assert rows[0] == rows[1] - assert rows[0]["naturalness"] == {"windows": [ - {"start_frame": 0, "end_frame": 30, "score": 4.0}, {"start_frame": 30, "end_frame": 35, "score": 4.0}]} + if score_tail == 4.0: + assert rows[0]["naturalness"] == {"windows": [ + {"start_frame": 0, "end_frame": 30, "score": 4.0}, {"start_frame": 30, "end_frame": 35, "score": 4.0}]} + else: + assert rows[0] == {"error": {"naturalness": "nonfinite_or_out_of_scale_score", + "windows": [{"start_frame": 30, "end_frame": 35}]}} + assert sc.windows(35, 10, 3, min_tail_seconds=1) == [(0, 35)] assert [name for name, _ in calls].count("create") == 1 predictions = [kwargs for name, kwargs in calls if name == "predict"] assert len(predictions) == 2 diff --git a/tests/test_import_audio.py b/tests/test_import_audio.py new file mode 100644 index 0000000..41cfce8 --- /dev/null +++ b/tests/test_import_audio.py @@ -0,0 +1,34 @@ +import wave + +import pytest + +from inferencekit import evidence as ev +from inferencekit.quality.audio import load_benchmark +from inferencekit.registry import OpError, call + + +def test_import_valid_pcm_and_wav_for_quality(tmp_path, store): + fixture_file = tmp_path / "fixtures.jsonl" + fixture_file.write_text('{"id":"line","text":"Good morning."}\n') + params = {"fixtures": str(fixture_file), "dir": str(tmp_path), "label": "native", "model": "model", + "model_revision": "rev", "engine_version": "engine", "sample_rate": 24000, "rounds": 2} + pcm = b"\x00\x00\x01\x00\x01\x00" + for i in range(2): + (tmp_path / f"r{i}").mkdir() + (tmp_path / "r0" / "line.pcm").write_bytes(pcm) + with wave.open(str(tmp_path / "r1" / "line.wav"), "wb") as handle: + handle.setparams((1, 2, 24000, 0, "NONE", "not compressed")) + handle.writeframes(pcm) + + record = call("import_audio", params) + assert ev.validate(record) == [] + assert record["data"]["timing_verdict"] == "NOT RUN" + assert record["metrics"] == [] + loaded, rows = load_benchmark(store, record["evidence_id"]) + assert loaded["data"]["source"] == "import audio" + assert len(rows) == 2 + assert all(item["pcm"] == pcm for item in rows.values()) + + (tmp_path / "r0" / "line.pcm").write_bytes(pcm[:-1]) + with pytest.raises(OpError, match="incomplete"): + call("import_audio", params) diff --git a/tests/test_intake.py b/tests/test_intake.py index 20fe5ce..ccc0bc7 100644 --- a/tests/test_intake.py +++ b/tests/test_intake.py @@ -171,6 +171,46 @@ def test_capacity_lead_needs_an_slo_and_adds_load_mode(): "unit": "streams/GPU", "buckets": list(intake.BUCKETS), "applies": True} +@pytest.mark.parametrize("policy", ["maximum_performance", "shared_device_efficiency"]) +def test_explicit_resource_policy_is_retained(policy): + result = intake.check(base(goals={"resource_policy": policy})) + assert result["intake"]["goals"]["resource_policy"] == policy + assert result["derived"]["resource_policy"] == policy + + +def test_resource_policy_remains_opt_in(): + assert "resource_policy" not in intake.check(base())["derived"] + with pytest.raises(OpError, match="resource_policy"): + intake.check(base(goals={"resource_policy": "fast"})) + + +def test_native_tts_capacity_has_speech_slos_and_device_identity(): + data = base(modality="tts", hardware={"device": "Galaxy S25"}, budget={"approved": True}, + workload={"client": "audio_capacity", "concurrency": [1, 2, 3]}, + goals={"lead": "capacity", "slo": {"first_audio_p95_ms": 400, "rtf_p95": 1}, + "targets": {"n_star": 2, "audio_s_per_s": 1.5}}) + result = intake.check(data) + assert result["derived"]["modes"] == ["audio_capacity"] + assert result["intake"]["goals"]["slo"]["max_underruns"] == 0 + assert result["derived"]["targets"][0]["unit"] == "streams/device" + assert "galaxy-s25" in result["intake"]["run_id"] + data["workload"]["concurrency"] = [1, 3] + with pytest.raises(OpError, match="consecutive concurrency"): + intake.check(data) + data["workload"]["concurrency"] = [1, 2] + del data["goals"]["slo"]["rtf_p95"] + with pytest.raises(OpError, match="goals.slo.rtf_p95"): + intake.check(data) + + +def test_native_tts_realtime_capacity_does_not_need_a_first_audio_gate(): + data = base(modality="tts", hardware={"device": "Galaxy S25"}, budget={"approved": True}, + workload={"client": "audio_capacity"}, + goals={"lead": "capacity", "slo": {"rtf_p95": 1}}) + result = intake.check(data) + assert result["intake"]["goals"]["slo"] == {"rtf_p95": 1, "max_underruns": 0, "max_fail": 0} + + def test_ceiling_fraction_target_is_supported_and_warns_for_nvfp4(): d = base(goals={"targets": {"decode_ceiling_fraction": 0.7, "buckets": ["short"]}}) d["model"]["precision"] = "nvfp4" diff --git a/tests/test_interfaces.py b/tests/test_interfaces.py index e7160b0..3d717df 100644 --- a/tests/test_interfaces.py +++ b/tests/test_interfaces.py @@ -10,7 +10,7 @@ from inferencekit.mcp_server import fit from inferencekit.registry import OpError -CONTRACT_TOOLS = {"doctor", "roofline", "measure_bandwidth", "bench_decode", "capture_nsys", "analyze_nsys", +CONTRACT_TOOLS = {"import_audio", "import_audio_capacity", "doctor", "roofline", "measure_bandwidth", "bench_decode", "capture_nsys", "analyze_nsys", "capture_ncu", "summarize_ncu", "compare", "check_correctness", "list_evidence", "inspect_evidence", "report"} CONTRACT_TOOLS |= {"intake_template", "intake_validate", "intake_budget", "intake_resume"} # intake diff --git a/tests/test_quality_audio.py b/tests/test_quality_audio.py index aa16771..cd85301 100644 --- a/tests/test_quality_audio.py +++ b/tests/test_quality_audio.py @@ -145,6 +145,17 @@ def quality(run, scores=None): fixtures=str(fixtures), protocol=protocol["evidence_id"], scores=str(scores) if scores else None) +@pytest.mark.parametrize("unstable", [False, True]) +def test_exact_quality_with_independent_aa_control(tmp_path, store, unstable): + fixtures, protocol, baseline, candidate = setup_run(tmp_path, store) + repeat = bench(store, fixtures, "baseline-repeat", different=unstable) + record = invoke("quality_audio", baseline=baseline["evidence_id"], candidate=candidate["evidence_id"], + fixtures=str(fixtures), protocol=protocol["evidence_id"], baseline_repeat=repeat["evidence_id"]) + result = record["data"] + assert result["checks"]["baseline_repeat"]["verdict"] == ("FAIL" if unstable else "PASS") + assert result["verdict"] == ("UNPROVEN" if unstable else "PASS") + + def test_parity_signed_stereo_and_no_truncation(): a = struct.pack("<4h", -1000, 1000, -2000, 2000) b = struct.pack("<4h", -1001, 1001, -2002, 2002) @@ -282,6 +293,14 @@ def test_cer_and_wer_outliers_are_corroborated_independently_without_selecting_b assert quality(run, path)["data"]["checks"]["cer"]["verdict"] == "FAIL" +def test_asr_only_policy_reuses_registered_wer_margin(tmp_path): + doc = json.loads(policy_file(tmp_path, "task_noninferiority").read_text()) + doc["acceptance"] = "asr_noninferiority" + del doc["metrics"]["naturalness"] + del doc["scorers"]["naturalness"] + assert qa.validate_policy(doc)["acceptance"] == "asr_noninferiority" + + @pytest.mark.parametrize("mutation", ["missing_asr", "missing_task_metric", "missing_naturalness", "missing_outlier", "zero_outlier", "negative_margin", "wrong_outlier"]) def test_cer_requires_registered_task_margin_outliers_and_asr(tmp_path, mutation): diff --git a/tests/test_skills.py b/tests/test_skills.py index 8834c2c..f983f78 100644 --- a/tests/test_skills.py +++ b/tests/test_skills.py @@ -37,6 +37,11 @@ VIDEO_REFERENCES = (PLUGIN / "skills" / "optimizing-inference" / "references" / "video.md", PLUGIN / "skills" / "benchmarking" / "references" / "video-checks.md") MAX_VIDEO_REFERENCE_BYTES = 16_000 +# The on-device protocol is loaded only for phone workloads. +PHONE_REFERENCE = SKILLS / "inference-triage" / "references" / "phones.md" +MAX_PHONE_REFERENCE_BYTES = 4_000 +RESOURCE_POLICY_REFERENCE = SKILLS / "optimizing-inference" / "references" / "resource-policy.md" +MAX_RESOURCE_POLICY_REFERENCE_BYTES = 3_500 # Loaded only after the final report or for an explicit contribution request; keep # publication details out of the optimization context while bounding this reference. CONTRIBUTION_REFERENCE = PLUGIN / "skills" / "optimizing-inference" / "references" / "engine-contribution.md" @@ -180,10 +185,14 @@ def test_file_sizes_bounded(): skel = sum(p.stat().st_size for p in files if SKELETON in p.parents) video = sum(p.stat().st_size for p in VIDEO_REFERENCES) contribution = CONTRIBUTION_REFERENCE.stat().st_size - total = sum(p.stat().st_size for p in files) - skel - video - contribution + phone = PHONE_REFERENCE.stat().st_size + resource_policy = RESOURCE_POLICY_REFERENCE.stat().st_size + total = sum(p.stat().st_size for p in files) - skel - video - contribution - phone - resource_policy assert total <= MAX_PLUGIN_BYTES, f"plugin/ (without the skeleton and separately capped references) is {total} bytes" assert skel <= MAX_SKELETON_BYTES, f"the engine skeleton is {skel} bytes" assert video <= MAX_VIDEO_REFERENCE_BYTES, f"the video references are {video} bytes" + assert phone <= MAX_PHONE_REFERENCE_BYTES, f"the phone reference is {phone} bytes" + assert resource_policy <= MAX_RESOURCE_POLICY_REFERENCE_BYTES, f"the resource policy reference is {resource_policy} bytes" assert contribution <= MAX_CONTRIBUTION_REFERENCE_BYTES, f"the contribution reference is {contribution} bytes" @@ -313,7 +322,8 @@ def test_unprefixed_command_spans_exist(): for md in AGENT_MD: for ln, snippet in _code_snippets(md): m = re.match(r"^(?:\$IK\s+)?([a-z]+)[ \t]+([a-z][a-z0-9-]*)(\s.*)?$", _strip_quoted_args(snippet.strip())) - if not m or m.group(1) not in GROUPS - RAW_TOOLS: + # Require the CLI prefix for import, which is also a Python statement. + if not m or m.group(1) not in GROUPS - RAW_TOOLS - {"import"}: continue cmd = (m.group(1), m.group(2)) if cmd not in CLI: