Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ Each check ends PASS, FAIL, UNPROVEN (not enough evidence) or NOT RUN. A missing

## Features

- **9 skills, 4 commands and 5 read-only subagents**, backed by [47 executable CLI/MCP operations](https://github.com/KittenML/inferencekit/tree/main/src/inferencekit/ops/).
- **9 skills, 4 commands and 5 read-only subagents**, backed by [49 executable CLI/MCP operations](https://github.com/KittenML/inferencekit/tree/main/src/inferencekit/ops/).
- **Reusable run tools:** [ikstep](https://github.com/KittenML/inferencekit/blob/main/plugin/skills/optimizing-inference/references/remote-host.md) manages jobs on local or remote Linux hosts, readiness checks and concise status output.
- **Compact output:** CLI summaries and bounded MCP responses keep full evidence records on disk, with paging for large lists.
- **Speed-of-light ceilings:** measured memory bandwidth (`ceiling bw`) and exact bytes per token read from the checkpoint's safetensors headers (`roofline`), including NVFP4, FP8 and mixture-of-experts models.
Expand Down
7 changes: 7 additions & 0 deletions docs/audio-quality.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ The acceptance policy is explicit:
| Policy | Required evidence | Meaning of PASS |
|---|---|---|
| `exact` | Identical PCM, frame counts, rate and channels | Preservation of the named baseline outputs |
| `asr_noninferiority` | Paired WER and/or CER | Noninferiority under the registered ASR metrics |
| `task_noninferiority` | Paired WER/CER and predicted naturalness; optional speaker metric | Noninferiority under the registered task/proxy metrics |
| `perceptual_and_task_noninferiority` | Task metrics plus blinded, paired listening ratings | Noninferiority under the registered task and listening protocol |

Expand Down Expand Up @@ -124,3 +125,9 @@ For WER/CER the worse-direction difference is candidate minus baseline; for larg
SNR is `10*log10(sum(baseline²)/sum((baseline-candidate)²))`; maxabs is the largest signed-sample difference divided by 32768. Both use unmodified, aligned PCM. Unequal lengths have no SNR/maxabs value. Identical PCM has `snr_status: "identical"` and a null numeric SNR, avoiding a fabricated finite or nonfinite JSON value.

The retained `audio_score.json` lists per-pair parity, source IDs, policy, coverage and checks; `imported_scores.json` retains external observations. Report comparison quality can use only evidence bound to its exact baseline/candidate IDs. Proxy acceptance remains labelled as such; missing/changed observations become UNPROVEN, previous FAIL records remain visible, and lifecycle/reliability requirements are not satisfied by quality evidence.

For an independent exact PCM control, pass the repeat baseline capture with `--baseline-repeat` to `quality audio`. An unstable or incomplete A/A comparison leaves the candidate’s exactness unproven.

## Import native PCM

Run `inferencekit import audio --help` for the saved WAV or PCM import. Match the synthesis settings and seed base across arms, then pass the evidence IDs to `quality audio`. The import retains PCM and declared provenance for quality and carries no timing measurements. Use a policy registered before the declared capture start for an acceptance verdict.
2 changes: 2 additions & 0 deletions docs/audio-scorers.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,8 @@ inferencekit quality audio --baseline BASELINE_ID --candidate CANDIDATE_ID \
inferencekit report COMPARISON_ID QUALITY_ID --format md
```

Set UTMOSv2 `min_tail_seconds` to merge a shorter trailing window into its preceding window. An out-of-range score marks its utterance incomplete for naturalness and retains the other observations.

The score file binds each observation to its source PCM, text, seed, fixture and round. Models are reused within a scoring invocation. Missing or failed observations cannot become a quality PASS. Keep the observations with the evidence; they contain transcripts and are not automatically safe to publish.

## Optional checks
Expand Down
1 change: 1 addition & 0 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ python3 ~/.local/share/inferencekit/plugin/opencode/install.py
## The CLI or the MCP server alone

- **CLI:** `uvx --from 'inferencekit-cli==0.1.1' inferencekit doctor`, or `uv tool install 'inferencekit-cli==0.1.1'`.
- **Hosts without a build environment:** install the published wheel in an existing virtual environment with `python -m pip install --only-binary=:all: inferencekit-cli==0.1.1`. For unreleased builds, use the wheel from the matching source revision.
- **MCP server, for any MCP client:** `uvx --from 'inferencekit-cli[mcp]==0.1.1' inferencekit mcp serve` (stdio). Every CLI command is also a tool with the same arguments.
- **Unreleased source changes:** the plugin's MCP configuration uses the published CLI version above, not the current checkout. New operations require a release or an MCP server installed from the matching source; check the advertised tool list before relying on them.
- **If the MCP server does not start** (no `uvx`, no network, a failed install): install the CLI pinned to the plugin's version and let the agent call it from the shell. The skills say how.
Expand Down
2 changes: 1 addition & 1 deletion docs/media-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Repeat audio events as needed. `sample_count` counts frames per channel. The fin

Failed request rows have a null `sample_count` and no valid speed metrics. Their `received_pcm_frames` and `chunks` preserve how much decoded PCM arrived and when, including partial delivery before a timeout. These diagnostics do not establish a complete or usable speech output.

The client reports first PCM arrival, first nonzero signal arrival, simulated nonzero playback onset, complete generation time, real-time factor (wall time / generated audio duration), gaps and underruns with zero extra prebuffer. `first_audio_p95_ms` in round comparisons uses simulated nonzero onset, accounting for leading zero PCM and stalls. A nonzero sample is not proof of audible speech. Default timing limits are p95 onset <=2 seconds, maximum RTF <=1 and zero underruns; use the SLO flags to register different limits before measuring.
The client reports first PCM arrival (`first_pcm_p95_ms` for round comparisons), first nonzero signal arrival, simulated nonzero playback onset, complete generation time, real-time factor (wall time / generated audio duration), gaps and underruns with zero extra prebuffer. `first_audio_p95_ms` in round comparisons uses simulated nonzero onset, accounting for leading zero PCM and stalls. A nonzero sample is not proof of audible speech. Default timing limits are p95 onset <=2 seconds, maximum RTF <=1 and zero underruns; use the SLO flags to register different limits before measuring.

`inferencekit review build RUN_DIR` displays saved PNG images and speech outputs. Raw PCM is wrapped in a WAV under `review/media/` only after its hash, byte count, encoding, sample rate and channel count match `outputs/LABEL/index.json`; the source PCM stays unchanged. The review data lists each playback copy and its source hash. For speech captions, pass `--prompts FIXTURES.jsonl`: unbucketed fixture texts fill every observed round of the matching id. Attach the original evidence records to display per-request timing and failures. The page keeps simulated nonzero onset distinct from first PCM arrival and semantic quality remains UNPROVEN until separate quality evidence is supplied.

Expand Down
2 changes: 2 additions & 0 deletions plugin/skills/benchmarking/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ Choose the workload before the metric: complete-request latency, sustained throu

`bench rounds --mode audio|image|clip` runs fresh-process media comparisons; CPU manifests use `hardware.backend: cpu` for a CPU lock and process-group telemetry. Compare the final evidence ID for each engine in the run plan; [media contracts and round scheduling](https://github.com/KittenML/inferencekit/blob/main/docs/media-benchmarks.md).

For native phone speech, use [on-device speech](../inference-triage/references/phones.md) to import first audio, RTF, underruns and throughput under concurrent open-loop load.

Speed numbers that survive review: a timer outside the engine, one precision for every engine, settings proven in effect, warm engines, fresh-process rounds in a balanced order, confidence intervals, a speed-of-light ceiling and anti-cheat gates. [optimizing-inference](../optimizing-inference/SKILL.md) calls this skill.

**Use** for latency and tok/s per input bucket, capacity (`N*` within an SLO, goodput, $/1M tokens), comparisons of engines, configs or builds, and any reported number. A change-loop step needs only a screen; the full protocol is for milestones (fixed rounds: [references/statistics.md](references/statistics.md)) and anything published ([optimizing-kernels](../optimizing-kernels/SKILL.md), "Checks"). **Not** for *why* it is slow ([profiling](../profiling/SKILL.md)), nor while builds' outputs differ unchecked ([proving-correctness](../proving-correctness/SKILL.md) first).
Expand Down
1 change: 1 addition & 0 deletions plugin/skills/inference-triage/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ Explicit session authorization persists. An approved finite-deadline run may spe
NVIDIA is fully supported, and the only hardware inferencekit has been run on so far. On other hardware, the skills guide the same flow, and the CLI does less: [references/hardware.md](references/hardware.md) has the table.
- **Hardware-neutral, not yet run off NVIDIA:** `bench rounds`, `bench serve`, `gates run` and quality (they talk HTTP to any OpenAI-compatible server), `roofline` with a numeric `--gpu-bw`, and the intake, manifest, ledger and report tools. Treat a first run there as a test.
- **NOT RUN off NVIDIA:** `ceiling bw` and nsys/ncu captures. `doctor` records selected AMD DRM/rocminfo and Apple sysctl/system_profiler facts with a vendor profiling route; unknown details remain unknown. On AMD, `rocprof capture` and `rocprof analyze` create trace evidence with rocprofv3; device/version support must be checked. Use current official docs for unfamiliar tools and a vendor or STREAM-style bandwidth test. Captures made outside the CLI are vendor evidence files.
- **Phone workloads:** [on-device speech](references/phones.md). Capture request timings on the device.
- **Platform guides:** [AMD ROCm](references/amd-rocm.md), [x86 CPU](references/cpu.md), [Apple Silicon](references/apple-silicon.md), [Intel GPU and TPU](references/other-accelerators.md).

## Preflight checklist
Expand Down
27 changes: 27 additions & 0 deletions plugin/skills/inference-triage/references/phones.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# On-device speech and concurrency

Identify the phone model, SoC, OS build, runtime and actual CPU, GPU or NPU placement. For Android gather `adb shell getprop`, process affinity and `adb shell dumpsys thermalservice` before and after each cell. Read VmHWM or PSS separately with the process and measurement method identified. For iOS use the selected device's OS and runtime facts plus Instruments or native timers when available. Report inaccessible counters as NOT RUN.

Run the client and engine on the phone. With Android device labs, push a native arm64 test or start an app with ADB, collect its artifacts, and use the same phone for the baseline and candidate. A forwarded HTTP endpoint adds transport time. Label native measurements by where the timer starts and ends, and verify settings, operator placement and process isolation from runtime logs or vendor tools.

## Capacity under speech SLOs

- Pin the model, precision, voice, seed policy, text and sentence boundaries. Start with sentence streaming for interactive output. Benchmark whole-utterance delivery on the same text and report its first audio and complete latency. Native smaller chunks can be a separate workload.
- Set the RTF limit before testing. Register a first-audio limit if the user requires one. Schedule numbered waves of distinct fixtures at fixed times regardless of previous completion. At each level dispatch the stated number of requests together. Use the same ordered fixture set and period across configurations. Count time from scheduled release, including queueing. Warm up each fresh process outside timing.
- In each measured round all stream slots must meet the registered p95 limits. Simulate playback starting at first PCM with no extra buffer and require zero underruns. Track first PCM separately from the first nonzero playback sample. A response running beyond its wave period fails that level.
- Start at one stream and measure consecutive levels, including the first failing level above a contiguous passing prefix. A screen can identify a candidate; repeat with balanced fresh-process rounds, enough requests per stream to characterize the tail and a sustained thermal check. If the top measured level passes, report a lower bound. Report the wave period and output audio seconds per second of scheduled wall time.
- Establish completion and speech quality on saved output from the exact configurations under test. Native timing import does not attest to fixture contents, correctness, actual audio hardware playback, or the capture implementation.

Use `hardware.device`, `workload.client: audio_capacity` and `goals.lead: capacity` in the intake. Set `goals.slo.rtf_p95` and `max_underruns: 0`. Optional `first_audio_p95_ms` sets a separate gate.

Register the plan before timing. Fields are `levels` (consecutive from 1), `rounds`, `waves`, `period_ms`, `sample_rate`, `mode` (`stream` or `full`), `chunks_per_request` (1 for full), `fixture_ids` and `fixture_sha256` (digest of the fixture file). List one distinct ID per wave and stream at the largest level. For each run use fixture index `wave * concurrency + stream`.

Export one JSONL row per request with zero-based `round`, `wave`, `stream`, and fields `concurrency`, `id`, `complete_ms`, and `chunks`. Each chunk is an array of arrival milliseconds since scheduled release, PCM frame count and leading silent frame count. Use null completion for failed requests. Retain a separate record of the process launches, resolved settings, thermal state and capture code.

```bash
inferencekit import audio-capacity --timings requests.jsonl --plan plan.json --fixtures fixtures.tsv --label candidate \
--model MODEL --model-revision REV --engine-version VER --hardware-name DEVICE \
--first-audio-as-metric --slo-rtf-p95 1
```

For a latency-gated run, use `--slo-first-audio-p95-ms` in place of `--first-audio-as-metric`.
6 changes: 3 additions & 3 deletions plugin/skills/optimizing-inference/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,15 +49,15 @@ Use the derived run id; `--run-dir` pins the intake and starts the clock. SSH ru

Validate [intake](references/intake.md); pin revision, component precisions, processing, dataset, shapes and output work. Derive workload details from examples/logs ([workload.md](references/workload.md)); record assumptions.

Choose response latency, sustained throughput, peak memory or affordable hardware under a latency/quality limit ([affordability.md](references/affordability.md)). Token metrics and GSM8K belong to text workloads. Speech needs content, completion and speaker checks; images need semantic/visual checks; video needs temporal checks. Other architectures need a registered task-specific adapter and protocol; unsupported CLI modalities remain UNPROVEN rather than being relabelled as text.
Choose response latency, sustained throughput, peak memory or affordable hardware under a latency/quality limit ([affordability.md](references/affordability.md)). Apply a requested [compute resource policy](references/resource-policy.md). Token metrics and GSM8K belong to text workloads. Speech needs content, completion and speaker checks; images need semantic/visual checks; video needs temporal checks. Other architectures need a registered task-specific adapter and protocol; unsupported CLI modalities remain UNPROVEN rather than being relabelled as text.

Use [researching-optimizations](../researching-optimizations/SKILL.md) now, from the pinned model, shapes, hardware and runtime. It returns a short experiment queue, including existing engine paths to try before custom code.

## 2. Hardware, resource limits and headroom

Run `inferencekit doctor`. Check competing work, memory, swap, thermals and profiler permissions before occupying the device. Stop only the run's own jobs; do not kill unrelated services to obtain a clean measurement.

On NVIDIA, measured `ceiling bw` and `roofline` inform supported text models. Elsewhere use the [hardware route](../inference-triage/references/hardware.md) and vendor measurements; a PyTorch CUDA namespace does not establish NVIDIA-tool support. For non-token workloads derive bytes/FLOPs per request or step with assumptions. Unsupported ceilings stay NOT RUN.
On NVIDIA, measured `ceiling bw` and `roofline` inform supported text models. Elsewhere use the [hardware route](../inference-triage/references/hardware.md) and vendor measurements; a PyTorch CUDA namespace does not establish NVIDIA-tool support. For [phones](../inference-triage/references/phones.md), capture the SoC, OS, runtime placement and thermal state on-device and use its supported timers. For non-token workloads derive bytes/FLOPs per request or step with assumptions. Unsupported ceilings stay NOT RUN.

Estimate critical-path share, attainable efficiency, exact byte savings and permitted algorithmic savings ([ceilings.md](references/ceilings.md), [stop-point.md](references/stop-point.md)). Recompute after changing precision or work. An inferred ceiling orders experiments; it cannot establish a speedup.

Expand All @@ -71,7 +71,7 @@ For text, a fixed baseline with route build may skip the survey and use the [eng

## 4. Screen and measure

Use [harness-commands.md](references/harness-commands.md) for text, [affordability.md](references/affordability.md) for image/audio and [video.md](references/video.md) for video. Screen representative cases before full rounds. Text screens include `short` and `long` lengths. Promote secondary wins the user values without hiding the lead metric.
For phone speech concurrency use the [on-device speech capacity protocol](../inference-triage/references/phones.md). Use [harness-commands.md](references/harness-commands.md) for text, [affordability.md](references/affordability.md) for image/audio and [video.md](references/video.md) for video. Screen representative cases before full rounds. Text screens include `short` and `long` lengths. Promote secondary wins the user values without hiding the lead metric.

Final comparisons use balanced fresh-process rounds, excluded warmup, registered stopping and confidence intervals. Include complete output work and transfers/synchronization/decoding. [Capacity](references/capacity.md) needs its own tuning and measured concurrency.

Expand Down
Loading