[AlpinDale]

• Shard prefill indexer rows across TP ranks and remove CuTe tensor-equality overhead.
• Decode K=V once for BF16 prefill, vectorize shared staging, and size clustered-decode storage to the candidate width.
• Extend fused Q/KV RMSNorm, query quantization and KV RoPE to prefill. Preserve existing rounding and cache formats.

Enable sharding with indexer_tp_shard_min_rows=2048. Depends on #1788.

Attention update benchmarks

29cfe09f43 → 01f4bebe90, TP4 on 4× RTX PRO 6000 Blackwell Server GPUs; MNBT 4096, DSpark K7, paired P2P, max context 262,144, max sequences 8, capture 64, shard threshold 2048.

Input	C	Variant	Prefill tok/s	TTFT p50 / p95 (s)	Steady decode tok/s	Cold output tok/s
2,674	1	baseline	7,077.5	0.374 / 0.375	448.0	407.1
2,674	1	optimized	7,222.6	0.369 / 0.371	456.8	411.8
2,674	8	baseline	7,654.2	1.711 / 2.837	1,091.8	884.1
2,674	8	optimized	7,779.4	1.759 / 2.787	1,094.3	896.7
8,185	1	baseline	7,554.1	1.081 / 1.083	354.0	296.8
8,185	1	optimized	7,682.8	1.065 / 1.067	357.6	298.4
8,185	8	baseline	7,909.9	5.227 / 8.577	857.2	574.0
8,185	8	optimized	8,032.6	5.146 / 8.401	846.9	584.2
131,071	1	baseline	7,691.6	17.013 / 17.055	340.8	90.4
131,071	1	optimized	7,788.5	16.786 / 16.819	357.8	91.1
131,071	8	baseline	7,868.8	76.254 / 135.756	867.0	107.4
131,071	8	optimized	7,977.8	75.239 / 133.984	879.6	108.1

Fresh ABBA comparison; two servers per variant, six measured batches per cell; warmups excluded. Throughputs are aggregate medians. TTFT pools 6/48 requests at C=1/C=8 (nearest-rank p95).

Prefill uses fresh prompts with one output token. TTFT/cold output use separate fresh prompts with 2048 output tokens. Steady decode uses cached prompts and exact token IDs in the common-active window; all full generations ignore EOS.

DSpark acceptance varies across repeats; token/s changes are not isolated kernel-speed measurements.

Frozen attention replays remain bit-exact; 28 focused GPU tests and type checks for Python 3.11–3.13 pass.

Earlier MNBT sweep (29cfe09)

MNBT	Input tokens	C	Prefill tok/s	TTFT p50 / p95 (s)	Steady decode tok/s	Cold output tok/s
2048	2,674	1	5,215.1	0.513 / 0.515	443.4	396.8
2048	2,674	8	6,620.5	1.930 / 3.271	1,077.0	860.3
2048	8,185	1	7,143.8	1.145 / 1.146	353.1	297.0
2048	8,185	8	7,443.4	5.335 / 9.219	854.8	565.6
2048	131,071	1	7,242.3	18.071 / 18.109	370.6	86.3
2048	131,071	8	7,392.1	81.791 / 147.147	879.7	101.4
4096	2,674	1	7,093.5	0.375 / 0.376	452.0	431.4
4096	2,674	8	7,654.4	1.710 / 2.835	1,079.8	875.8
4096	8,185	1	7,579.7	1.080 / 1.081	346.3	294.0
4096	8,185	8	7,903.8	5.229 / 8.585	852.8	568.2
4096	131,071	1	7,716.2	16.970 / 16.985	373.1	90.9
4096	131,071	8	7,872.7	76.229 / 135.723	882.6	107.0
8192	2,674	1	7,089.3	0.376 / 0.382	459.8	419.6
8192	2,674	8	6,970.1	2.174 / 3.088	1,088.2	867.0
8192	8,185	1	6,812.2	1.194 / 1.203	339.6	286.5
8192	8,185	8	7,054.0	6.399 / 9.491	850.3	556.2
8192	131,071	1	6,910.3	18.951 / 18.984	356.6	82.6
8192	131,071	8	7,038.2	85.515 / 150.437	848.7	97.0

MNBT 4096 gave the best prefill rates in that sweep; other budgets were not rerun for the attention update.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	🔄 Running since 2026-10-28T21:18:08.832713Z	01f4beb	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.