[AlpinDale]

Through the NF12 package, this PR adds support for losslessly compressing BF16 models to roughly 12-bits, inspired by the NyoomFloat12 paper.

Throughput

Tested on an RTX PRO 6000 Blackwell Server Edition:

Model	Input tokens	Concurrency	BF16 output tok/s	NF12 output tok/s	Improvement
Qwen3-8B	8,192	1	77.42	89.34	+15.4%
Qwen3-8B	8,192	8	390.44	425.03	+8.9%
Qwen3-8B	8,192	16	543.68	575.19	+5.8%
Qwen3-8B	131,072	1	27.62	28.88	+4.6%
Qwen3-30B-A3B	8,192	1	127.05	149.69	+17.8%
Qwen3-30B-A3B	8,192	8	441.21	500.45	+13.4%
Qwen3-30B-A3B	8,192	16	614.10	680.09	+10.7%
Qwen3-30B-A3B	131,072	1	34.75	36.28	+4.4%

Memory savings

Model	BF16	NF12	Saved	Reduction
Qwen3-8B	15.26 GiB	12.05 GiB	3.20 GiB	21.0%
Qwen3-30B-A3B	56.87 GiB	43.00 GiB	13.88 GiB	24.4%

Use by loading a normal BF16 model and adding --quantization nf12 or quantization="nf12" if using the python API.

KL divergence

KL(BF16 || NF12), WikiText-2,

Model	Format	Mean KL (nats/token)	p95 KL	Max KL	Top-1 agreement
Qwen3-8B	NF12	0.000822	0.002360	0.014351	98.24%
Qwen3-8B	Official FP8	0.008362	0.028947	0.168127	96.29%
Qwen3-30B-A3B	NF12	0.002667	0.009884	0.053281	99.22%
Qwen3-30B-A3B	Official FP8	0.008214	0.035690	0.118317	96.68%

The reason why the agreement isn't 100% is because NF12 and BF16 in Sonar use different execution pipelines, so small numerical differences accumulate. NF12 guarantees lossless weight compression. Any lossless guarantees on the output tokens needs to be implemented on the execution side. Regardless, the difference is so minuscule and variant from run-to-run that it doesn't matter.

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-07T06:52:48.730916Z	67080eb	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.