English | 한국어
One repo for turning an autoregressive LLM into a block-diffusion LLM that also makes typed decisions, and for serving it.
Two independent stacks:
| stack | what it does | |
|---|---|---|
| 🏋️ | train/ |
AR → diffusion conversion. Block-diffusion SFT starting from a pretrained AR checkpoint (no diffusion pretraining), with an AR auxiliary loss so the same weights keep a working causal head. Optional typed-decision joint training (a pointer head for yes/no, choice and score questions) and image inputs for vision-language checkpoints. |
| ⚡ | inference/<model>/ |
nano-inference: one small, nanoGPT-style serving stack per model. inference/zoro-1/ serves trillionlabs/Zoro-1: self-speculative generation and typed decisions from one checkpoint, OpenAI-compatible, on SGLang, vLLM or MLX. inference/gravityocr/ serves trillionlabs/GravityOCR (document OCR, GLM-OCR based) with self-speculative decoding on upstream SGLang. |
train/ and inference/ do not import each other. benchmark/ holds the heavier agentic eval
harnesses; inference/ ships its own quick quality/throughput benchmarks (gsm8k / mmlu_pro / ifeval).
Checkpoints trained with this repo, on Hugging Face:
| model | what it is | serve it with |
|---|---|---|
4B two-stream block-diffusion model (Qwen3.5-4B), jointly trained as an agent model and a decision model (decision_head.pt pointer head). Gated: request access, then hf auth login. |
inference/zoro-1/: --mode self-spec (default) / causal / decide |
|
4B block-diffusion model (Qwen3.5-4B, hybrid attention), formerly Trida2.0-4B, with a causal head for lossless self-speculative decoding. Apache-2.0. |
inference/zoro-1/mlx/ (Apple Silicon), inference/zoro-1/vllm/ |
The agentic benchmark harnesses in benchmark/ still default to the legacy
trillionlabs/Trida-7B-Preview; moving them to Zoro-1 is tracked in #62.
Zoro-1 Chat is a native SwiftUI app that runs Zoro-1 as an agent on your Mac, on the MLX stack. The model runs locally: no cloud and no API key.
- Fast and on-device. Self-speculative decoding on Apple Silicon. Every reply shows its speed, tokens per forward and how much of the prompt came from the cache.
- An agent with guard rails. It reads files and PDFs in a workspace folder you pick, uses a terminal, browses the web, and looks at images and screenshots. Read-only commands run in the macOS sandbox; anything else asks first.
- Decisions at work. The decision head routes tools, checks each reply, judges
approvals in auto mode, and answers your own
/decidequestions with a probability per option. - It learns from you. Facts from your chats go into a local knowledge graph you can browse (and
undo), or add on demand with
/update-kg. Your 👍 / 👎, edits and corrected decisions export as training data. - A mascot. The pixel mascot above shows what the model is doing: thinking, using a tool, answering, checking.
Requires an Apple Silicon Mac on macOS 14 or later. To build and install it (needs the Command Line
Tools for swift):
cd inference/zoro-1/mlx && uv sync
cd app && zsh build_app.sh --installOn first launch, Get a Model downloads Zoro-1 and converts it to MLX. See the app README for every feature.
The trainer is architecture-agnostic in principle (any HF causal LM whose decoder is reachable via
model.get_decoder()), but the block-diffusion masking and the cached decoders are validated on:
| family | attention | notes |
|---|---|---|
| Qwen3 | full attention (dense) | the baseline path: dense block masks, standard KV cache |
| Qwen3.5 | hybrid — gated-delta linear attention layers + periodic full attention layers | needs hybrid-aware masking and cache handling: linear layers have no length dimension to crop, so speculative rollback uses conv/recurrent state snapshot + restore rather than a KV crop |
The hybrid (Qwen3.5) path is the two-stream model behind Zoro-1, served in inference/zoro-1/
(--mode causal / self-spec / decide).
Block-diffusion SFT (Fast-dLLM v2 style). Each row places a noised and a clean copy of the response after the prompt:
[ S (prompt) | x_t (response, noised) | x_0 (response, clean) ]
A custom attention mask ties them together:
- S (prompt) — causal within itself, visible to everything after it (conditioning).
- x_t (noised block k) — bidirectional within its own block, and attends to the prompt and the previous response blocks (their clean x_0 copies). So generation is autoregressive across blocks but diffusion within a block.
- x_0 (clean) — block-causal over itself (token-causal when the AR aux loss is on).
Loss = masked cross-entropy at the noised x_t positions (the diffusion objective) + an optional
next-token CE on x_0 (the AR auxiliary), combined as (diff + w·ar) / (1 + w). Fast-dLLM v2's
complementary masking supervises every token across two paired views.
Two knobs matter most:
--bd_size— block length (e.g. 32). Decoding fills one block at a time.--ar_loss_weight(w) — weight of the AR auxiliary loss.> 0keeps a usable causal head, which is what makes speculative decoding (and plain AR serving) possible from the same checkpoint.
--within_block_causal makes x_t token-causal within the block too; combined with a causal x_0 this
makes the model fully causal — loadable by a standard autoregressive serving engine — at the cost
of some draft quality.
The same weights can also be trained as a decision model: given a state (any JSON, plus an optional image) and typed questions, it returns calibrated probabilities instead of free text.
| type | question | answer |
|---|---|---|
noul |
yes/no | P(yes) |
choice |
{key: description} |
a distribution over the keys |
score |
a list of levels, lowest first | the expected level |
--decision_data <kev-format JSONL> adds this objective in kev's
scheme (train/data/decision.py / train/models/decision.py): a small pointer head on the clean causal stream scores each option's
</opt> state against the question's <decide> state. Each optimizer step runs the usual SFT
micro-batches, then --decision_micro_per_step decision micro-batches whose loss is added with weight
--decision_loss_weight. Every checkpoint gets a jev_head.pt; train/tools/export_decision_checkpoint.py
exports kev's layout. Details: train/README.md.
A Qwen3.5 checkpoint with a vision_config trains on conversations with image parts
({"type": "image", "image": "imgs/0001.png"}) alongside text-only data. The vision encoder is
frozen; the projector trains at --projector_lr_scale × --lr; images above --max_image_pixels
are downscaled. train/tools/compose_vl_checkpoint.py attaches a base vision tower to a text-only
checkpoint. Details: train/README.md.
| notebook | what it walks through | model |
|---|---|---|
train/getting_started.ipynb |
the data format and a minimal block-diffusion SFT run (1 GPU, 2 steps) | your own AR base, e.g. Qwen/Qwen3.5-4B; Zoro-1 is an example output |
inference/zoro-1/getting_started.ipynb |
serve, call, benchmark, and read the numbers |
Qwen3.5 two-stream (hybrid) runs need the one-time kernel fetch first —
bash train/block_gated_delta_rule/fetch_kernels.sh(see Install).
PYTHONPATH=. torchrun --nproc_per_node=8 train.py \
--model_id Qwen/Qwen3-4B \
--bd_size 32 --ar_loss_weight 0.2 \
--max_length 32768 --pack \
--save_dir checkpoints/my-bd-runNative-torch FSDP2, no DeepSpeed. python train.py --help lists all flags (packing,
length/response bucketing, multi-turn supervision, activation offload, fused CE, …).
Add a typed-decision objective to the same run with --decision_data <kev-format JSONL files/dirs>
(see Typed decisions); --decision_only trains the decision objective alone (no SFT data, no diffusion loss).
The saved checkpoint is a stock HF causal LM plus an added <|mask|> token and a
block_diffusion.json sidecar (bd_size, mask_id, ar_loss_weight) — so it loads with plain
AutoModelForCausalLM.from_pretrained.
Each model gets its own small, readable serving stack under inference/<model>/, in the spirit of nanoGPT.
inference/zoro-1/ serves trillionlabs/Zoro-1
(gated: request access, then hf auth login):
python inference/zoro-1/serve.py --port 30000 # default: self-spec (diffusion draft -> AR verify, b15_g8)
python inference/zoro-1/serve.py --mode causal --port 30000 # native AR
python inference/zoro-1/serve.py --mode decide --port 30001 # typed decisions + chat (stock SGLang)Self-speculative decoding is lossless at matched sampling settings: every committed token is
either an accepted draft that equals the AR token, or the AR correction. Its benefit is
tokens per forward, not a different answer. See inference/zoro-1/README.md
for chat, decisions, benchmarks and the model contract.
--mode decide runs serve_decide.py on stock SGLang ≥ 0.5.22 (no fork or patch): typed
decisions at POST /v1/systemone (TypeSafe's System One API), reason-then-decide
("reasoning": true), and OpenAI chat with tool calls, from one engine and one copy of the weights.
curl -s localhost:30001/v1/systemone -H 'content-type: application/json' -d '{
"model": "zoro-1",
"state": {"ticket": "Order #W1 arrived broken. I want my money back."},
"questions": {
"intent": {"type": "choice", "instructions": "What does the customer want?",
"criteria": {"refund": "money back", "exchange": "a replacement", "info": "just information"}},
"urgent": {"type": "noul", "instructions": "Should this jump the queue?"}}}'
# -> {"model": "...", "answers": {"intent": {"type": "choice", "choice": "refund", "probabilities": {...}},
# "urgent": {"type": "noul", "noul": 0.93}},
# "usage": {"input_tokens": ..., "output_tokens": 0}, ...}POST /v1/decide, the earlier {"request": {...}} envelope, still works but is deprecated and goes
away in the next release.
See inference/zoro-1/README.md#decisions.
| backend | where | what |
|---|---|---|
| SGLang (patched) | inference/zoro-1/sglang/ |
the self-speculative path (diffusion draft, AR verify) and causal decoding |
| SGLang (stock) | inference/zoro-1/serve_decide.py |
--mode decide: typed decisions + chat |
| vLLM | inference/zoro-1/vllm/ |
out-of-tree plugin, self-spec |
| MLX | inference/zoro-1/mlx/ |
on-device (Apple Silicon): lossless self-spec, OpenAI-compatible server, image input, a small agent; q8 fits 16–18 GB |
train/ AR -> diffusion SFT trainer (entry: train.py -> train.train:main)
README.md launching, resume, model-family notes
models/ dense/hybrid models, construction, decision head
data/ encoding, packing, data loaders
cli.py training options
distributed.py distributed setup and FSDP sharding
optimizer.py, checkpoint.py optimizer updates, saving and resume
train.py training helpers and main()
tools/ export_decision_checkpoint.py, compose_vl_checkpoint.py, parity/FSDP checks
getting_started.ipynb training tutorial
block_gated_delta_rule/ fetch recipe for the block-causal gated-delta Triton kernels
(PolyForm-NC upstream, NOT vendored -- run fetch_kernels.sh first)
inference/ nano-inference: one small serving stack per model
zoro-1/ trillionlabs/Zoro-1
README.md serving, decisions, benchmarks
getting_started.ipynb serving tutorial
MODEL_CONTRACT.md what the serving code expects from the checkpoint
serve.py --mode causal | self-spec | decide; launches the server
serve_decide.py, decide_*.py, jev*.py typed-decision server (/v1/systemone) on stock SGLang
chat.py OpenAI-compatible client
eval.py / eval_ifeval.py gsm8k / mmlu_pro / ifeval quality + throughput
ifeval_lib/ vendored IFEval scorers
configs/ self-spec decode config (b15_g8)
sglang/ recipe for the patched SGLang backend
vllm/ out-of-tree vLLM plugin (self-spec)
mlx/ on-device backend (Apple Silicon): MLX self-spec decoder, OpenAI server, agent
gravityocr/ trillionlabs/GravityOCR (document OCR, GLM-OCR based)
README.md install, serving, request format, security, known limits
MODEL_CONTRACT.md what the serving code expects from the checkpoint
serve.sh MODE=self-spec | causal; launches the patched SGLang server
ocr_request.py one-crop OCR client (OpenAI-compatible, image as a data URL)
sglang/ upstream SGLang v0.5.10 + self-spec patch, its new files, env lock, setup.sh
benchmark/ heavier agentic eval harnesses (bfcl_v4, tau2, functionchat, ko_agentbench,
swe_bench, terminal_bench) + serving/
data/ dataset download scripts + catalog
pip install -r requirements.txt # training stack
pip install -r inference/zoro-1/requirements.txt # serving/eval clientServing needs the diffusion-serving SGLang backend — install separately (not vendored here);
see inference/zoro-1/README.md. The vLLM backend is a plugin:
pip install -e inference/zoro-1/vllm and VLLM_PLUGINS=trida_diffusion — see inference/zoro-1/vllm/README.md.
GravityOCR builds its own patched upstream-SGLang environment (Python 3.12) with
SGLANG_SRC=<dir outside this repo> bash inference/gravityocr/sglang/setup.sh
and needs none of the fetches below — see inference/gravityocr/README.md.
Three pieces of this stack are PolyForm Noncommercial 1.0.0 upstream, so they are not vendored in this Apache-2.0 repo. Each ships a pinned fetch-and-patch recipe instead. Run the one for the path you use, before training or serving — without it the import fails.
| you are… | run | what it assembles |
|---|---|---|
| training the Qwen3.5 two-stream (hybrid) path | bash train/block_gated_delta_rule/fetch_kernels.sh |
two-stream Gated-DeltaNet + ShortConv Triton kernels — recipe |
| serving via SGLang | follow inference/zoro-1/sglang/README.md |
the patched HybridDiffusion SGLang backend (clone → patch → install; serve.py shapes the checkpoint on first use) |
| serving via vLLM | follow inference/zoro-1/vllm/vllm_native_diffusion/KERNELS.md |
block_causal_readout.py, the block-end readout kernel |
Each recipe pins an upstream commit (yuchen-zhu-zyc/HybridDiffusion@6ca547a) and applies our
patch. The fetched code is noncommercial-licensed and is not covered by this repo's Apache-2.0
license — see NOTICE and COMPLIANCE.md.
| Questions about install or usage | Discussions · Q&A |
| A reproducible bug | Issues |
| A feature idea | Discussions · Ideas |
| A problem with a model revision | Zoro-1 Community |
| Contributing | CONTRIBUTING.md |
| A security vulnerability | the private channel in SECURITY.md, not a public issue |
What each channel is for and what response to expect: COMMUNITY.md. Code of conduct: CODE_OF_CONDUCT.md. Who decides what, and where decisions are recorded: GOVERNANCE.md.
See LICENSE, plus NOTICE and COMPLIANCE.md for third-party attribution.
