Skip to content

About

One repo for turning an autoregressive LLM into a block-diffusion LLM, and for serving it.

Resources

Code of conduct

Contributing

Security policy

Stars

23 stars

Watchers

0 watching

Forks

Latest commit

 

History

66 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Trida-Stack

The Zoro-1 mascot decides which tool to use (ANSWER, 0.81), then replies "Hi! I'm Zoro-1. I write in blocks, not one by one." as each block of text resolves from noise

English | 한국어

One repo for turning an autoregressive LLM into a block-diffusion LLM that also makes typed decisions, and for serving it.

Zoro-1 Zoro License

Two independent stacks:

stack what it does
🏋️ train/ AR → diffusion conversion. Block-diffusion SFT starting from a pretrained AR checkpoint (no diffusion pretraining), with an AR auxiliary loss so the same weights keep a working causal head. Optional typed-decision joint training (a pointer head for yes/no, choice and score questions) and image inputs for vision-language checkpoints.
⚡ inference/<model>/ nano-inference: one small, nanoGPT-style serving stack per model. inference/zoro-1/ serves trillionlabs/Zoro-1: self-speculative generation and typed decisions from one checkpoint, OpenAI-compatible, on SGLang, vLLM or MLX. inference/gravityocr/ serves trillionlabs/GravityOCR (document OCR, GLM-OCR based) with self-speculative decoding on upstream SGLang.

train/ and inference/ do not import each other. benchmark/ holds the heavier agentic eval harnesses; inference/ ships its own quick quality/throughput benchmarks (gsm8k / mmlu_pro / ifeval).


Models

Checkpoints trained with this repo, on Hugging Face:

model what it is serve it with
Zoro-1 4B two-stream block-diffusion model (Qwen3.5-4B), jointly trained as an agent model and a decision model (decision_head.pt pointer head). Gated: request access, then hf auth login. inference/zoro-1/: --mode self-spec (default) / causal / decide
Zoro 4B block-diffusion model (Qwen3.5-4B, hybrid attention), formerly Trida2.0-4B, with a causal head for lossless self-speculative decoding. Apache-2.0. inference/zoro-1/mlx/ (Apple Silicon), inference/zoro-1/vllm/

The agentic benchmark harnesses in benchmark/ still default to the legacy trillionlabs/Trida-7B-Preview; moving them to Zoro-1 is tracked in #62.


Zoro-1 Chat (macOS app)

Zoro-1 Chat is a native SwiftUI app that runs Zoro-1 as an agent on your Mac, on the MLX stack. The model runs locally: no cloud and no API key.

  • Fast and on-device. Self-speculative decoding on Apple Silicon. Every reply shows its speed, tokens per forward and how much of the prompt came from the cache.
  • An agent with guard rails. It reads files and PDFs in a workspace folder you pick, uses a terminal, browses the web, and looks at images and screenshots. Read-only commands run in the macOS sandbox; anything else asks first.
  • Decisions at work. The decision head routes tools, checks each reply, judges approvals in auto mode, and answers your own /decide questions with a probability per option.
  • It learns from you. Facts from your chats go into a local knowledge graph you can browse (and undo), or add on demand with /update-kg. Your 👍 / 👎, edits and corrected decisions export as training data.
  • A mascot. The pixel mascot above shows what the model is doing: thinking, using a tool, answering, checking.

Requires an Apple Silicon Mac on macOS 14 or later. To build and install it (needs the Command Line Tools for swift):

cd inference/zoro-1/mlx && uv sync
cd app && zsh build_app.sh --install

On first launch, Get a Model downloads Zoro-1 and converts it to MLX. See the app README for every feature.


Supported model families

The trainer is architecture-agnostic in principle (any HF causal LM whose decoder is reachable via model.get_decoder()), but the block-diffusion masking and the cached decoders are validated on:

family attention notes
Qwen3 full attention (dense) the baseline path: dense block masks, standard KV cache
Qwen3.5 hybrid — gated-delta linear attention layers + periodic full attention layers needs hybrid-aware masking and cache handling: linear layers have no length dimension to crop, so speculative rollback uses conv/recurrent state snapshot + restore rather than a KV crop

The hybrid (Qwen3.5) path is the two-stream model behind Zoro-1, served in inference/zoro-1/ (--mode causal / self-spec / decide).


How the conversion works

Block-diffusion SFT (Fast-dLLM v2 style). Each row places a noised and a clean copy of the response after the prompt:

[ S (prompt) | x_t (response, noised) | x_0 (response, clean) ]

A custom attention mask ties them together:

  • S (prompt) — causal within itself, visible to everything after it (conditioning).
  • x_t (noised block k) — bidirectional within its own block, and attends to the prompt and the previous response blocks (their clean x_0 copies). So generation is autoregressive across blocks but diffusion within a block.
  • x_0 (clean) — block-causal over itself (token-causal when the AR aux loss is on).

Loss = masked cross-entropy at the noised x_t positions (the diffusion objective) + an optional next-token CE on x_0 (the AR auxiliary), combined as (diff + w·ar) / (1 + w). Fast-dLLM v2's complementary masking supervises every token across two paired views.

Two knobs matter most:

  • --bd_size — block length (e.g. 32). Decoding fills one block at a time.
  • --ar_loss_weight (w) — weight of the AR auxiliary loss. > 0 keeps a usable causal head, which is what makes speculative decoding (and plain AR serving) possible from the same checkpoint.

--within_block_causal makes x_t token-causal within the block too; combined with a causal x_0 this makes the model fully causal — loadable by a standard autoregressive serving engine — at the cost of some draft quality.

Typed decisions (joint training)

Zoro-1

The same weights can also be trained as a decision model: given a state (any JSON, plus an optional image) and typed questions, it returns calibrated probabilities instead of free text.

type question answer
noul yes/no P(yes)
choice {key: description} a distribution over the keys
score a list of levels, lowest first the expected level

--decision_data <kev-format JSONL> adds this objective in kev's scheme (train/data/decision.py / train/models/decision.py): a small pointer head on the clean causal stream scores each option's </opt> state against the question's <decide> state. Each optimizer step runs the usual SFT micro-batches, then --decision_micro_per_step decision micro-batches whose loss is added with weight --decision_loss_weight. Every checkpoint gets a jev_head.pt; train/tools/export_decision_checkpoint.py exports kev's layout. Details: train/README.md.

Images (vision-language checkpoints)

A Qwen3.5 checkpoint with a vision_config trains on conversations with image parts ({"type": "image", "image": "imgs/0001.png"}) alongside text-only data. The vision encoder is frozen; the projector trains at --projector_lr_scale × --lr; images above --max_image_pixels are downscaled. train/tools/compose_vl_checkpoint.py attaches a base vision tower to a text-only checkpoint. Details: train/README.md.


Quickstart

Tutorials

notebook what it walks through model
train/getting_started.ipynb the data format and a minimal block-diffusion SFT run (1 GPU, 2 steps) your own AR base, e.g. Qwen/Qwen3.5-4B; Zoro-1 is an example output
inference/zoro-1/getting_started.ipynb serve, call, benchmark, and read the numbers Zoro-1

Train (AR → diffusion)

Qwen3.5 two-stream (hybrid) runs need the one-time kernel fetch first — bash train/block_gated_delta_rule/fetch_kernels.sh (see Install).

PYTHONPATH=. torchrun --nproc_per_node=8 train.py \
  --model_id Qwen/Qwen3-4B \
  --bd_size 32 --ar_loss_weight 0.2 \
  --max_length 32768 --pack \
  --save_dir checkpoints/my-bd-run

Native-torch FSDP2, no DeepSpeed. python train.py --help lists all flags (packing, length/response bucketing, multi-turn supervision, activation offload, fused CE, …).

Add a typed-decision objective to the same run with --decision_data <kev-format JSONL files/dirs> (see Typed decisions); --decision_only trains the decision objective alone (no SFT data, no diffusion loss).

The saved checkpoint is a stock HF causal LM plus an added <|mask|> token and a block_diffusion.json sidecar (bd_size, mask_id, ar_loss_weight) — so it loads with plain AutoModelForCausalLM.from_pretrained.

Inference (serving — nano-inference, one stack per model)

Zoro-1

Each model gets its own small, readable serving stack under inference/<model>/, in the spirit of nanoGPT. inference/zoro-1/ serves trillionlabs/Zoro-1 (gated: request access, then hf auth login):

python inference/zoro-1/serve.py                  --port 30000   # default: self-spec (diffusion draft -> AR verify, b15_g8)
python inference/zoro-1/serve.py --mode causal    --port 30000   # native AR
python inference/zoro-1/serve.py --mode decide    --port 30001   # typed decisions + chat (stock SGLang)

Self-speculative decoding is lossless at matched sampling settings: every committed token is either an accepted draft that equals the AR token, or the AR correction. Its benefit is tokens per forward, not a different answer. See inference/zoro-1/README.md for chat, decisions, benchmarks and the model contract.

Decisions (--mode decide)

--mode decide runs serve_decide.py on stock SGLang ≥ 0.5.22 (no fork or patch): typed decisions at POST /v1/systemone (TypeSafe's System One API), reason-then-decide ("reasoning": true), and OpenAI chat with tool calls, from one engine and one copy of the weights.

curl -s localhost:30001/v1/systemone -H 'content-type: application/json' -d '{
  "model": "zoro-1",
  "state": {"ticket": "Order #W1 arrived broken. I want my money back."},
  "questions": {
    "intent": {"type": "choice", "instructions": "What does the customer want?",
               "criteria": {"refund": "money back", "exchange": "a replacement", "info": "just information"}},
    "urgent": {"type": "noul", "instructions": "Should this jump the queue?"}}}'
# -> {"model": "...", "answers": {"intent": {"type": "choice", "choice": "refund", "probabilities": {...}},
#                                 "urgent": {"type": "noul", "noul": 0.93}},
#     "usage": {"input_tokens": ..., "output_tokens": 0}, ...}

POST /v1/decide, the earlier {"request": {...}} envelope, still works but is deprecated and goes away in the next release. See inference/zoro-1/README.md#decisions.

Backends

backend where what
SGLang (patched) inference/zoro-1/sglang/ the self-speculative path (diffusion draft, AR verify) and causal decoding
SGLang (stock) inference/zoro-1/serve_decide.py --mode decide: typed decisions + chat
vLLM inference/zoro-1/vllm/ out-of-tree plugin, self-spec
MLX inference/zoro-1/mlx/ on-device (Apple Silicon): lossless self-spec, OpenAI-compatible server, image input, a small agent; q8 fits 16–18 GB

Layout

train/                   AR -> diffusion SFT trainer    (entry: train.py -> train.train:main)
  README.md                  launching, resume, model-family notes
  models/                    dense/hybrid models, construction, decision head
  data/                      encoding, packing, data loaders
  cli.py                     training options
  distributed.py             distributed setup and FSDP sharding
  optimizer.py, checkpoint.py optimizer updates, saving and resume
  train.py                   training helpers and main()
  tools/                     export_decision_checkpoint.py, compose_vl_checkpoint.py, parity/FSDP checks
  getting_started.ipynb      training tutorial
  block_gated_delta_rule/    fetch recipe for the block-causal gated-delta Triton kernels
                             (PolyForm-NC upstream, NOT vendored -- run fetch_kernels.sh first)
inference/               nano-inference: one small serving stack per model
  zoro-1/                    trillionlabs/Zoro-1
    README.md                  serving, decisions, benchmarks
    getting_started.ipynb      serving tutorial
    MODEL_CONTRACT.md          what the serving code expects from the checkpoint
    serve.py                   --mode causal | self-spec | decide; launches the server
    serve_decide.py, decide_*.py, jev*.py   typed-decision server (/v1/systemone) on stock SGLang
    chat.py                    OpenAI-compatible client
    eval.py / eval_ifeval.py   gsm8k / mmlu_pro / ifeval quality + throughput
    ifeval_lib/                vendored IFEval scorers
    configs/                   self-spec decode config (b15_g8)
    sglang/                    recipe for the patched SGLang backend
    vllm/                      out-of-tree vLLM plugin (self-spec)
    mlx/                       on-device backend (Apple Silicon): MLX self-spec decoder, OpenAI server, agent
  gravityocr/                trillionlabs/GravityOCR (document OCR, GLM-OCR based)
    README.md                  install, serving, request format, security, known limits
    MODEL_CONTRACT.md          what the serving code expects from the checkpoint
    serve.sh                   MODE=self-spec | causal; launches the patched SGLang server
    ocr_request.py             one-crop OCR client (OpenAI-compatible, image as a data URL)
    sglang/                    upstream SGLang v0.5.10 + self-spec patch, its new files, env lock, setup.sh
benchmark/               heavier agentic eval harnesses (bfcl_v4, tau2, functionchat, ko_agentbench,
                         swe_bench, terminal_bench) + serving/
data/                    dataset download scripts + catalog

Install

pip install -r requirements.txt              # training stack
pip install -r inference/zoro-1/requirements.txt    # serving/eval client

Serving needs the diffusion-serving SGLang backend — install separately (not vendored here); see inference/zoro-1/README.md. The vLLM backend is a plugin: pip install -e inference/zoro-1/vllm and VLLM_PLUGINS=trida_diffusion — see inference/zoro-1/vllm/README.md. GravityOCR builds its own patched upstream-SGLang environment (Python 3.12) with SGLANG_SRC=<dir outside this repo> bash inference/gravityocr/sglang/setup.sh and needs none of the fetches below — see inference/gravityocr/README.md.

Third-party kernels — one-time fetch (required)

Three pieces of this stack are PolyForm Noncommercial 1.0.0 upstream, so they are not vendored in this Apache-2.0 repo. Each ships a pinned fetch-and-patch recipe instead. Run the one for the path you use, before training or serving — without it the import fails.

you are… run what it assembles
training the Qwen3.5 two-stream (hybrid) path bash train/block_gated_delta_rule/fetch_kernels.sh two-stream Gated-DeltaNet + ShortConv Triton kernels — recipe
serving via SGLang follow inference/zoro-1/sglang/README.md the patched HybridDiffusion SGLang backend (clone → patch → install; serve.py shapes the checkpoint on first use)
serving via vLLM follow inference/zoro-1/vllm/vllm_native_diffusion/KERNELS.md block_causal_readout.py, the block-end readout kernel

Each recipe pins an upstream commit (yuchen-zhu-zyc/HybridDiffusion@6ca547a) and applies our patch. The fetched code is noncommercial-licensed and is not covered by this repo's Apache-2.0 license — see NOTICE and COMPLIANCE.md.

Getting help, contributing, reporting

Questions about install or usage Discussions · Q&A
A reproducible bug Issues
A feature idea Discussions · Ideas
A problem with a model revision Zoro-1 Community
Contributing CONTRIBUTING.md
A security vulnerability the private channel in SECURITY.md, not a public issue

What each channel is for and what response to expect: COMMUNITY.md. Code of conduct: CODE_OF_CONDUCT.md. Who decides what, and where decisions are recorded: GOVERNANCE.md.

License

See LICENSE, plus NOTICE and COMPLIANCE.md for third-party attribution.

Star History

Star History Chart

About

One repo for turning an autoregressive LLM into a block-diffusion LLM, and for serving it.

Resources

Code of conduct

Contributing

Security policy

Stars

23 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages