Clean PyTorch implementations of landmark vision models, organized by task:
| Task | Models | Key Idea |
|---|---|---|
| Image Classification | ViT | Patch → Transformer → [CLS] classification |
| Object Detection | DETR, Deformable DETR, DINO, RT-DETR, YOLO | Set prediction / anchor-free / real-time |
| Image Segmentation | SAM, SAM2 | Promptable segmentation (point/box/mask) |
| Video Tracking | SAM2 | Memory-guided object tracking in video |
| 3D Reconstruction | Occupancy Network | Image → continuous occupancy function → infinite-res mesh |
| BEV Perception | BEVFormer, TinyBEV | Multi-view camera → BEV → Detection + Forecasting + Planning |
| BEV Perception | TinyBEV | Multi-view camera → LSS BEV → Detection + Forecasting + Planning |
| BEV Perception | TinyBEV | Multi-view camera → LSS BEV → Detection + Forecasting + Planning |
| Image Generation | DDPM, Stable Diffusion, DiT | Forward noise → U-Net/Transformer predicts noise → reverse denoise |
| Generative Model | VAE | Reparameterization trick + KL → smooth latent space → generation |
| Multimodal | CLIP | Contrastive image-text alignment → zero-shot transfer |
Input Image (H × W × C)
│
▼
┌──────────────────┐
│ Patch Embedding │ Conv2d: split image into P×P patches, project to D dims
└──────────────────┘
│
▼
┌──────────────────┐
│ + [CLS] Token │ Prepend a learnable class token
│ + Pos Embed │ Add learnable positional embeddings
└──────────────────┘
│
▼
┌──────────────────┐
│ Transformer ×L │ L blocks of: LayerNorm → MSA → Res → LayerNorm → MLP → Res
└──────────────────┘
│
▼
┌──────────────────┐
│ Layer Norm │
│ [CLS] → Head │ Linear classification head on the [CLS] token
└──────────────────┘
│
▼
Class Logits
cv/
├── implementations/ # From-scratch model implementations
│ ├── vit.py # Vision Transformer
│ ├── detr.py # DETR (base)
│ ├── deformable_detr.py # Deformable DETR (multi-scale deformable attention)
│ ├── dino.py # DINO (de-noising training for DETR)
│ ├── rt_detr.py # RT-DETR (real-time DETR)
│ ├── sam.py # SAM (Segment Anything Model)
│ ├── sam2_from_scratch.py # SAM 2 (video tracking)
│ ├── yolo_simplified.py # YOLO (simplified single-shot detector)
│ ├── clip.py # CLIP (contrastive language-image pre-training)
│ ├── diffusion.py # DDPM (denoising diffusion probabilistic models)
│ ├── dit.py # DiT (Diffusion Transformer, adaLN-Zero conditioning)
│ ├── vae.py # VAE (Variational Autoencoder, reparameterization trick)
│ ├── stable_diffusion.py # Stable Diffusion (latent diffusion + text conditioning)
│ ├── occupancy_network.py # Occupancy Network (implicit 3D reconstruction)
│ ├── bev_former.py # BEVFormer (spatiotemporal BEV transformer)
│ ├── tiny_bev.py # TinyBEV (lightweight BEV perception + planning)
│ └── cnn_test.py # SimpleCNN with image visualization
├── pretrained_demos/ # Demos using official pretrained weights
│ ├── detr_pretrained_demo.py
│ ├── sam_pretrained_demo.py
│ ├── sam2_pretrained_demo.py
│ ├── yolo_demo.py
│ ├── yolo_pretrained_demo.py
│ ├── yolo_real_demo.py
│ ├── clip_pretrained_demo.py # CLIP zero-shot classification & retrieval
│ ├── stable_diffusion_pretrained_demo.py # SD 1.5 text-to-image generation
│ └── tiny_bev_demo.py # TinyBEV synthetic demo
├── video_apps/ # Video application demos
│ ├── detr_video_demo.py # DETR video object detection
│ └── sam2_scratch_video_demo.py
├── training/ # Training scripts
│ ├── train.py # ViT training on CIFAR-10
│ └── demo.py
├── weights/ # Pretrained model weights
├── data/ # Datasets
└── outputs/ # Demo output images
| Variant | Emb Size | Depth | Heads | Params (224px) |
|---|---|---|---|---|
| ViT-Tiny | 192 | 12 | 3 | ~5.7M |
| ViT-Small | 384 | 12 | 6 | ~22M |
| ViT-Base | 768 | 12 | 12 | ~86M |
| ViT-Large | 1024 | 24 | 16 | ~307M |
| ViT-Huge | 1280 | 32 | 16 | ~632M |
pip install -r requirements.txtpython vit.pyExpected output:
Model: ViT-Tiny (32x32, patch=4, 10 classes)
Parameters: X.XXM
Input shape: torch.Size([4, 3, 32, 32])
Output shape: torch.Size([4, 10])
✓ Vision Transformer is working!
# Default: ViT-Tiny on 32x32 images, 200 epochs
python train.py
# Use ViT-Small with larger batch
python train.py --model vit_small --batch_size 64 --epochs 100
# Resume from checkpoint
python train.py --resume checkpoints/vit_tiny_best.pth| Argument | Default | Description |
|---|---|---|
--model |
vit_tiny |
Model variant: vit_tiny, vit_small, vit_base |
--img_size |
32 |
Input image size |
--patch_size |
4 |
Patch size (4 for 32×32, 16 for 224×224) |
--epochs |
200 |
Number of training epochs |
--batch_size |
128 |
Batch size |
--lr |
3e-4 |
Base learning rate |
--weight_decay |
0.05 |
Weight decay |
--warmup_epochs |
10 |
LR warmup epochs |
--dropout |
0.1 |
Dropout rate |
--seed |
42 |
Random seed |
-
Patch Embedding: Uses a
Conv2dwithkernel_size=stride=patch_sizeto efficiently split and project patches in one operation. -
[CLS] Token: A learnable token prepended to the patch sequence; its final output serves as the image representation for classification.
-
Positional Embedding: Learnable 1D positional embeddings added to the full token sequence (including [CLS]).
-
Pre-LN Transformer: Layer normalization is applied before self-attention and MLP (pre-norm), which improves training stability.
-
Combined QKV Projection: A single linear layer computes Q, K, V together for efficiency.
-
Training Enhancements:
- AdamW optimizer with weight decay
- Linear warmup + cosine decay LR schedule
- Gradient clipping (max norm = 1.0)
- Data augmentation (RandomCrop, HorizontalFlip, ColorJitter, Rotation)
@article{dosovitskiy2020image,
title={An image is worth 16x16 words: Transformers for image recognition at scale},
author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
journal={arXiv preprint arXiv:2010.11929},
year={2020}
}Input Image -> CNN Backbone -> + 2D Pos Encoding -> Flatten
-> Transformer Encoder (6 layers)
-> Transformer Decoder (6 layers, with Object Queries)
-> Classification Head + BBox Head
-> N detections (class + box, no NMS needed!)
- Object Queries: N learnable embeddings, each forced to detect at most one object
- Hungarian Matching: Bipartite matching for one-to-one assignment (no duplicates)
- No Anchors, No NMS: Set prediction directly outputs fixed number of detections
python demo.pyGenerates a synthetic image with colored shapes, runs DETR, and saves visualization to detr_demo_output.png.
python detr.py使用预训练 DETR 模型进行视频目标检测:
# 视频文件检测
python video_apps/detr_video_demo.py --video path/to/video.mp4
# 实时摄像头检测
python video_apps/detr_video_demo.py --camera
# 指定输出路径和置信度阈值
python video_apps/detr_video_demo.py --video input.mp4 --output output.mp4 --threshold 0.7| Component | Description |
|---|---|
PositionalEncoding2D |
2D sine positional encoding for spatial features |
SimpleBackbone |
Lightweight CNN feature extractor |
Transformer |
Encoder-decoder with multi-head attention |
HungarianMatcher |
Optimal bipartite matching via scipy |
SetCriterion |
Combined loss: CE + L1 bbox + GIoU |
post_process |
Filter detections by confidence threshold |
| Feature | ViT | DETR |
|---|---|---|
| Task | Image Classification | Object Detection |
| Input | Patches (1D sequence) | CNN feature map (2D) |
| Positional Enc. | Learnable 1D | Sine 2D |
| Transformer | Encoder only | Encoder + Decoder |
| Special Tokens | [CLS] token | Object queries |
| Output | Class logits | (class, box) per query |
| Post-processing | Argmax | Confidence filter (no NMS) |
| Loss | CrossEntropy | Hungarian matching + set loss |
@article{carion2020end,
title={End-to-end object detection with transformers},
author={Carion, Nicolas and Massa, Francisco and Synnaere, Gabriel and Usunier, Nicolas and Kirillov, Alexander and Zagoruyko, Sergey},
journal={arXiv preprint arXiv:2005.12872},
year={2020}
}"Learning Transferable Visual Models From Natural Language Supervision" (Radford et al., 2021)
┌──────────────────┐ ┌─────────────────┐
│ Image (3×224×224)│ │ Text (77 tokens)│
└────────┬─────────┘ └───────┬─────────┘
│ │
┌────────▼─────────┐ ┌───────▼─────────┐
│ ViT-B/32 Encoder │ │ Transformer │
│ (768d, 12 layers)│ │ (512d, 12 lay) │
└────────┬─────────┘ └───────┬─────────┘
│ │
┌────────▼─────────┐ ┌───────▼─────────┐
│ Visual Proj │ │ Text Proj │
│ 768 → 512 │ │ 512 → 512 │
└────────┬─────────┘ └───────┬─────────┘
│ │
└──────┬────────────────┘
│
┌────────▼────────┐
│ Cosine Similarity│ + learnable temperature
│ InfoNCE Loss │
└─────────────────┘
-
Contrastive Learning: Given N (image, text) pairs, maximize similarity of matching pairs on the diagonal and minimize off-diagonal — symmetric cross-entropy loss (InfoNCE).
-
Zero-Shot Classification: No training data needed for new classes! Create prompts like "a photo of a {class}" → compute image-text similarity → softmax → class probabilities.
-
Learnable Temperature: A scalar
logit_scale(initialized to ln(100)) controls softmax sharpness. Learned during training, clamped to [0, ln(100)]. -
Causal Text Attention: Text encoder uses upper-triangle causal mask (like GPT), so each token only attends to preceding tokens.
python implementations/clip.py# Install CLIP package first
pip install git+https://github.com/openai/CLIP.git
# Run zero-shot classification on CIFAR-10
python pretrained_demos/clip_pretrained_demo.py| Variant | Patch | Vision (emb/depth/heads) | Text (emb/depth/heads) | Proj Dim | Total Params |
|---|---|---|---|---|---|
| ViT-B/32 | 32 | 768/12/12 | 512/12/8 | 512 | ~151M |
| ViT-B/16 | 16 | 768/12/12 | 512/12/8 | 512 | ~151M |
| ViT-L/14 | 14 | 1024/24/16 | 768/12/12 | 768 | ~428M |
@article{radford2021learning,
title={Learning transferable visual models from natural language supervision},
author={Radford, Alec and Kim, Jong Wook and Hallacy, Chris and Ramesh, Aditya and Goh, Gabriel and Agarwal, Sandhini and Sastry, Girish and Askell, Amanda and Mishkin, Pamela and Clark, Jack and others},
journal={arXiv preprint arXiv:2103.00020},
year={2021}
}"Denoising Diffusion Probabilistic Models" (Ho et al., 2020) Also implements DDIM accelerated sampling (Song et al., 2021)
Forward Process (fixed, no learning):
x₀ → x₁ → x₂ → ... → xₜ → ... → x_T (pure Gaussian noise)
q(xₜ | x₀) = N(xₜ; √ᾱₜ · x₀, (1-ᾱₜ) · I)
Reverse Process (learned via U-Net):
x_T → xₜ → ... → x₁ → x₀ (clean image)
p(xₜ₋₁ | xₜ) uses ε_θ(xₜ, t) to predict the noise
U-Net Architecture:
┌─────────────────────────────────────────────┐
│ Noisy Image xₜ + Timestep t (sinusoidal) │
│ │ │ │
│ ┌────▼────┐ ┌─────▼─────┐ │
│ │ Conv In │ │ Time MLP │ │
│ └────┬────┘ └─────┬─────┘ │
│ ┌────▼──────────────▼────────┐ │
│ │ Encoder (DownBlocks) │ │
│ │ [ResBlock + Attn] × 4 │ │
│ └────────────┬───────────────┘ │
│ ┌────────────▼───────────────┐ │
│ │ Bottleneck (Res+Attn+Res) │ │
│ └────────────┬───────────────┘ │
│ ┌────────────▼───────────────┐ │
│ │ Decoder (UpBlocks) │ │
│ │ [ResBlock + Attn] × 4 │ ← skip conns│
│ └────────────┬───────────────┘ │
│ ┌────────────▼───────────────┐ │
│ │ Conv Out → predicted ε │ │
│ └────────────────────────────┘ │
└─────────────────────────────────────────────┘
-
Forward Process: Gradually add Gaussian noise to an image over T steps. Can jump to any timestep t directly via closed-form:
xₜ = √ᾱₜ · x₀ + √(1-ᾱₜ) · ε. -
Reverse Process: Learn to remove noise step by step. The U-Net predicts the noise ε at each timestep. The mean is computed analytically from the predicted noise.
-
Training Loss: Simply predict the noise!
L = E[||ε - ε_θ(xₜ, t)||²]. No adversarial training, no complex objectives. -
Noise Schedule: Linear (DDPM original) or Cosine (Improved DDPM). Cosine schedule preserves signal longer and works better in practice.
-
DDIM Sampling: Deterministic variant that can skip timesteps (50 instead of 1000), making generation 20× faster with minimal quality loss.
python implementations/diffusion.py# Quick test (10 epochs)
python training/train_diffusion.py --epochs 10
# Longer training
python training/train_diffusion.py --epochs 100 --lr 2e-4
# Generate images from trained model
python training/train_diffusion.py --generate_only --resume checkpoints/ddpm_best.pthpip install diffusers transformers accelerate
python pretrained_demos/diffusion_pretrained_demo.py@article{ho2020denoising,
title={Denoising diffusion probabilistic models},
author={Ho, Jonathan and Jain, Ajay and Abbeel, Pieter},
journal={arXiv preprint arXiv:2006.11239},
year={2020}
}
@article{song2021denoising,
title={Denoising diffusion implicit models},
author={Song, Jiaming and Meng, Chenlin and Ermon, Stefano},
journal={arXiv preprint arXiv:2010.02502},
year={2021}
}"High-Resolution Image Synthesis with Latent Diffusion Models" (Rombach et al., 2022)
┌──────────┐ ┌──────────────────────┐ ┌──────────┐
│ VAE │ │ U-Net with │ │ CLIP │
│ Encoder │──→ │ Cross-Attention │←── │ Text │
│ (x → z) │ │ ε_θ(z_t, t, c) │ │ Enc(c) │
└──────────┘ └──────────────────────┘ └──────────┘
│ │
│ ▼
│ ┌──────────┐
│ │ VAE │
└──────────→│ Decoder │──→ Generated Image
│ (z → x) │
└──────────┘
DDPM vs Stable Diffusion:
DDPM: Diffusion in PIXEL space (3, 512, 512) → 786K dims
SD: Diffusion in LATENT space (4, 64, 64) → 16K dims (48× fewer!)
DDPM U-Net: No text input → generates random images
SD U-Net: Cross-Attn(text) → generates what you describe
DDPM: 1000 steps (slow)
SD: 50 steps + CFG (fast + controllable)
| # | Improvement | What it does | Why it matters |
|---|---|---|---|
| 1 | Latent Diffusion (VAE) | Compress images 8× spatially (512→64) | 48× fewer dims → much faster diffusion |
| 2 | Text Conditioning (Cross-Attention) | U-Net features attend to text tokens | Control what gets generated |
| 3 | Classifier-Free Guidance | Amplify text signal: ε = ε_uncond + s·(ε_cond - ε_uncond) |
Higher quality + better text adherence |
-
VAE (Variational Autoencoder): Pre-trained separately. Encoder compresses images to 4-channel latents (8× spatial downsampling). Decoder reconstructs latents back to images. Uses KL regularization to keep latent space smooth.
-
Cross-Attention in U-Net: At every resolution level, spatial features (Q) attend to text embeddings (K, V). This allows each pixel region to "decide" which text tokens are relevant. Example: when generating "a red cat", the cat region attends strongly to the "red cat" tokens.
-
Classifier-Free Guidance (CFG):
- Training: Randomly drop text 10% of the time → model learns both conditional and unconditional generation
- Inference:
ε_guided = ε_uncond + guidance_scale × (ε_cond - ε_uncond) guidance_scale=1.0→ no amplification (unguided)guidance_scale=7.5→ SD default (strong text following)guidance_scale=15+→ too much → artifacts
-
Latent Space Scaling: SD uses
z_scaled = 0.18215 × z_vaeto normalize the latent distribution before diffusion.
python implementations/stable_diffusion.pypip install diffusers transformers accelerate
python pretrained_demos/stable_diffusion_pretrained_demo.py| Component | Params | Details |
|---|---|---|
| VAE | 83.7M | 4-channel latent, 8× spatial compression |
| Text Encoder (CLIP-ViT/L) | 123.7M | 77 tokens, 768-dim |
| U-Net | 860.9M | 320 base channels, 4 levels, cross-attention |
| Total | ~1.07B |
| Feature | DDPM | Stable Diffusion |
|---|---|---|
| Diffusion Space | Pixel (3, 512, 512) | Latent (4, 64, 64) |
| Dimensions | 786,432 | 16,384 (48× less) |
| Text Control | ❌ None | ✅ CLIP + Cross-Attention |
| Quality Control | ❌ None | ✅ Classifier-Free Guidance |
| U-Net Attention | Deep levels only | ALL levels (text needs every scale) |
| Typical Steps | 1000 (DDPM) | 50 (DDIM) |
| Model Size | ~114M | ~1.07B |
@article{rombach2022high,
title={High-resolution image synthesis with latent diffusion models},
author={Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj{\"o}rn},
journal={CVPR},
year={2022}
}
@article{ho2022classifierfree,
title={Classifier-free diffusion guidance},
author={Ho, Jonathan and Salakhans, Tim},
journal={NeurIPS Workshop},
year={2022}
}"Occupancy Networks: Learning 3D Reconstruction in Function Space" (Mescheder et al., 2019)
┌───────────────────────────────────────────────────────────────┐
│ Occupancy Network │
│ │
│ Input Image ──→ CNN Encoder ──→ Global Feature Vector c │
│ │
│ Query Point p = (x, y, z) ──→ ┐ │
│ │ │
│ ┌─────────────▼──────────────┐ │
│ │ Conditional MLP │ │
│ │ f(p, c) → occupancy [0,1] │ │
│ └────────────────────────────┘ │
│ │
│ Any 3D point p + image feature c → occupied probability │
└───────────────────────────────────────────────────────────────┘
Key Idea: Replace discrete voxel grids (O(N³) memory) with a
continuous function f: R³ → [0,1]. The neural network IS the
3D representation — query at ANY resolution, O(1) memory.
-
Continuous Occupancy Function: Instead of storing discrete voxels, learn a function f(p, c) that maps any 3D point to an occupancy probability. This enables theoretically infinite resolution 3D reconstruction.
-
Conditional MLP: The occupancy predictor is a shared MLP that takes (encoded 3D point, image feature) → [0, 1]. All query points share the same network — just like NeRF's implicit representation.
-
Three Conditioning Variants:
- Simple Concat: Concatenate [point, c] at MLP input (paper default)
- FiLM: Use c to modulate scale (γ) and shift (β) at every layer:
h' = γ(c) ⊙ h + β(c) - Concat+Skip: Inject c at every layer for strongest conditioning
-
Sinusoidal Positional Encoding: 3D coordinates are encoded with multi-frequency sinusoids (like NeRF), allowing the MLP to capture fine spatial detail.
-
Importance Sampling: Training points are sampled densely near the object surface (where the decision boundary is) and sparsely far away, for efficient and sharp boundary learning.
-
Mesh Extraction: At inference, query f(p, c) on a dense 3D grid and run Marching Cubes to extract the isosurface at 0.5.
python3 implementations/occupancy_network.pyThe smoke test includes a 300-iteration training demo that teaches the network to reconstruct a unit sphere, reaching ~97% accuracy.
| Variant | Cond. Method | Hidden | Layers | Params |
|---|---|---|---|---|
| Concat (default) | Input concat | 512 | 5 | ~1.4M |
| FiLM | Feature-wise modulation | 512 | 5 | ~3.1M |
| Concat+Skip | Every-layer concat | 256 | 8 | ~1.5M |
| Method | Resolution | Memory | Topology |
|---|---|---|---|
| Voxel (32³) | 32³ fixed | 32K values | Arbitrary |
| Voxel (256³) | 256³ fixed | 16M values | Arbitrary |
| Point Cloud | N points | O(N) | Needs postproc |
| Mesh (template) | F faces | O(F) | Fixed template |
| Occupancy Net | Infinite | O(1) weights | Arbitrary |
@inproceedings{mescheder2019occupancy,
title={Occupancy networks: Learning 3d reconstruction in function space},
author={Mescheder, Lars and Oechsle, Michael and Niemeyer, Michael and Nowozin, Sebastian and Geiger, Andreas},
booktitle={CVPR},
year={2019}
}"Scalable Diffusion Models with Transformers" (Peebles & Xie, 2023)
┌──────────────────────────────────────────────────────────────────────┐
│ DiT Architecture (Latent Diffusion) │
│ │
│ Noisy Latent zₜ (B, 4, 32, 32) ← from VAE encoder │
│ │ │
│ ┌────▼────────┐ │
│ │ Patchify │ Conv2d(4→D, kernel=P, stride=P) │
│ │ P=2: 32→16 │ 16×16 = 256 patches of dim D │
│ └────┬────────┘ │
│ │ │
│ ┌────▼─────────────────────────────┐ │
│ │ + 2D Sinusoidal Position Embed │ (fixed, not learnable) │
│ └────┬─────────────────────────────┘ │
│ │ │
│ ┌────▼────┐ ┌──────────┐ ┌──────────┐ │
│ │ Time t │ │ Class y │ │ c = t+y │ ← conditioning │
│ │ Embedder │ │ Embedder │ │ (add) │ │
│ └────┬────┘ └─────┬────┘ └─────┬────┘ │
│ │ │ │ │
│ ┌────▼──────────────▼──────────────▼────────┐ │
│ │ DiTBlock × depth (with adaLN-Zero) │ │
│ │ │ │
│ │ c → SiLU → Linear → 6·D params │ │
│ │ [shift_msa, scale_msa, gate_msa, │ │
│ │ shift_mlp, scale_mlp, gate_mlp] │ │
│ │ │ │
│ │ x = x + gate_msa · Attn(adaLN(Norm(x))) │ │
│ │ x = x + gate_mlp · MLP(adaLN(Norm(x))) │ │
│ └──────────────────┬───────────────────────┘ │
│ ┌──────────────────▼───────────────────────┐ │
│ │ FinalLayer → Unpatchify → (B, C, H, W) │ │
│ └──────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────┘
-
Transformer replaces U-Net: DiT replaces the standard U-Net backbone with a Transformer that operates on latent patches. Same block repeated N times — no encoder-decoder asymmetry, no skip connections.
-
adaLN-Zero Conditioning: The core innovation. Conditioning signal c (timestep + class) modulates LayerNorm parameters through adaptive scale and shift. Additionally, residual connections are gated by c, with gates initialized to zero — so the network starts as identity and gradually learns to use each block.
-
Patchify & Unpatchify: Conv2d converts the latent grid into a sequence of patch tokens (like ViT). After the Transformer, the output tokens are rearranged back to the spatial grid. Smaller patch size = more tokens = more compute = better quality.
-
Classifier-Free Guidance (CFG): During training, class labels are randomly dropped 10% → null class embedding. During inference:
ε = ε_uncond + s × (ε_cond - ε_uncond)with s > 1. -
Scalability: More Gflops (through depth, width, or more patches) consistently yields lower FID. DiT-XL/2 achieves FID 2.27 on ImageNet 256×256 (SOTA at publication).
python implementations/dit.py| Model | Depth | Hidden | Heads | Patch | Params | Gflops |
|---|---|---|---|---|---|---|
| DiT-S/2 | 12 | 384 | 6 | 2 | ~33M | 0.6 |
| DiT-S/4 | 12 | 384 | 6 | 4 | ~33M | 0.2 |
| DiT-S/8 | 12 | 384 | 6 | 8 | ~33M | 0.1 |
| DiT-B/2 | 12 | 768 | 12 | 2 | ~131M | 4.0 |
| DiT-B/4 | 12 | 768 | 12 | 4 | ~130M | 1.3 |
| DiT-B/8 | 12 | 768 | 12 | 8 | ~131M | 0.5 |
| DiT-L/2 | 24 | 1024 | 16 | 2 | ~458M | 20.2 |
| DiT-L/4 | 24 | 1024 | 16 | 4 | ~458M | 6.1 |
| DiT-L/8 | 24 | 1024 | 16 | 8 | ~459M | 2.4 |
| DiT-XL/2 | 28 | 1152 | 16 | 2 | ~675M | 34.2 |
| DiT-XL/4 | 28 | 1152 | 16 | 4 | ~675M | 10.3 |
| DiT-XL/8 | 28 | 1152 | 16 | 8 | ~676M | 4.1 |
| Strategy | How c enters | FID ↓ | Notes |
|---|---|---|---|
| In-context (concat) | c as extra token | ~70 | Weak modulation |
| Cross-attention | c as K,V in cross-attn | ~50 | Extra attention cost |
| adaLN-Zero ★ | c modulates LayerNorm + gates | ~30 | Best FID, simplest |
| Property | U-Net (SD 1.x) | DiT |
|---|---|---|
| Architecture | Encoder-Bottleneck-Decoder + skip connections | N identical blocks |
| Conditioning | Time → ResBlock, Class → Cross-Attention | Time+Class → adaLN-Z |
| Skip connections | Yes (critical) | No (not needed) |
| Scalability | Complex (multi-scale) | Simple (just N×block) |
| Positional encoding | Not needed | 2D sincos (required) |
DiT is the backbone behind many modern diffusion models:
- Stable Diffusion 3 (MMDiT, multi-modal DiT variant)
- FLUX.1 (Black Forest Labs)
- Sora (OpenAI video generation)
- PixArt-Σ / PixArt-α
- Hunyuan-DiT (Tencent)
@article{peebles2023scalable,
title={Scalable Diffusion Models with Transformers},
author={Peebles, William and Xie, Saining},
journal={ICCV},
year={2023}
}"Auto-Encoding Variational Bayes" (Kingma & Welling, 2014)
┌───────────────────────────────────────────────────────────┐
│ Variational Autoencoder │
│ │
│ Input x ──→ Encoder ──→ μ, log σ² ──→ z = μ + σ·ε │
│ │ │
│ ↓ │
│ Reconstructed x̂ ←── Decoder(z) │
│ │
│ Loss = Reconstruction + β × KL(N(μ,σ²) ‖ N(0,1)) │
└───────────────────────────────────────────────────────────┘
Regular AE vs VAE:
AE: Encoder(x) → z (deterministic) → Decoder → x̂
VAE: Encoder(x) → μ, σ² (distribution) → z~N(μ,σ²) → Decoder → x̂
Reparameterization Trick (makes sampling differentiable):
z = μ + σ · ε, where ε ~ N(0,1)
Gradients flow through μ and σ (deterministic paths)
ε is "noise" from outside the computation graph
-
Reparameterization Trick: Sampling
z ~ N(μ, σ²)is not differentiable. Solution:z = μ + σ · εwhereε ~ N(0,1). Now gradients flow through μ and σ. -
KL Divergence Regularizer:
D_KL(N(μ,σ²) ‖ N(0,1)) = ½ Σ(σ² + μ² - 1 - log σ²). Forces latent distributions to stay near standard normal, making the latent space smooth and continuous. -
β-VAE: Adjustable KL weight. β=1 is standard VAE; β>1 encourages disentangled representations (each latent dimension captures an independent factor).
-
Why VAE matters for diffusion: SD and DiT operate in VAE latent space (4×64×64 instead of 3×512×512). The VAE's KL regularization ensures the latent space is smooth enough for diffusion to work. Scaling factor
z_scaled = 0.18215 × znormalizes for the noise schedule.
python implementations/vae.py| Variant | Encoder/Decoder | Input | Latent | Use Case |
|---|---|---|---|---|
| Vanilla VAE (MLP) | Linear + ReLU | (B, 784) | (B, 20) | MNIST / flat data |
| Conv VAE (CNN) | Conv / ConvT | (B, 1, 28, 28) | (B, 64) | Image generation |
| SD VAE (ResBlock) | ResBlock + GroupNorm | (B, 3, 256, 256) | (B, 4, 32, 32) | Latent Diffusion |
@article{kingma2014auto,
title={Auto-Encoding Variational Bayes},
author={Kingma, Diederik P and Welling, Max},
journal={ICLR},
year={2014}
}
@article{higgins2017betavae,
title={$\beta$-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework},
author={Higgins, Irina and others},
journal={ICLR},
year={2017}
}implementations/moe.py implements the core designs of modern MoE architectures, from Switch Transformer to DeepSeek-V3.
Decouple parameter count from compute cost: a dense FFN is replaced by N expert FFNs, and each token is routed to only the top-K experts by a lightweight router. The model has N× the parameters but only K× the FFN compute per token.
logits = W_r · x # router scores, (T, N)
gate = Softmax(TopK(logits)) # sparse gate weights, renormalized
y = Σ_{i ∈ TopK} gate_i · Expert_i(x) + SharedExpert(x)
| Component | Class / Mechanism | Source |
|---|---|---|
| Sparse routing | MoELayer top-K gating + weight renormalization |
Shazeer 2017, GShard |
| Expert FFN | ExpertFFN (Linear → GELU → Linear) |
standard Transformer FFN |
| Load-balance loss | aux_loss(): L = N · Σ f_i · P_i |
Switch Transformer |
| Capacity & token drop | capacity_factor, overflow tokens skip MoE (kept by residual) |
Switch/GShard |
| Shared experts | num_shared_experts > 0, always active for common knowledge |
DeepSeekMoE |
| Aux-loss-free balancing | expert_bias buffer, update_expert_bias(): b_i += u · sign(1/N − f_i) |
DeepSeek-V3 |
| Full block | MoETransformerBlock (pre-norm attention + MoE FFN) |
Mixtral/DeepSeek style |
- Selection vs. weighting: with aux-free balancing, the bias is added to router logits only for expert selection; the gate weights used for combining always come from unbiased softmax probabilities, so the main task gradient is never disturbed.
- Aux loss gradient path:
f_i(token fraction) is a detached discrete count; gradients flow to the router only throughP_i(mean route probability) — exactly as in Switch Transformer. - Token dropping: when an expert exceeds its capacity
ceil(T·K/N × capacity_factor), overflow tokens receive zero MoE output; the block's residual connection preserves their information. - Trade-off: MoE saves FLOPs, not memory — all expert parameters must be loaded for inference.
python implementations/moe.pyCovers: (1) Switch-style layer with aux loss, (2) gradient flow to router + experts, (3) DeepSeek-style bias rebalancing from a 95.7%→13.7% skewed load, (4) full MoE transformer block.
@article{shazeer2017outrageously,
title={Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer},
author={Shazeer, Noam and others},
journal={ICLR},
year={2017}
}
@article{fedus2021switch,
title={Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity},
author={Fedus, William and Zoph, Barret and Shazeer, Noam},
journal={JMLR},
year={2022}
}
@article{deepseekv3,
title={DeepSeek-V3 Technical Report},
author={DeepSeek-AI},
journal={arXiv:2412.19437},
year={2024}
}