Skip to content
happyerPublic

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

Repository files navigation

Computer Vision — From Scratch Implementations

Clean PyTorch implementations of landmark vision models, organized by task:

Task Models Key Idea
Image Classification ViT Patch → Transformer → [CLS] classification
Object Detection DETR, Deformable DETR, DINO, RT-DETR, YOLO Set prediction / anchor-free / real-time
Image Segmentation SAM, SAM2 Promptable segmentation (point/box/mask)
Video Tracking SAM2 Memory-guided object tracking in video
3D Reconstruction Occupancy Network Image → continuous occupancy function → infinite-res mesh
BEV Perception BEVFormer, TinyBEV Multi-view camera → BEV → Detection + Forecasting + Planning
BEV Perception TinyBEV Multi-view camera → LSS BEV → Detection + Forecasting + Planning
BEV Perception TinyBEV Multi-view camera → LSS BEV → Detection + Forecasting + Planning
Image Generation DDPM, Stable Diffusion, DiT Forward noise → U-Net/Transformer predicts noise → reverse denoise
Generative Model VAE Reparameterization trick + KL → smooth latent space → generation
Multimodal CLIP Contrastive image-text alignment → zero-shot transfer

Architecture Overview

Input Image (H × W × C)
       │
       ▼
┌──────────────────┐
│  Patch Embedding │   Conv2d: split image into P×P patches, project to D dims
└──────────────────┘
       │
       ▼
┌──────────────────┐
│   + [CLS] Token  │   Prepend a learnable class token
│   + Pos Embed    │   Add learnable positional embeddings
└──────────────────┘
       │
       ▼
┌──────────────────┐
│  Transformer ×L  │   L blocks of: LayerNorm → MSA → Res → LayerNorm → MLP → Res
└──────────────────┘
       │
       ▼
┌──────────────────┐
│  Layer Norm      │
│  [CLS] → Head    │   Linear classification head on the [CLS] token
└──────────────────┘
       │
       ▼
   Class Logits

Files

cv/
├── implementations/           # From-scratch model implementations
│   ├── vit.py                  # Vision Transformer
│   ├── detr.py                 # DETR (base)
│   ├── deformable_detr.py      # Deformable DETR (multi-scale deformable attention)
│   ├── dino.py                 # DINO (de-noising training for DETR)
│   ├── rt_detr.py              # RT-DETR (real-time DETR)
│   ├── sam.py                  # SAM (Segment Anything Model)
│   ├── sam2_from_scratch.py    # SAM 2 (video tracking)
│   ├── yolo_simplified.py      # YOLO (simplified single-shot detector)
│   ├── clip.py                 # CLIP (contrastive language-image pre-training)
│   ├── diffusion.py            # DDPM (denoising diffusion probabilistic models)
│   ├── dit.py                  # DiT (Diffusion Transformer, adaLN-Zero conditioning)
│   ├── vae.py                  # VAE (Variational Autoencoder, reparameterization trick)
│   ├── stable_diffusion.py     # Stable Diffusion (latent diffusion + text conditioning)
│   ├── occupancy_network.py   # Occupancy Network (implicit 3D reconstruction)
│   ├── bev_former.py          # BEVFormer (spatiotemporal BEV transformer)
│   ├── tiny_bev.py            # TinyBEV (lightweight BEV perception + planning)
│   └── cnn_test.py            # SimpleCNN with image visualization
├── pretrained_demos/           # Demos using official pretrained weights
│   ├── detr_pretrained_demo.py
│   ├── sam_pretrained_demo.py
│   ├── sam2_pretrained_demo.py
│   ├── yolo_demo.py
│   ├── yolo_pretrained_demo.py
│   ├── yolo_real_demo.py
│   ├── clip_pretrained_demo.py  # CLIP zero-shot classification & retrieval
│   ├── stable_diffusion_pretrained_demo.py  # SD 1.5 text-to-image generation
│   └── tiny_bev_demo.py       # TinyBEV synthetic demo
├── video_apps/                 # Video application demos
│   ├── detr_video_demo.py       # DETR video object detection
│   └── sam2_scratch_video_demo.py
├── training/                   # Training scripts
│   ├── train.py                # ViT training on CIFAR-10
│   └── demo.py
├── weights/                    # Pretrained model weights
├── data/                       # Datasets
└── outputs/                    # Demo output images

Model Variants

Variant Emb Size Depth Heads Params (224px)
ViT-Tiny 192 12 3 ~5.7M
ViT-Small 384 12 6 ~22M
ViT-Base 768 12 12 ~86M
ViT-Large 1024 24 16 ~307M
ViT-Huge 1280 32 16 ~632M

Quick Start

Install dependencies

pip install -r requirements.txt

Smoke test

python vit.py

Expected output:

Model: ViT-Tiny (32x32, patch=4, 10 classes)
Parameters: X.XXM
Input shape:  torch.Size([4, 3, 32, 32])
Output shape: torch.Size([4, 10])
✓ Vision Transformer is working!

Train on CIFAR-10

# Default: ViT-Tiny on 32x32 images, 200 epochs
python train.py

# Use ViT-Small with larger batch
python train.py --model vit_small --batch_size 64 --epochs 100

# Resume from checkpoint
python train.py --resume checkpoints/vit_tiny_best.pth

Training Arguments

Argument Default Description
--model vit_tiny Model variant: vit_tiny, vit_small, vit_base
--img_size 32 Input image size
--patch_size 4 Patch size (4 for 32×32, 16 for 224×224)
--epochs 200 Number of training epochs
--batch_size 128 Batch size
--lr 3e-4 Base learning rate
--weight_decay 0.05 Weight decay
--warmup_epochs 10 LR warmup epochs
--dropout 0.1 Dropout rate
--seed 42 Random seed

Key Implementation Details

  1. Patch Embedding: Uses a Conv2d with kernel_size=stride=patch_size to efficiently split and project patches in one operation.

  2. [CLS] Token: A learnable token prepended to the patch sequence; its final output serves as the image representation for classification.

  3. Positional Embedding: Learnable 1D positional embeddings added to the full token sequence (including [CLS]).

  4. Pre-LN Transformer: Layer normalization is applied before self-attention and MLP (pre-norm), which improves training stability.

  5. Combined QKV Projection: A single linear layer computes Q, K, V together for efficiency.

  6. Training Enhancements:

    • AdamW optimizer with weight decay
    • Linear warmup + cosine decay LR schedule
    • Gradient clipping (max norm = 1.0)
    • Data augmentation (RandomCrop, HorizontalFlip, ColorJitter, Rotation)

Reference

@article{dosovitskiy2020image,
  title={An image is worth 16x16 words: Transformers for image recognition at scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={arXiv preprint arXiv:2010.11929},
  year={2020}
}

DETR: End-to-End Object Detection with Transformers

Architecture Overview

Input Image -> CNN Backbone -> + 2D Pos Encoding -> Flatten
    -> Transformer Encoder (6 layers)
    -> Transformer Decoder (6 layers, with Object Queries)
    -> Classification Head + BBox Head
    -> N detections (class + box, no NMS needed!)

Key Design

  • Object Queries: N learnable embeddings, each forced to detect at most one object
  • Hungarian Matching: Bipartite matching for one-to-one assignment (no duplicates)
  • No Anchors, No NMS: Set prediction directly outputs fixed number of detections

DETR Demo

python demo.py

Generates a synthetic image with colored shapes, runs DETR, and saves visualization to detr_demo_output.png.

DETR Smoke Test

python detr.py

DETR Video Detection Demo

使用预训练 DETR 模型进行视频目标检测:

# 视频文件检测
python video_apps/detr_video_demo.py --video path/to/video.mp4

# 实时摄像头检测
python video_apps/detr_video_demo.py --camera

# 指定输出路径和置信度阈值
python video_apps/detr_video_demo.py --video input.mp4 --output output.mp4 --threshold 0.7

DETR Components

Component Description
PositionalEncoding2D 2D sine positional encoding for spatial features
SimpleBackbone Lightweight CNN feature extractor
Transformer Encoder-decoder with multi-head attention
HungarianMatcher Optimal bipartite matching via scipy
SetCriterion Combined loss: CE + L1 bbox + GIoU
post_process Filter detections by confidence threshold

ViT vs DETR Comparison

Feature ViT DETR
Task Image Classification Object Detection
Input Patches (1D sequence) CNN feature map (2D)
Positional Enc. Learnable 1D Sine 2D
Transformer Encoder only Encoder + Decoder
Special Tokens [CLS] token Object queries
Output Class logits (class, box) per query
Post-processing Argmax Confidence filter (no NMS)
Loss CrossEntropy Hungarian matching + set loss

DETR Reference

@article{carion2020end,
  title={End-to-end object detection with transformers},
  author={Carion, Nicolas and Massa, Francisco and Synnaere, Gabriel and Usunier, Nicolas and Kirillov, Alexander and Zagoruyko, Sergey},
  journal={arXiv preprint arXiv:2005.12872},
  year={2020}
}

CLIP: Contrastive Language-Image Pre-training

"Learning Transferable Visual Models From Natural Language Supervision" (Radford et al., 2021)

Architecture Overview

┌──────────────────┐     ┌─────────────────┐
│  Image (3×224×224)│     │  Text (77 tokens)│
└────────┬─────────┘     └───────┬─────────┘
         │                       │
┌────────▼─────────┐     ┌───────▼─────────┐
│  ViT-B/32 Encoder │     │  Transformer    │
│  (768d, 12 layers)│     │  (512d, 12 lay) │
└────────┬─────────┘     └───────┬─────────┘
         │                       │
┌────────▼─────────┐     ┌───────▼─────────┐
│  Visual Proj      │     │  Text Proj       │
│  768 → 512        │     │  512 → 512       │
└────────┬─────────┘     └───────┬─────────┘
         │                       │
         └──────┬────────────────┘
                │
       ┌────────▼────────┐
       │  Cosine Similarity│  + learnable temperature
       │  InfoNCE Loss     │
       └─────────────────┘

Key Concepts

  1. Contrastive Learning: Given N (image, text) pairs, maximize similarity of matching pairs on the diagonal and minimize off-diagonal — symmetric cross-entropy loss (InfoNCE).

  2. Zero-Shot Classification: No training data needed for new classes! Create prompts like "a photo of a {class}" → compute image-text similarity → softmax → class probabilities.

  3. Learnable Temperature: A scalar logit_scale (initialized to ln(100)) controls softmax sharpness. Learned during training, clamped to [0, ln(100)].

  4. Causal Text Attention: Text encoder uses upper-triangle causal mask (like GPT), so each token only attends to preceding tokens.

CLIP Smoke Test

python implementations/clip.py

CLIP Pretrained Demo

# Install CLIP package first
pip install git+https://github.com/openai/CLIP.git

# Run zero-shot classification on CIFAR-10
python pretrained_demos/clip_pretrained_demo.py

CLIP Variants

Variant Patch Vision (emb/depth/heads) Text (emb/depth/heads) Proj Dim Total Params
ViT-B/32 32 768/12/12 512/12/8 512 ~151M
ViT-B/16 16 768/12/12 512/12/8 512 ~151M
ViT-L/14 14 1024/24/16 768/12/12 768 ~428M

CLIP Reference

@article{radford2021learning,
  title={Learning transferable visual models from natural language supervision},
  author={Radford, Alec and Kim, Jong Wook and Hallacy, Chris and Ramesh, Aditya and Goh, Gabriel and Agarwal, Sandhini and Sastry, Girish and Askell, Amanda and Mishkin, Pamela and Clark, Jack and others},
  journal={arXiv preprint arXiv:2103.00020},
  year={2021}
}

DDPM: Denoising Diffusion Probabilistic Models

"Denoising Diffusion Probabilistic Models" (Ho et al., 2020) Also implements DDIM accelerated sampling (Song et al., 2021)

Architecture Overview

Forward Process (fixed, no learning):
  x₀ → x₁ → x₂ → ... → xₜ → ... → x_T  (pure Gaussian noise)
  q(xₜ | x₀) = N(xₜ; √ᾱₜ · x₀, (1-ᾱₜ) · I)

Reverse Process (learned via U-Net):
  x_T → xₜ → ... → x₁ → x₀  (clean image)
  p(xₜ₋₁ | xₜ) uses ε_θ(xₜ, t) to predict the noise

U-Net Architecture:
  ┌─────────────────────────────────────────────┐
  │  Noisy Image xₜ + Timestep t (sinusoidal)   │
  │         │              │                     │
  │    ┌────▼────┐   ┌─────▼─────┐               │
  │    │ Conv In  │   │ Time MLP  │               │
  │    └────┬────┘   └─────┬─────┘               │
  │    ┌────▼──────────────▼────────┐            │
  │    │  Encoder (DownBlocks)      │            │
  │    │  [ResBlock + Attn] × 4     │            │
  │    └────────────┬───────────────┘            │
  │    ┌────────────▼───────────────┐            │
  │    │  Bottleneck (Res+Attn+Res) │            │
  │    └────────────┬───────────────┘            │
  │    ┌────────────▼───────────────┐            │
  │    │  Decoder (UpBlocks)        │            │
  │    │  [ResBlock + Attn] × 4    │ ← skip conns│
  │    └────────────┬───────────────┘            │
  │    ┌────────────▼───────────────┐            │
  │    │  Conv Out → predicted ε    │            │
  │    └────────────────────────────┘            │
  └─────────────────────────────────────────────┘

Key Concepts

  1. Forward Process: Gradually add Gaussian noise to an image over T steps. Can jump to any timestep t directly via closed-form: xₜ = √ᾱₜ · x₀ + √(1-ᾱₜ) · ε.

  2. Reverse Process: Learn to remove noise step by step. The U-Net predicts the noise ε at each timestep. The mean is computed analytically from the predicted noise.

  3. Training Loss: Simply predict the noise! L = E[||ε - ε_θ(xₜ, t)||²]. No adversarial training, no complex objectives.

  4. Noise Schedule: Linear (DDPM original) or Cosine (Improved DDPM). Cosine schedule preserves signal longer and works better in practice.

  5. DDIM Sampling: Deterministic variant that can skip timesteps (50 instead of 1000), making generation 20× faster with minimal quality loss.

Diffusion Smoke Test

python implementations/diffusion.py

Train DDPM on CIFAR-10

# Quick test (10 epochs)
python training/train_diffusion.py --epochs 10

# Longer training
python training/train_diffusion.py --epochs 100 --lr 2e-4

# Generate images from trained model
python training/train_diffusion.py --generate_only --resume checkpoints/ddpm_best.pth

Pretrained Demo (diffusers)

pip install diffusers transformers accelerate
python pretrained_demos/diffusion_pretrained_demo.py

DDPM Reference

@article{ho2020denoising,
  title={Denoising diffusion probabilistic models},
  author={Ho, Jonathan and Jain, Ajay and Abbeel, Pieter},
  journal={arXiv preprint arXiv:2006.11239},
  year={2020}
}

@article{song2021denoising,
  title={Denoising diffusion implicit models},
  author={Song, Jiaming and Meng, Chenlin and Ermon, Stefano},
  journal={arXiv preprint arXiv:2010.02502},
  year={2021}
}

Stable Diffusion: Latent Diffusion with Text Conditioning

"High-Resolution Image Synthesis with Latent Diffusion Models" (Rombach et al., 2022)

Architecture Overview

┌──────────┐     ┌──────────────────────┐     ┌──────────┐
│  VAE      │     │  U-Net with          │     │  CLIP    │
│  Encoder  │──→  │  Cross-Attention     │←──  │  Text    │
│  (x → z) │     │  ε_θ(z_t, t, c)      │     │  Enc(c)  │
└──────────┘     └──────────────────────┘     └──────────┘
       │                   │
       │                   ▼
       │            ┌──────────┐
       │            │  VAE     │
       └──────────→│  Decoder │──→ Generated Image
                    │  (z → x) │
                    └──────────┘

DDPM vs Stable Diffusion:
  DDPM: Diffusion in PIXEL space (3, 512, 512) → 786K dims
  SD:   Diffusion in LATENT space (4, 64, 64)  → 16K dims  (48× fewer!)

  DDPM U-Net:  No text input → generates random images
  SD   U-Net:  Cross-Attn(text) → generates what you describe

  DDPM: 1000 steps (slow)
  SD:   50 steps + CFG (fast + controllable)

Three Key Improvements over DDPM

# Improvement What it does Why it matters
1 Latent Diffusion (VAE) Compress images 8× spatially (512→64) 48× fewer dims → much faster diffusion
2 Text Conditioning (Cross-Attention) U-Net features attend to text tokens Control what gets generated
3 Classifier-Free Guidance Amplify text signal: ε = ε_uncond + s·(ε_cond - ε_uncond) Higher quality + better text adherence

Key Concepts

  1. VAE (Variational Autoencoder): Pre-trained separately. Encoder compresses images to 4-channel latents (8× spatial downsampling). Decoder reconstructs latents back to images. Uses KL regularization to keep latent space smooth.

  2. Cross-Attention in U-Net: At every resolution level, spatial features (Q) attend to text embeddings (K, V). This allows each pixel region to "decide" which text tokens are relevant. Example: when generating "a red cat", the cat region attends strongly to the "red cat" tokens.

  3. Classifier-Free Guidance (CFG):

    • Training: Randomly drop text 10% of the time → model learns both conditional and unconditional generation
    • Inference: ε_guided = ε_uncond + guidance_scale × (ε_cond - ε_uncond)
    • guidance_scale=1.0 → no amplification (unguided)
    • guidance_scale=7.5 → SD default (strong text following)
    • guidance_scale=15+ → too much → artifacts
  4. Latent Space Scaling: SD uses z_scaled = 0.18215 × z_vae to normalize the latent distribution before diffusion.

Stable Diffusion Smoke Test

python implementations/stable_diffusion.py

Pretrained Demo (SD 1.5)

pip install diffusers transformers accelerate
python pretrained_demos/stable_diffusion_pretrained_demo.py

SD 1.5 Model Components

Component Params Details
VAE 83.7M 4-channel latent, 8× spatial compression
Text Encoder (CLIP-ViT/L) 123.7M 77 tokens, 768-dim
U-Net 860.9M 320 base channels, 4 levels, cross-attention
Total ~1.07B

DDPM vs Stable Diffusion Comparison

Feature DDPM Stable Diffusion
Diffusion Space Pixel (3, 512, 512) Latent (4, 64, 64)
Dimensions 786,432 16,384 (48× less)
Text Control ❌ None ✅ CLIP + Cross-Attention
Quality Control ❌ None ✅ Classifier-Free Guidance
U-Net Attention Deep levels only ALL levels (text needs every scale)
Typical Steps 1000 (DDPM) 50 (DDIM)
Model Size ~114M ~1.07B

Stable Diffusion Reference

@article{rombach2022high,
  title={High-resolution image synthesis with latent diffusion models},
  author={Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj{\"o}rn},
  journal={CVPR},
  year={2022}
}

@article{ho2022classifierfree,
  title={Classifier-free diffusion guidance},
  author={Ho, Jonathan and Salakhans, Tim},
  journal={NeurIPS Workshop},
  year={2022}
}

Occupancy Network: Learning 3D Reconstruction in Function Space

"Occupancy Networks: Learning 3D Reconstruction in Function Space" (Mescheder et al., 2019)

Architecture Overview

┌───────────────────────────────────────────────────────────────┐
│                   Occupancy Network                           │
│                                                               │
│  Input Image ──→ CNN Encoder ──→ Global Feature Vector c      │
│                                                               │
│  Query Point p = (x, y, z) ──→ ┐                              │
│                                  │                             │
│                    ┌─────────────▼──────────────┐             │
│                    │  Conditional MLP            │             │
│                    │  f(p, c) → occupancy [0,1]  │             │
│                    └────────────────────────────┘             │
│                                                               │
│  Any 3D point p + image feature c → occupied probability      │
└───────────────────────────────────────────────────────────────┘

Key Idea: Replace discrete voxel grids (O(N³) memory) with a
continuous function f: R³ → [0,1]. The neural network IS the
3D representation — query at ANY resolution, O(1) memory.

Key Concepts

  1. Continuous Occupancy Function: Instead of storing discrete voxels, learn a function f(p, c) that maps any 3D point to an occupancy probability. This enables theoretically infinite resolution 3D reconstruction.

  2. Conditional MLP: The occupancy predictor is a shared MLP that takes (encoded 3D point, image feature) → [0, 1]. All query points share the same network — just like NeRF's implicit representation.

  3. Three Conditioning Variants:

    • Simple Concat: Concatenate [point, c] at MLP input (paper default)
    • FiLM: Use c to modulate scale (γ) and shift (β) at every layer: h' = γ(c) ⊙ h + β(c)
    • Concat+Skip: Inject c at every layer for strongest conditioning
  4. Sinusoidal Positional Encoding: 3D coordinates are encoded with multi-frequency sinusoids (like NeRF), allowing the MLP to capture fine spatial detail.

  5. Importance Sampling: Training points are sampled densely near the object surface (where the decision boundary is) and sparsely far away, for efficient and sharp boundary learning.

  6. Mesh Extraction: At inference, query f(p, c) on a dense 3D grid and run Marching Cubes to extract the isosurface at 0.5.

Occupancy Network Smoke Test

python3 implementations/occupancy_network.py

Quick Training Demo (unit sphere)

The smoke test includes a 300-iteration training demo that teaches the network to reconstruct a unit sphere, reaching ~97% accuracy.

Occupancy Network Variants

Variant Cond. Method Hidden Layers Params
Concat (default) Input concat 512 5 ~1.4M
FiLM Feature-wise modulation 512 5 ~3.1M
Concat+Skip Every-layer concat 256 8 ~1.5M

3D Representation Comparison

Method Resolution Memory Topology
Voxel (32³) 32³ fixed 32K values Arbitrary
Voxel (256³) 256³ fixed 16M values Arbitrary
Point Cloud N points O(N) Needs postproc
Mesh (template) F faces O(F) Fixed template
Occupancy Net Infinite O(1) weights Arbitrary

Occupancy Network Reference

@inproceedings{mescheder2019occupancy,
  title={Occupancy networks: Learning 3d reconstruction in function space},
  author={Mescheder, Lars and Oechsle, Michael and Niemeyer, Michael and Nowozin, Sebastian and Geiger, Andreas},
  booktitle={CVPR},
  year={2019}
}

DiT: Scalable Diffusion Models with Transformers

"Scalable Diffusion Models with Transformers" (Peebles & Xie, 2023)

Architecture Overview

┌──────────────────────────────────────────────────────────────────────┐
│                    DiT Architecture (Latent Diffusion)               │
│                                                                      │
│  Noisy Latent zₜ  (B, 4, 32, 32)   ← from VAE encoder              │
│         │                                                            │
│    ┌────▼────────┐                                                  │
│    │  Patchify   │  Conv2d(4→D, kernel=P, stride=P)                │
│    │  P=2: 32→16 │  16×16 = 256 patches of dim D                    │
│    └────┬────────┘                                                  │
│         │                                                            │
│    ┌────▼─────────────────────────────┐                             │
│    │  + 2D Sinusoidal Position Embed   │  (fixed, not learnable)    │
│    └────┬─────────────────────────────┘                             │
│         │                                                            │
│    ┌────▼────┐   ┌──────────┐   ┌──────────┐                       │
│    │  Time t  │   │  Class y  │   │  c = t+y │  ← conditioning     │
│    │ Embedder │   │  Embedder │   │  (add)    │                       │
│    └────┬────┘   └─────┬────┘   └─────┬────┘                       │
│         │              │              │                              │
│    ┌────▼──────────────▼──────────────▼────────┐                    │
│    │   DiTBlock × depth  (with adaLN-Zero)     │                    │
│    │                                            │                    │
│    │   c → SiLU → Linear → 6·D params           │                    │
│    │   [shift_msa, scale_msa, gate_msa,         │                    │
│    │    shift_mlp, scale_mlp, gate_mlp]         │                    │
│    │                                            │                    │
│    │   x = x + gate_msa · Attn(adaLN(Norm(x)))  │                    │
│    │   x = x + gate_mlp · MLP(adaLN(Norm(x)))   │                    │
│    └──────────────────┬───────────────────────┘                    │
│    ┌──────────────────▼───────────────────────┐                    │
│    │  FinalLayer → Unpatchify → (B, C, H, W)  │                    │
│    └──────────────────────────────────────────┘                    │
└──────────────────────────────────────────────────────────────────────┘

Key Concepts

  1. Transformer replaces U-Net: DiT replaces the standard U-Net backbone with a Transformer that operates on latent patches. Same block repeated N times — no encoder-decoder asymmetry, no skip connections.

  2. adaLN-Zero Conditioning: The core innovation. Conditioning signal c (timestep + class) modulates LayerNorm parameters through adaptive scale and shift. Additionally, residual connections are gated by c, with gates initialized to zero — so the network starts as identity and gradually learns to use each block.

  3. Patchify & Unpatchify: Conv2d converts the latent grid into a sequence of patch tokens (like ViT). After the Transformer, the output tokens are rearranged back to the spatial grid. Smaller patch size = more tokens = more compute = better quality.

  4. Classifier-Free Guidance (CFG): During training, class labels are randomly dropped 10% → null class embedding. During inference: ε = ε_uncond + s × (ε_cond - ε_uncond) with s > 1.

  5. Scalability: More Gflops (through depth, width, or more patches) consistently yields lower FID. DiT-XL/2 achieves FID 2.27 on ImageNet 256×256 (SOTA at publication).

DiT Smoke Test

python implementations/dit.py

DiT Model Variants

Model Depth Hidden Heads Patch Params Gflops
DiT-S/2 12 384 6 2 ~33M 0.6
DiT-S/4 12 384 6 4 ~33M 0.2
DiT-S/8 12 384 6 8 ~33M 0.1
DiT-B/2 12 768 12 2 ~131M 4.0
DiT-B/4 12 768 12 4 ~130M 1.3
DiT-B/8 12 768 12 8 ~131M 0.5
DiT-L/2 24 1024 16 2 ~458M 20.2
DiT-L/4 24 1024 16 4 ~458M 6.1
DiT-L/8 24 1024 16 8 ~459M 2.4
DiT-XL/2 28 1152 16 2 ~675M 34.2
DiT-XL/4 28 1152 16 4 ~675M 10.3
DiT-XL/8 28 1152 16 8 ~676M 4.1

Conditioning Strategies Comparison (Paper Ablation)

Strategy How c enters FID ↓ Notes
In-context (concat) c as extra token ~70 Weak modulation
Cross-attention c as K,V in cross-attn ~50 Extra attention cost
adaLN-Zero ★ c modulates LayerNorm + gates ~30 Best FID, simplest

DiT vs U-Net Comparison

Property U-Net (SD 1.x) DiT
Architecture Encoder-Bottleneck-Decoder + skip connections N identical blocks
Conditioning Time → ResBlock, Class → Cross-Attention Time+Class → adaLN-Z
Skip connections Yes (critical) No (not needed)
Scalability Complex (multi-scale) Simple (just N×block)
Positional encoding Not needed 2D sincos (required)

Legacy & Impact

DiT is the backbone behind many modern diffusion models:

  • Stable Diffusion 3 (MMDiT, multi-modal DiT variant)
  • FLUX.1 (Black Forest Labs)
  • Sora (OpenAI video generation)
  • PixArt-Σ / PixArt-α
  • Hunyuan-DiT (Tencent)

DiT Reference

@article{peebles2023scalable,
  title={Scalable Diffusion Models with Transformers},
  author={Peebles, William and Xie, Saining},
  journal={ICCV},
  year={2023}
}

VAE: Variational Autoencoder

"Auto-Encoding Variational Bayes" (Kingma & Welling, 2014)

Architecture Overview

┌───────────────────────────────────────────────────────────┐
│                    Variational Autoencoder                │
│                                                          │
│  Input x ──→ Encoder ──→ μ, log σ² ──→ z = μ + σ·ε     │
│                                              │           │
│                                              ↓           │
│                    Reconstructed x̂ ←── Decoder(z)        │
│                                                          │
│  Loss = Reconstruction + β × KL(N(μ,σ²) ‖ N(0,1))       │
└───────────────────────────────────────────────────────────┘

Regular AE vs VAE:
  AE:   Encoder(x) → z (deterministic)  → Decoder → x̂
  VAE:  Encoder(x) → μ, σ² (distribution) → z~N(μ,σ²) → Decoder → x̂

Reparameterization Trick (makes sampling differentiable):
  z = μ + σ · ε,   where ε ~ N(0,1)
  Gradients flow through μ and σ (deterministic paths)
  ε is "noise" from outside the computation graph

Key Concepts

  1. Reparameterization Trick: Sampling z ~ N(μ, σ²) is not differentiable. Solution: z = μ + σ · ε where ε ~ N(0,1). Now gradients flow through μ and σ.

  2. KL Divergence Regularizer: D_KL(N(μ,σ²) ‖ N(0,1)) = ½ Σ(σ² + μ² - 1 - log σ²). Forces latent distributions to stay near standard normal, making the latent space smooth and continuous.

  3. β-VAE: Adjustable KL weight. β=1 is standard VAE; β>1 encourages disentangled representations (each latent dimension captures an independent factor).

  4. Why VAE matters for diffusion: SD and DiT operate in VAE latent space (4×64×64 instead of 3×512×512). The VAE's KL regularization ensures the latent space is smooth enough for diffusion to work. Scaling factor z_scaled = 0.18215 × z normalizes for the noise schedule.

VAE Smoke Test

python implementations/vae.py

VAE Variants

Variant Encoder/Decoder Input Latent Use Case
Vanilla VAE (MLP) Linear + ReLU (B, 784) (B, 20) MNIST / flat data
Conv VAE (CNN) Conv / ConvT (B, 1, 28, 28) (B, 64) Image generation
SD VAE (ResBlock) ResBlock + GroupNorm (B, 3, 256, 256) (B, 4, 32, 32) Latent Diffusion

VAE Reference

@article{kingma2014auto,
  title={Auto-Encoding Variational Bayes},
  author={Kingma, Diederik P and Welling, Max},
  journal={ICLR},
  year={2014}
}

@article{higgins2017betavae,
  title={$\beta$-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework},
  author={Higgins, Irina and others},
  journal={ICLR},
  year={2017}
}

MoE: Mixture of Experts — Sparse Activation for Parameter Scaling

implementations/moe.py implements the core designs of modern MoE architectures, from Switch Transformer to DeepSeek-V3.

Core Idea

Decouple parameter count from compute cost: a dense FFN is replaced by N expert FFNs, and each token is routed to only the top-K experts by a lightweight router. The model has N× the parameters but only K× the FFN compute per token.

logits  = W_r · x                       # router scores, (T, N)
gate    = Softmax(TopK(logits))         # sparse gate weights, renormalized
y       = Σ_{i ∈ TopK} gate_i · Expert_i(x) + SharedExpert(x)

Implemented Components

Component Class / Mechanism Source
Sparse routing MoELayer top-K gating + weight renormalization Shazeer 2017, GShard
Expert FFN ExpertFFN (Linear → GELU → Linear) standard Transformer FFN
Load-balance loss aux_loss(): L = N · Σ f_i · P_i Switch Transformer
Capacity & token drop capacity_factor, overflow tokens skip MoE (kept by residual) Switch/GShard
Shared experts num_shared_experts > 0, always active for common knowledge DeepSeekMoE
Aux-loss-free balancing expert_bias buffer, update_expert_bias(): b_i += u · sign(1/N − f_i) DeepSeek-V3
Full block MoETransformerBlock (pre-norm attention + MoE FFN) Mixtral/DeepSeek style

Key Design Details

  1. Selection vs. weighting: with aux-free balancing, the bias is added to router logits only for expert selection; the gate weights used for combining always come from unbiased softmax probabilities, so the main task gradient is never disturbed.
  2. Aux loss gradient path: f_i (token fraction) is a detached discrete count; gradients flow to the router only through P_i (mean route probability) — exactly as in Switch Transformer.
  3. Token dropping: when an expert exceeds its capacity ceil(T·K/N × capacity_factor), overflow tokens receive zero MoE output; the block's residual connection preserves their information.
  4. Trade-off: MoE saves FLOPs, not memory — all expert parameters must be loaded for inference.

MoE Smoke Test

python implementations/moe.py

Covers: (1) Switch-style layer with aux loss, (2) gradient flow to router + experts, (3) DeepSeek-style bias rebalancing from a 95.7%→13.7% skewed load, (4) full MoE transformer block.

MoE Reference

@article{shazeer2017outrageously,
  title={Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer},
  author={Shazeer, Noam and others},
  journal={ICLR},
  year={2017}
}
@article{fedus2021switch,
  title={Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity},
  author={Fedus, William and Zoph, Barret and Shazeer, Noam},
  journal={JMLR},
  year={2022}
}
@article{deepseekv3,
  title={DeepSeek-V3 Technical Report},
  author={DeepSeek-AI},
  journal={arXiv:2412.19437},
  year={2024}
}

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages