perf(server): keep CUDA graph replay when LoRA adapters are loaded #840
Copy link
Copy link
Open
Description
Problem
Loading a LoRA adapter takes a model off CUDA graph replay, and on short encoder calls the eager path is several times slower.
- On main, the ModernBERT and ColBERT flash adapters run every request eagerly once any PEFT LoRA is loaded, base-model requests included (
packages/sie_server/src/sie_server/adapters/modernbert_flash/__init__.py:if self._graphs is None or self._peft_model is not None: return None). Forgte-modernbert-baseon an L4, graphs are worth 17.5 → 2.4 ms per query (perf(server): replay ModernBERT flash forwards as CUDA graphs #423). - Draft feat(server): support runtime LoRA for native decision models #829 (runtime LoRA for GLiNER2.5-Decide, Laya, pplx-decider and GLiClass) keeps graphs for base requests, but adapted requests run eagerly. Eager Decide on an L4 is 46.8 ms p50 against 8.4 ms with graphs (perf(gliner2): CUDA graphs and one pass per batch for GLiNER2.5-Decide #721), so a tuned Decide would be about 5x slower than the base it was tuned from.
- Qwen3-Embedding-4B gives up its four batches in flight when LoRA is configured (perf(qwen3-embedding): keep up to four Qwen3-Embedding-4B batches in flight to SGLang #647).
Goal
Requests that select a LoRA adapter replay recorded graphs as base requests do, under the same numerics rule as the other graph work: no probability moves more than 0.02 against eager, or the result is no worse against float32.
Possible approaches
- Static adapter buffers. Capture graphs with the LoRA A/B matrices read from fixed buffers. Before replay, copy the selected adapter's weights into those buffers, so adapters swap without a recapture. Needs one graph set per rank and target-module layout.
- Per-adapter recordings for hot adapters. Record a graph set per (adapter, bucket) inside the existing graph memory budget, and evict a recording together with its adapter in the LoRA LRU.
- Merge at staging. Merge a hot adapter into a copy of the base weights and record graphs for that copy. Simplest numerically, but it costs memory per adapter.
Whichever approach is chosen, base-model graphs must stay unaffected when adapters are loaded.
Acceptance
- Base requests keep graph replay while any adapter is loaded, starting with the ModernBERT and ColBERT flash adapters.
- Adapted requests replay graphs on GLiNER2.5-Decide and at least one ModernBERT model. Report parity numbers and L4 p50 before and after.
- Graph memory stays within the existing budget, and evicting an adapter frees its recordings.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity