Repository navigation
perf(sm120): add checked host collectives for non-P2P TP4 - #1794
Merged
Merged
Conversation
Signed-off-by: AlpinDale <[email protected]>
Signed-off-by: AlpinDale <[email protected]>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Signed-off-by: AlpinDale <[email protected]>
AlpinDale
added a commit
that referenced
this pull request
Oct 30, 2026
* perf(sm120): optimize DeepSeek-V4.1 inference Signed-off-by: AlpinDale <[email protected]> * feat(sm120): add native DeepSeek V4.1 backend Signed-off-by: AlpinDale <[email protected]> * perf(sm120): add paired PCIe all-reduce Signed-off-by: AlpinDale <[email protected]> * test(distributed): initialize paired all-reduce dispatch fixtures Signed-off-by: AlpinDale <[email protected]> * perf(sm120): fuse DeepSeek V4.1 decode norm, draft KV and sampling Signed-off-by: AlpinDale <[email protected]> * perf(sm120): sample DSpark draft and verify on vocab shards Signed-off-by: AlpinDale <[email protected]> * perf(sm120): split paired all-reduce exchange and combine Signed-off-by: AlpinDale <[email protected]> * perf(sm120): fuse DeepSeek V4.1 KV RoPE into Q/KV norm Signed-off-by: AlpinDale <[email protected]> * perf(sm120): use the ll_bf16 router GEMM on SM120 Signed-off-by: AlpinDale <[email protected]> * perf(sm120): size DeepSeek V4.1 Engram lookups to live rows Signed-off-by: AlpinDale <[email protected]> * perf(sm120): overlap DeepSeek V4.1 mHC coefficients with the next sublayer Signed-off-by: AlpinDale <[email protected]> * fix(sm120): run DeepSeek V4.1 indexer chunks only on prepared plans Signed-off-by: AlpinDale <[email protected]> * fix(sm120): resolve DeepSeek V4.1 mypy errors Signed-off-by: AlpinDale <[email protected]> * perf(sm120): fuse DSpark draft sampling winner selection Signed-off-by: AlpinDale <[email protected]> * perf(sm120): build DeepSeek V4.1 attention metadata in one launch Signed-off-by: AlpinDale <[email protected]> * perf(sm120): prefetch DeepSeek V4.1 attention weights during the MoE all-reduce Signed-off-by: AlpinDale <[email protected]> * perf(sm120): gather small sampling packets over the paired PCIe transport Signed-off-by: AlpinDale <[email protected]> * perf(sm120): prefetch the router weights during the attention all-reduce Signed-off-by: AlpinDale <[email protected]> * perf(sm120): project the DSpark Markov bias inside the vocab-parallel draft sampler Signed-off-by: AlpinDale <[email protected]> * perf(sm120): decode V4.1 sparse attention in one cluster kernel Signed-off-by: AlpinDale <[email protected]> * fix(dsv41): keep reasoning across tool-result turns and parse tool images in prompt order Signed-off-by: AlpinDale <[email protected]> * perf(sm120): prepare every DSpark draft KV group in one launch Signed-off-by: AlpinDale <[email protected]> * perf(sm120): run the b12x W4A8 decode MoE in one fused kernel Signed-off-by: AlpinDale <[email protected]> * perf(sm120): shard eager prefill indexer rows across TP ranks Honor the configured row threshold, preserve global metadata offsets, and gather ragged selected-index shards before attention. Keep pure decode, capture, and the wide candidate producer on their existing paths. Document the existing nondeterministic boundary-tie selection policy rather than promising bit-identical production indices. Signed-off-by: AlpinDale <[email protected]> * perf(sm120): avoid tensor equality in CuTe kernel argument binding Signed-off-by: AlpinDale <[email protected]> * Optimize native SM120 attention without changing BF16 arithmetic Signed-off-by: AlpinDale <[email protected]> * refactor(sm120): isolate native kernels and feature integration Move SM120 and PCIe sources into subsystem directories with local operator registration. Keep model preparation, configuration, rendering and worker lifecycle helpers feature-owned. Include draft TP faults in final output gating, enforce fused MoE staging bounds and preserve canonical NVFP4 gate/up layout. Replace stale implementation-detail assertions with numerical and consumer-visible regressions. Signed-off-by: AlpinDale <[email protected]> * perf(sm120): add checked host collectives for non-P2P TP4 (#1794) * Add checked host-memory collectives for non-P2P SM120 TP4 Signed-off-by: AlpinDale <[email protected]> * Specialize small host collectives and shard large reductions Signed-off-by: AlpinDale <[email protected]> * Avoid redundant host input copies on lane owners Signed-off-by: AlpinDale <[email protected]> --------- Signed-off-by: AlpinDale <[email protected]> * Fix V4.1 decoder row retention for fixed-token prompt scoring Signed-off-by: AlpinDale <[email protected]> --------- Signed-off-by: AlpinDale <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enable
APHRODITE_ALLREDUCE_USE_PCIE_PULL=1andAPHRODITE_ALLREDUCE_PCIE_TRANSPORT=host. Depends on #1788.Performance
4× RTX PRO 6000 Blackwell Server GPUs, TP4, 1,048,576 context, MNBT4096, DSpark K7. Three measured runs plus an excluded warmup; 2,048 output tokens/request.
NCCL is the best observed result with 2 or 8 CTAs per cell/metric. P2P/NVLS disabled; SHM transport verified. Rates are aggregate tok/s. Fresh-prefix prefill/TTFT; prefix-warmed common-active decode.
Host reductions use rank-ordered FP32 accumulation and one BF16 rounding; NCCL BF16 differs, so output and draft acceptance can change. Workstation/Max-Q hardware was not tested.
Verified: 88 CPU tests; four-GPU graph/epoch/fault checks; paired compatibility; model reasoning/tools; late-draft faults publish no tokens.