Repository navigation
feat(mooncake): enable Ascend pinned receive and loader-side H2D - #1071
Open
curnane-lab wants to merge 1 commit into
Open
curnane-lab wants to merge 1 commit into
curnane-lab wants to merge 1 commit into
Conversation
curnane-lab
force-pushed
the
perf/mooncake-ascend-receive
branch
from
November 3, 2026 06:38
ace83e9 to
d58e652
Compare
curnane-lab
marked this pull request as ready for review
November 4, 2026 02:12
curnane-lab
force-pushed
the
perf/mooncake-ascend-receive
branch
from
November 20, 2026 12:50
bf714fa to
d7e41f0
Compare
curnane-lab
requested review from
FlamingoPg,
shuaills and
sleepcoo
as code owners
November 20, 2026 12:50
Collaborator
Author
|
@jiapingW Please take a look |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pinned receive pool and its device copy path were CUDA-only. This PR adds Ascend NPU support for pinned receive and loader-side H2D while preserving the existing CPU and CUDA behavior.
NPU loader-side H2D is explicitly opt-in. Existing pinned-receive configurations keep prefetched batches on host by default, rather than silently moving the whole prefetch queue into NPU memory. This is an enablement/correctness change, not a demonstrated throughput win.
Modifications
consumer_device()keeps NPU pinned consumers on host by default. Setdeployment.disaggregated.npu_loader_h2d: trueorDISAGG_NPU_LOADER_H2D=1to place floating-point features on the rank's NPU in loader workers.LOCAL_RANKis mirrored as for CUDA, explicit CPU trainers stay on host, and CUDA pinned delivery remains unchanged.receive_pool_bytesbounds receive slots, not device output copies or collate temporaries. For five BF16 features at 32K, B2, and prefetch depth 8, queued outputs alone are 12.5 GiB/rank, excluding the current training batch and collate allocations. Sizedata.dataloader_num_workersaccordingly when enabling H2D.copy_stream()and the slot-copy path dispatch totorch.cudaortorch.npu. Slot release waits for copy completion before reuse, and NPU outputs record the consumer stream when supported.FeatureDataLoaderrecords an NPU ready event after transform/collate. Consumption performswait_event()andrecord_stream()on the consumer stream, covering both execution dependency and allocator lifetime.npu_loader_h2dsetting is rendered by launch planning and can be overridden byDISAGG_NPU_LOADER_H2D; the configuration reference documents the device-memory tradeoff.Example for the opt-in path:
Validation
Current PR head:
d7e41f0, squashed ontoupstream/mainat1b5c0dde137a28447606ffb6078e01eddfa77980. The final tree is identical to the tested pre-squash tree.Current-head regression after syncing upstream:
torch_npuinstalled, the launch-plan suite's non-Ascend host detection was pinned to CUDA; nested Ascend-specific tests still exercised the Ascend branch. This is a test-host accommodation, not a product-code change.test_mooncake_receive_npu.pyon Ascend 910B3, including default-host behavior and explicit H2D enablement.Additional real-backend validation used a real Mooncake master, two real clients,
mooncake-transfer-engine-npu==0.3.13.post1, forced TCP, andnpu:1:A plain
release()can use a local tombstone before Mooncake's deferred physical removal completes. The real-backend check therefore distinguishes that localKeyErrorfrom cross-client visibility afterdrain_sample_removals(); it does not claim immediate cross-client deletion.Earlier supplementary validation:
d58e652+ this test-only patch, SHA-256d1f14c08b62ca3e59cb13f2113dfe9b684e61600004a761df6acf79ef1aff99d, adds two cases to the original eight NPU tests and passed 10/10 on 910B3. This is not a clean-PR-head test count.Benchmark & Profiling
The following measurements were recorded for the H2D-enabled path. They do not justify enabling it by default.
train_compute_time_swere unchanged within noise;data_wait_time_swas already zero after step 1 on both paths. Existing loader prefetch already hid receive latency at this scale, so no end-to-end speedup is claimed.d58e652used_FakeMooncakeStorewith a real NPU, not TCP/RDMA. For an 8.015625 MiB payload, it measured onlyget(npu:1) + torch.npu.synchronize(npu:1), with 10 warmups and 100 alternating-order iterations per mode. Median latency was 1.936 ms pageable vs. 2.220 ms pinned (0.872x pageable/pinned ratio). This microbenchmark shows no receive speedup at this size; raw samples and metadata are archived.Validation archive and reproduction notes.
Checklist