Skip to content

feat(mooncake): enable Ascend pinned receive and loader-side H2D - #1071

Open
curnane-lab wants to merge 1 commit into
sgl-project:mainfrom
curnane-lab:perf/mooncake-ascend-receive
Open

curnane-lab wants to merge 1 commit into
sgl-project:mainfrom
curnane-lab:perf/mooncake-ascend-receive

Conversation

@curnane-lab

@curnane-lab curnane-lab commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

The pinned receive pool and its device copy path were CUDA-only. This PR adds Ascend NPU support for pinned receive and loader-side H2D while preserving the existing CPU and CUDA behavior.

NPU loader-side H2D is explicitly opt-in. Existing pinned-receive configurations keep prefetched batches on host by default, rather than silently moving the whole prefetch queue into NPU memory. This is an enablement/correctness change, not a demonstrated throughput win.

Modifications

  • Opt-in NPU delivery: consumer_device() keeps NPU pinned consumers on host by default. Set deployment.disaggregated.npu_loader_h2d: true or DISAGG_NPU_LOADER_H2D=1 to place floating-point features on the rank's NPU in loader workers. LOCAL_RANK is mirrored as for CUDA, explicit CPU trainers stay on host, and CUDA pinned delivery remains unchanged.
  • Prefetch memory is explicit: receive_pool_bytes bounds receive slots, not device output copies or collate temporaries. For five BF16 features at 32K, B2, and prefetch depth 8, queued outputs alone are 12.5 GiB/rank, excluding the current training batch and collate allocations. Size data.dataloader_num_workers accordingly when enabling H2D.
  • Pinned host allocation: the pinned pool uses torch_npu's pinned host allocator when Ascend is available and still degrades to pageable host buffers on accelerator-less hosts.
  • Device stream dispatch: copy_stream() and the slot-copy path dispatch to torch.cuda or torch.npu. Slot release waits for copy completion before reuse, and NPU outputs record the consumer stream when supported.
  • Loader handoff: FeatureDataLoader records an NPU ready event after transform/collate. Consumption performs wait_event() and record_stream() on the consumer stream, covering both execution dependency and allocator lifetime.
  • Configuration wiring: the typed npu_loader_h2d setting is rendered by launch planning and can be overridden by DISAGG_NPU_LOADER_H2D; the configuration reference documents the device-memory tradeoff.

Example for the opt-in path:

deployment:
  disaggregated:
    receive_buffers: pinned
    npu_loader_h2d: true

Validation

Current PR head: d7e41f0, squashed onto upstream/main at 1b5c0dde137a28447606ffb6078e01eddfa77980. The final tree is identical to the tested pre-squash tree.

Current-head regression after syncing upstream:

  • 245 passed, 11 skipped, 1 xfailed across the seven relevant receive, loader, launch-plan, defaults, and schema test files, with backend autoload disabled. Because this Ascend workstation has torch_npu installed, the launch-plan suite's non-Ascend host detection was pinned to CUDA; nested Ascend-specific tests still exercised the Ascend branch. This is a test-host accommodation, not a product-code change.
  • 9 passed in test_mooncake_receive_npu.py on Ascend 910B3, including default-host behavior and explicit H2D enablement.
  • Black and isort checks passed for all 11 modified Python files.
  • GitHub lint, codec-parity, and draft-config passed on the current head; unit-test was still in progress when this body was generated.

Additional real-backend validation used a real Mooncake master, two real clients, mooncake-transfer-engine-npu==0.3.13.post1, forced TCP, and npu:1:

  • Pinned host slots, exact NPU-delivered floating-point values, and host integer features.
  • Loader-side H2D, delayed collate, and handoff to a separate NPU consumer stream.
  • Immediate slot reuse with exact values and five pool hits.
  • Two deferred removals drained successfully in one attempt; after the forced drain, the samples were unreadable from the other client.

A plain release() can use a local tombstone before Mooncake's deferred physical removal completes. The real-backend check therefore distinguishes that local KeyError from cross-client visibility after drain_sample_removals(); it does not claim immediate cross-client deletion.

Earlier supplementary validation:

  • d58e652 + this test-only patch, SHA-256 d1f14c08b62ca3e59cb13f2113dfe9b684e61600004a761df6acf79ef1aff99d, adds two cases to the original eight NPU tests and passed 10/10 on 910B3. This is not a clean-PR-head test count.
  • The positive handoff tests passed, but earlier negative controls also passed on pre-fix code; they do not demonstrate race-detection capability.
  • No new CUDA-hardware run is claimed for this follow-up. Shared CUDA behavior is covered by the existing tests and unchanged device-specific branches.

Benchmark & Profiling

The following measurements were recorded for the H2D-enabled path. They do not justify enabling it by default.

  • Training A/B (managed-local, 8 steps, same seed, main+N1 vs. this change): per-step losses were bit-identical. Steady-state step time and train_compute_time_s were unchanged within noise; data_wait_time_s was already zero after step 1 on both paths. Existing loader prefetch already hid receive latency at this scale, so no end-to-end speedup is claimed.
  • A mock-backend receive/copy microbenchmark on clean d58e652 used _FakeMooncakeStore with a real NPU, not TCP/RDMA. For an 8.015625 MiB payload, it measured only get(npu:1) + torch.npu.synchronize(npu:1), with 10 warmups and 100 alternating-order iterations per mode. Median latency was 1.936 ms pageable vs. 2.220 ms pinned (0.872x pageable/pinned ratio). This microbenchmark shows no receive speedup at this size; raw samples and metadata are archived.

Validation archive and reproduction notes.

Checklist

@curnane-lab
curnane-lab force-pushed the perf/mooncake-ascend-receive branch from ace83e9 to d58e652 Compare November 3, 2026 06:38
@curnane-lab
curnane-lab marked this pull request as ready for review November 4, 2026 02:12
@curnane-lab

Copy link
Copy Markdown
Collaborator Author

@jiapingW Please take a look

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants