Skip to content

fix(vmm): offload recursive directory deletion - #1456

Draft
Leechael wants to merge 2 commits into
nextfrom
fix/vmm-offload-directory-deletion
Draft

Leechael wants to merge 2 commits into
nextfrom
fix/vmm-offload-directory-deletion

Conversation

@Leechael

@Leechael Leechael commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Why

RemoveVm returns quickly, but its background task calls synchronous filesystem deletion on a Tokio worker. tokio::spawn does not move that work off the async executor. Unlinking a fully allocated qcow2 disk can spend seconds freeing filesystem extents.

Production sampling caught a Rocket worker spending about 3.7 seconds in kernel CPU during one removal. Waiting Status requests were dispatched immediately after the workdir deletion finished. The same timing pattern appeared across multiple removals.

Change

Run workdir deletion in spawn_blocking, after the VM exits and netd confirms interface release.

The blocking task also owns the per-VM lock, the existing removal guard, and final state/CID cleanup. This matters because cancelling the async waiter does not cancel an already-running blocking task. Offloading only the filesystem call would let cancellation release the guards while deletion continues, or leave a deleted VM occupying its CID.

Apply the same offload to failed-create cleanup and image deletion. The existing removing state and .removing recovery mechanism are unchanged; this PR adds no VM state or RPC changes.

Tests

  • cargo test --locked -p dstack-vmm: 224 passed, 0 failed, 3 ignored. The ignored tests require root and a network namespace.
  • Formatting and strict Clippy checks pass.
  • A single-worker regression test pauses deletion and checks that Status can still complete. It fails against the old implementation and passes with this fix. Cancellation, netd ordering, orphan retention, and marker ordering also have regression checks.

The async-progress test covers VM removal. The two other deletion paths have result/error tests, not async-progress tests.

This removes a confirmed source of executor blocking. It does not establish why one blocked worker delayed multiple requests on the production 128-worker runtime, or explain stalls without a removal.

@Leechael
Leechael force-pushed the fix/vmm-offload-directory-deletion branch from 4e73819 to 109599d Compare October 17, 2026 10:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant