Repository navigation
Conversation
Signed-off-by: Leechael Yim <[email protected]>
Leechael
force-pushed
the
fix/vmm-offload-directory-deletion
branch
from
October 17, 2026 10:13
4e73819 to
109599d
Compare
Signed-off-by: Leechael Yim <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
RemoveVmreturns quickly, but its background task calls synchronous filesystem deletion on a Tokio worker.tokio::spawndoes not move that work off the async executor. Unlinking a fully allocated qcow2 disk can spend seconds freeing filesystem extents.Production sampling caught a Rocket worker spending about 3.7 seconds in kernel CPU during one removal. Waiting Status requests were dispatched immediately after the workdir deletion finished. The same timing pattern appeared across multiple removals.
Change
Run workdir deletion in
spawn_blocking, after the VM exits and netd confirms interface release.The blocking task also owns the per-VM lock, the existing removal guard, and final state/CID cleanup. This matters because cancelling the async waiter does not cancel an already-running blocking task. Offloading only the filesystem call would let cancellation release the guards while deletion continues, or leave a deleted VM occupying its CID.
Apply the same offload to failed-create cleanup and image deletion. The existing
removingstate and.removingrecovery mechanism are unchanged; this PR adds no VM state or RPC changes.Tests
cargo test --locked -p dstack-vmm: 224 passed, 0 failed, 3 ignored. The ignored tests require root and a network namespace.The async-progress test covers VM removal. The two other deletion paths have result/error tests, not async-progress tests.
This removes a confirmed source of executor blocking. It does not establish why one blocked worker delayed multiple requests on the production 128-worker runtime, or explain stalls without a removal.