Skip to content

fix: raise the meta busy-retry ceiling and quiesce the network when the stopped flip fails - #281

Merged
CMGS merged 2 commits into
masterfrom
fix/meta-busy-ceiling
Oct 5, 2026
Merged

CMGS merged 2 commits into
masterfrom
fix/meta-busy-ceiling

Conversation

@CMGS

@CMGS CMGS commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Concurrent vm hibernate / snapshot save storms fail with writer contended past retry ceiling: store busy once roughly 32 or more run at once.

Root cause

When meta.db sits on the same filesystem as the memory dumps, most commits are sub-millisecond. The occasional WAL fsync, though, queues behind the dump flush (SyncTree): p90 is about 160ms and the max about 330ms. One slow commit stalls every writer queued behind it, and the waits starve past the 5s ceiling. The failure lands after the dump is already captured, so that work is thrown away and a hibernating VM is resumed.

Isolation on bare metal: 64 x 512M VMs, 64 concurrent hibernates, two arms interleaved x3. The arms differ only in where meta.db lives; snapshot data stays on xfs in both.

meta.db on failed per round
xfs (shared with dumps) 18 / 22 / 20
tmpfs 0 / 0 / 0

Changes

  1. Busy-retry ceiling 5s -> 30s. Both arms are built from master and differ only in this constant, interleaved:

    storm 5s failed 30s failed 30s max wait
    hibernate, 64-way x3 22 / 19 / 19 0 / 0 / 0 9-11s
    snapshot save, 64-way x2 21 / 23 0 / 0 9-10s
    hibernate, 128-way x1 82 / 128 0 / 128 21s

    Throughput is the same in every arm, about 6 dumps/s, because it is disk-bound. The wait grows with queue depth, so callers that fan out dumps should still bound their concurrency; the ceiling is a backstop, not admission control.

  2. Stop and hibernate quiesce the network when the Stopped flip fails. After a confirmed VMM exit the flip is warn-and-continue. A failed flip also skipped the network quiesce, because the record still reads Running with QuiescePending unset. A CNI VM therefore kept its host plumbing up with a dead TAP (the fix(cni): quiesce host NICs on stop to end the idle-TAP softirq storm #130 precondition) until a daemon reconcile or the next start/restore. With the 5s ceiling this was observed: hibernate exited 0 after logging mark stopped ...: store busy. The quiesce now keys on the VM having gone down, and the flip outcome only fences the generation.

Tests

  • New regression tests: TestHibernateSequenceQuiescesWhenTheStoppedFlipFails and TestStopQuiescesWhenTheStoppedFlipFails. Both inject a busy store after terminate; both fail without the fix (quiesced = 0) and pass with it.
  • go test -race ./... passes on darwin.
  • ./hypervisor/... ./meta/... pass on linux.
  • make lint reports 0 issues on linux and darwin, and asl is clean.

CMGS added 2 commits October 5, 2026 16:56
Concurrent memory dumps (vm hibernate, snapshot save) on the same
filesystem as meta.db make the occasional WAL fsync slow (p90 ~160ms),
and the writer queue behind it starves past 5s. The failure lands after
the dump is already captured, so the work is thrown away and a
hibernating VM is resumed. At 64 concurrent hibernates on bare metal,
5s failed 19-22 of 64 per round; 30s failed 0 of 448 operations across
hibernate and save storms up to 128-way, with throughput unchanged
(disk-bound). Max observed wait was 11s at 64-way and 21s at 128-way.
…ped flip fails

After a confirmed VMM exit the Stopped flip is warn-and-continue, but a
failed flip also skipped the network quiesce: the record still reads
Running with QuiescePending unset, so a CNI VM kept its host plumbing up
with a dead TAP (the #130 softirq precondition) until a daemon reconcile
or the next start/restore. The quiesce now keys on the VM having gone
down, and the flip outcome only fences the generation.
@CMGS
CMGS merged commit a71e95e into master Oct 5, 2026
4 checks passed
@CMGS
CMGS deleted the fix/meta-busy-ceiling branch October 5, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant