Skip to content

pool: a dropped pool's golden is deleted, and a start sweeps goldens no pool uses - #334

Merged
CMGS merged 1 commit into
mainfrom
fix/333-golden-gc
Oct 6, 2026
Merged

CMGS merged 1 commit into
mainfrom
fix/333-golden-gc

Conversation

@CMGS

@CMGS CMGS commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Closes #333.

Problem

A pool removed from the desired set left its golden on disk forever: goldens/<key hash>/ plus goldens/<key hash>.stamp. Every image roll leaked one multi-GB snapshot per retired key, and a start whose pool set no longer held a key never cleaned it up either.

Change

  • Drop: a pool leaves m.pools in two places, applyPools and refillOnce, and both now call dropPool. Under m.mu it removes the stamp and renames the golden to a *.tmp sibling; a goroutine then deletes the renamed directory and logs one line naming the pool.
    • adoptGolden runs under the same lock and needs both the stamp and the directory. So a key that is re-added at once either finds nothing and rebuilds, or never sees the golden at all. It cannot adopt a directory that is being deleted.
  • Start: Reconcile already removed goldens/*.tmp. That glob becomes one pass over goldens/ that removes every entry that is not a pooled key's directory or stamp, and logs one line per orphan golden. It covers keys dropped from the config between restarts, goldens left by earlier versions, and a delete that a crash interrupted.
  • Docs: sandboxd-api.md (PUT /v1/pools) and deploy.md state that an omitted pool's golden is deleted once the pool is idle, that warm: 0 keeps it, and that a start deletes any golden no pool uses.

Answers to the issue's questions

  1. A clone does not depend on its golden after cloning. cocoon copies the disk (reflink or sparse copy) and hard-links the memory file into the VM's own directory, with a symlink only when the two are on different filesystems. Hibernation snapshots, checkpoints and templates are separate exports of the running VM. Deleting a golden is safe for live claims.
  2. Keeping a removed pool's golden was not intended; adoption exists for restarts. warm: 0 is the documented way to keep a key registered, so an omitted key is deleted at once, with no grace period.

Not changed

  • ReloadConfig and checkImagesOnce. A golden they retire is rebuilt at the same path (exportGolden removes the old directory before the rename), so it is not a leak.
  • No size in the log line: the apparent size of a sparse golden overstates what is freed.

Cost and known limits

  • Hot path: none. Nothing is added to the claim path or to a refill tick that drops no pool.
  • Dropping a pool costs one unlink of the stamp and one rename under m.mu. The delete of the large files runs off the lock.
  • At start, orphan goldens are deleted synchronously before the server listens. This happens once per orphan.
  • An on-demand claim that resolved the golden just before its pool was dropped can fail its clone. The same can happen today when a rebuild replaces a golden. The claim returns an error and a retry cold-boots.
  • The memory file's space is freed only when the last clone that hard-links it is gone.

Verification

  • New tests in sandboxd/pool:
    • TestSetPoolsRemovalDeletesTheIdlePoolsGolden: the directory and stamp are gone when SetPools returns, and no *.tmp is left.
    • TestAReaddedPoolRebuildsTheGoldenItsRemovalDeleted: remove then re-add cold-boots a second golden and refills.
    • TestARemovedPoolKeepsItsGoldenUntilItSettles (build, refill): the golden stays while a build or refill is in flight and goes when refillOnce drops the pool.
    • TestSetPoolsWarmZeroKeepsTheGolden: a key kept at warm: 0 keeps its original golden.
    • TestReconcileSweepsGoldensNoPoolUses: an orphan directory, its stamp and a staging directory are removed; the pooled golden is kept and adopted.
  • Six mutations of the production code each fail at least one of these tests: stamp not removed, delete without the rename, start sweep limited to *.tmp, warm: 0 also drops, and a plain delete at each of the two call sites.
  • go test -race ./... in sandboxd: all packages ok on macOS and on Linux (golang:1.27.1 container, non-root). The new tests pass 300 runs with -race on Linux.
  • make go-lint: 0 issues. ×10 (5 modules × linux/darwin), no fmt diff.
  • asl -forwarder=false ./...: 0 findings on darwin and linux.

Not verified: no run on hardware. The case worth one check there is a node whose data_dir and cocoon run directory are on different filesystems, where a clone's memory file is a symlink into the golden.

…no pool uses

A pool removed from the desired set left its golden (goldens/<key hash> and
its .stamp) on disk forever: one multi-GB snapshot leaked per retired key on
every image roll, and a start whose pool set no longer held a key never
cleaned it up either.

dropPool now owns both places a pool leaves m.pools (applyPools and
refillOnce). Under m.mu it removes the stamp and renames the golden aside to
a *.tmp sibling, so adoptGolden, which runs under the same lock, never adopts
a golden that is being deleted; the unlink of the large files runs off the
lock. Reconcile's *.tmp sweep becomes one pass that removes every goldens/
entry that is not a pooled key's directory or stamp, which also finishes a
delete a crash interrupted.

A golden retired by a reload or an image change is rebuilt at the same path,
so those paths are unchanged. A key kept at warm 0 keeps its golden. A clone
does not depend on its golden after cloning, so live claims are unaffected;
an on-demand claim that resolved the golden just before its pool was dropped
can fail its clone, as it can today when a rebuild replaces the golden.

Closes #333
@CMGS
CMGS merged commit 252019a into main Oct 6, 2026
1 check passed
@CMGS
CMGS deleted the fix/333-golden-gc branch October 6, 2026 18:15
CMGS added a commit that referenced this pull request Oct 7, 2026
…claim's clone (#335)

* build: refresh Go and cargo dependencies

Go: projecteru2/core v0.1.7, aws-sdk-go-v2 service/s3 v1.114.0 and
feature/s3/transfermanager v0.4.13, puddle v2.2.3, go-colorable v0.1.16,
genproto; all five modules tidied. Cargo: libc 0.2.190 in silkd and
boot/init, mio 1.2.4 and tokio 1.53.2 in silkd.

* review: glob-free fork sweep, peer tar names from the store, errors.Is

The restart sweep of fork-* staging dirs matched with filepath.Glob on
data_dir, which matches nothing when the path holds a glob metacharacter;
it now uses the ReadDir helper the dir store's staging sweep uses. The peer
transport spelled the record's meta.json and export/ names itself although
they are the staging layout the puller hands to store.Publish. ReloadConfig's
godoc listed two of its three refusal causes.

* review: silkd checks an empty lsp manifest once, where argv is indexed

read_manifest and its only caller both rejected an empty argv.

* docs: sync the reference with the code

- sandboxd-api: a reload also answers 409 when it drops a pool's egress
  policy that live claims use.
- deploy: the tenants DDL carries the DEFERRABLE unique constraint the node
  creates, so a table created ahead of time accepts a token swap in one PUT.
- egress: the golden stamp also records the image id and a sized pool's disk.
- sdk-python, README, index: the claim env and the tenant verbs are Go-only,
  like pool retuning.
- README: the credentials e2e driver, the guestserver helper and the mypy
  gate were missing from their lists.
- Go SDK: SetPools says an omitted pool loses its golden once idle.

* build: the android flavor's local silkd default follows the release

The fallback stayed at 0.1.13 through three release bumps; CI passes the
pinned carrier, so only a local build took the old daemon.

* ci: a PR that edits the silkd or boot-init gate runs it

Both workflows listed their own file under push only, unlike the shell,
python and sandboxd gates.

* fix(pool): a dropped pool keeps its golden while a claim clones from it

A claim that missed the warm pool cloned from the pool's golden with no
hold on it, so a PUT /v1/pools that dropped the key in that window renamed
and deleted the directory under the clone and the claim failed. The pool
now counts those clones and is dropped once they are done, like a build
or a refill in flight.

A removed pool no longer hands its golden to new claims: they cold-boot,
as they do once the pool is gone, so a busy key cannot keep it alive.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pool: a removed pool's golden stays on disk forever

1 participant