Repository navigation
pool: a dropped pool's golden is deleted, and a start sweeps goldens no pool uses - #334
Merged
Merged
Conversation
…no pool uses A pool removed from the desired set left its golden (goldens/<key hash> and its .stamp) on disk forever: one multi-GB snapshot leaked per retired key on every image roll, and a start whose pool set no longer held a key never cleaned it up either. dropPool now owns both places a pool leaves m.pools (applyPools and refillOnce). Under m.mu it removes the stamp and renames the golden aside to a *.tmp sibling, so adoptGolden, which runs under the same lock, never adopts a golden that is being deleted; the unlink of the large files runs off the lock. Reconcile's *.tmp sweep becomes one pass that removes every goldens/ entry that is not a pooled key's directory or stamp, which also finishes a delete a crash interrupted. A golden retired by a reload or an image change is rebuilt at the same path, so those paths are unchanged. A key kept at warm 0 keeps its golden. A clone does not depend on its golden after cloning, so live claims are unaffected; an on-demand claim that resolved the golden just before its pool was dropped can fail its clone, as it can today when a rebuild replaces the golden. Closes #333
CMGS
added a commit
that referenced
this pull request
Oct 7, 2026
…claim's clone (#335) * build: refresh Go and cargo dependencies Go: projecteru2/core v0.1.7, aws-sdk-go-v2 service/s3 v1.114.0 and feature/s3/transfermanager v0.4.13, puddle v2.2.3, go-colorable v0.1.16, genproto; all five modules tidied. Cargo: libc 0.2.190 in silkd and boot/init, mio 1.2.4 and tokio 1.53.2 in silkd. * review: glob-free fork sweep, peer tar names from the store, errors.Is The restart sweep of fork-* staging dirs matched with filepath.Glob on data_dir, which matches nothing when the path holds a glob metacharacter; it now uses the ReadDir helper the dir store's staging sweep uses. The peer transport spelled the record's meta.json and export/ names itself although they are the staging layout the puller hands to store.Publish. ReloadConfig's godoc listed two of its three refusal causes. * review: silkd checks an empty lsp manifest once, where argv is indexed read_manifest and its only caller both rejected an empty argv. * docs: sync the reference with the code - sandboxd-api: a reload also answers 409 when it drops a pool's egress policy that live claims use. - deploy: the tenants DDL carries the DEFERRABLE unique constraint the node creates, so a table created ahead of time accepts a token swap in one PUT. - egress: the golden stamp also records the image id and a sized pool's disk. - sdk-python, README, index: the claim env and the tenant verbs are Go-only, like pool retuning. - README: the credentials e2e driver, the guestserver helper and the mypy gate were missing from their lists. - Go SDK: SetPools says an omitted pool loses its golden once idle. * build: the android flavor's local silkd default follows the release The fallback stayed at 0.1.13 through three release bumps; CI passes the pinned carrier, so only a local build took the old daemon. * ci: a PR that edits the silkd or boot-init gate runs it Both workflows listed their own file under push only, unlike the shell, python and sandboxd gates. * fix(pool): a dropped pool keeps its golden while a claim clones from it A claim that missed the warm pool cloned from the pool's golden with no hold on it, so a PUT /v1/pools that dropped the key in that window renamed and deleted the directory under the clone and the claim failed. The pool now counts those clones and is dropped once they are done, like a build or a refill in flight. A removed pool no longer hands its golden to new claims: they cold-boot, as they do once the pool is gone, so a busy key cannot keep it alive.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #333.
Problem
A pool removed from the desired set left its golden on disk forever:
goldens/<key hash>/plusgoldens/<key hash>.stamp. Every image roll leaked one multi-GB snapshot per retired key, and a start whose pool set no longer held a key never cleaned it up either.Change
m.poolsin two places,applyPoolsandrefillOnce, and both now calldropPool. Underm.muit removes the stamp and renames the golden to a*.tmpsibling; a goroutine then deletes the renamed directory and logs one line naming the pool.adoptGoldenruns under the same lock and needs both the stamp and the directory. So a key that is re-added at once either finds nothing and rebuilds, or never sees the golden at all. It cannot adopt a directory that is being deleted.Reconcilealready removedgoldens/*.tmp. That glob becomes one pass overgoldens/that removes every entry that is not a pooled key's directory or stamp, and logs one line per orphan golden. It covers keys dropped from the config between restarts, goldens left by earlier versions, and a delete that a crash interrupted.sandboxd-api.md(PUT /v1/pools) anddeploy.mdstate that an omitted pool's golden is deleted once the pool is idle, thatwarm: 0keeps it, and that a start deletes any golden no pool uses.Answers to the issue's questions
warm: 0is the documented way to keep a key registered, so an omitted key is deleted at once, with no grace period.Not changed
ReloadConfigandcheckImagesOnce. A golden they retire is rebuilt at the same path (exportGoldenremoves the old directory before the rename), so it is not a leak.Cost and known limits
m.mu. The delete of the large files runs off the lock.Verification
sandboxd/pool:TestSetPoolsRemovalDeletesTheIdlePoolsGolden: the directory and stamp are gone whenSetPoolsreturns, and no*.tmpis left.TestAReaddedPoolRebuildsTheGoldenItsRemovalDeleted: remove then re-add cold-boots a second golden and refills.TestARemovedPoolKeepsItsGoldenUntilItSettles(build,refill): the golden stays while a build or refill is in flight and goes whenrefillOncedrops the pool.TestSetPoolsWarmZeroKeepsTheGolden: a key kept atwarm: 0keeps its original golden.TestReconcileSweepsGoldensNoPoolUses: an orphan directory, its stamp and a staging directory are removed; the pooled golden is kept and adopted.*.tmp,warm: 0also drops, and a plaindeleteat each of the two call sites.go test -race ./...insandboxd: all packages ok on macOS and on Linux (golang:1.27.1container, non-root). The new tests pass 300 runs with-raceon Linux.make go-lint:0 issues.×10 (5 modules × linux/darwin), no fmt diff.asl -forwarder=false ./...: 0 findings on darwin and linux.Not verified: no run on hardware. The case worth one check there is a node whose
data_dirand cocoon run directory are on different filesystems, where a clone's memory file is a symlink into the golden.