Repository navigation
Fix card 308, orchestration and join races, cluster freeze with nodes down, clock skew ordering, log noise - #1229
Conversation
…onfig files found at start are discovered, 30s at most (card 308) A daemon serves its api before the object monitors fill the status data the object listings read: right after a start, or a join restarting the daemon, the listings answered empty, and om took an empty expansion for "object not found" without falling back to the local files. discover now tracks the objects of the configuration files the initial walk found, until the status of each is known or its configuration failed to load, 30s at most, and raises a "discovered" flag. Until then the object paths, objects, instances and resources listings, and the object actions of a node without an instance, answer a 503 "Starting" problem, which om takes to read its local files, and the api declares it.
…card 308) The join restarts the daemon and said Joined once it answered, before it discovered the cluster object: a command right after the join did not find the cluster. The join now waits, 35s at most, for the daemon to list it, and warns when it does not.
…nd an orchestration once every peer cleared it The end of an orchestration, which ends the --wait of its requester, was announced as the accepting node cleared it, its peers clearing their part a heartbeat later: a next request asked at once to a peer was refused as "already in progress" (qatests 411.03 on debian 13). The accepting node now announces the end once every peer cleared the orchestration, 10s at most. A node whose part is done, the orchestration not cleared yet, holds a new request rather than refuse it, and takes it on once it clears, 10s at most, as v2 took a request once the instances were idle: the requester is answered it queued. An abort, or a newer request, refuses the request held.
… first instance node it hears A node without an instance of the object handed a request to the first node of a map iteration, a random one: two requests in a row were judged by two nodes. It now hands them to the first instance node, in name order, it hears, else the first of them all.
| if aID == uuid.Nil || bID == uuid.Nil { | ||
| return false | ||
| } |
There was a problem hiding this comment.
🟡 Medium - An ended equal-stamp monitor can hide an active orchestration from other peers
When one node has completed its orchestration and advertises GlobalExpect=None with stamp S while another node concurrently carries an active request at the same S, expectstamp.Newer returns false whenever either ID is nil. A third monitor can therefore select the ended entry over the active one and converge to None; the active node also treats that peer as converged, so the request runs only on the accepting node instead of being adopted cluster-wide.
Show fix
Define a total ordering for ended versus active entries at an equal stamp, preserving the active orchestration as newer than an ended marker (or retain the ended marker's orchestration ID for tie-breaking). Apply that ordering both when selecting the most recent peer and when checking convergence, and add a three-node test with an ended equal-stamp peer, an active equal-stamp peer, and a lagging third monitor.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Partially fixed in commit 340e4c6.
This issue remains because daemon/expectstamp/main.go:38-40 still makes equal-stamp comparisons with either nil ID return false, while daemon/imon/converge_global_expect.go:40 and :96 continue using Newer for adoption and convergence, so an active equal-stamp orchestration can still be treated as converged against an ended peer; only peer candidate selection was changed to NewerCandidate.
There was a problem hiding this comment.
Fixed in d33556e: the peer selection uses expectstamp.NewerCandidate, which prefers the orchestration running over the one ended at an equal stamp, the greater id between two running (TestALaggingNodeAdoptsTheRequestRunning, three nodes). expectstamp.Newer keeps a nil id as no newer and no older for the local comparisons only: a node never adopts back an orchestration it ended itself, and a peer having ended it is converged.
There was a problem hiding this comment.
Fixed in d33556e: the peer selection uses expectstamp.NewerCandidate, which prefers the orchestration running over the one ended at an equal stamp, the greater id between two running (TestALaggingNodeAdoptsTheRequestRunning, three nodes). expectstamp.Newer keeps a nil id as no newer and no older for the local comparisons only: a node never adopts back an orchestration it ended itself, and a peer having ended it is converged.
There was a problem hiding this comment.
Partially fixed in commit 8a01062.
This issue remains because daemon/expectstamp/main.go:38-40 still makes Newer return false whenever either ID is nil, and daemon/imon/converge_global_expect.go:40 uses it for adoption, so an equal-stamp ended local monitor can reject the active peer orchestration; NewerCandidate only fixes selecting the most recent peer.
There was a problem hiding this comment.
@AikidoSec ignore The case left needs two different requests accepted at once on two nodes with exactly the same stamp, one of them ended on a node before it sees the other. A node never adopts back an orchestration it ended itself, which Newer keeps by taking a nil id as no newer and no older; telling an ended orchestration from another one of the same stamp would mean keeping the id of ended orchestrations, which changes what an ended monitor carries across the cluster, for a case the stamps after every stamp seen and the deterministic forwarding make rare. Left as is.
There was a problem hiding this comment.
✅ Based on your feedback, we ignored this issue because of the following reason:
@AikidoSec ignore The case left needs two different requests accepted at once on two nodes with exactly the same stamp, one of them ended on a node before it sees the other. A node never adopts back an orchestration it ended itself, which Newer keeps by taking a nil id as no newer and no older; telling an ended orchestration from another one of the same stamp would mean keeping the id of ended orchestrations, which changes what an ended monitor carries across the cluster, for a case the stamps after every stamp seen and the deterministic forwarding make rare. Left as is.
| if mostRecentNode == "" || expectstamp.NewerCandidate(nodeTime, data.OrchestrationID, mostRecentUpdated, t.nodeMonitor[mostRecentNode].OrchestrationID) { | ||
| mostRecentNode = nodename | ||
| mostRecentUpdated = nodeTime | ||
| } |
There was a problem hiding this comment.
🟡 Medium - A newer ended node expectation can be hidden by an older active one
When one peer has cleared an orchestration and advertises GlobalExpect=None at a newer stamp while another still advertises the older active request, convergence considers only the active candidate and ignores the newer ended stamp. A lagging node can therefore keep or adopt the stale orchestration and run it after another node has already ended it, instead of converging to the newer ended state.
Show fix
Compare ended None markers and active expectations through the same total ordering, including timestamp precedence, rather than discarding the ended candidate whenever an active candidate exists. Ensure convergence and isConvergedGlobalExpect use that ordering and add a three-node test with an older active stamp, newer ended stamp, and lagging monitor.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Partially fixed in commit 8a01062.
This issue remains because daemon/nmon/converge_global_expect.go:41 routes a newer ended marker to alignEndedGlobalExpect, but that function at lines 83-86 refuses to change any local active GlobalExpect, and isConvergedGlobalExpect at line 109 then considers the older local stamp converged against the newer peer marker, allowing the stale orchestration to continue.
There was a problem hiding this comment.
@AikidoSec ignore A lagging node no longer adopts a request older than one a peer ended since. The case left is a node carrying such an older request already: it missed the whole newer node orchestration, accepted and ended while it did not see it, and runs its own to its end, a node freeze or thaw ending on its own. Abandoning a local orchestration mid-action on a newer ended marker risks leaving the node in an action success state with no global expect, for a case that needs a missed node orchestration. Left as is.
There was a problem hiding this comment.
✅ Based on your feedback, we ignored this issue because of the following reason:
@AikidoSec ignore A lagging node no longer adopts a request older than one a peer ended since. The case left is a node carrying such an older request already: it missed the whole newer node orchestration, accepted and ended while it did not see it, and runs its own to its end, a node freeze or thaw ending on its own. Abandoning a local orchestration mid-action on a newer ended marker risks leaving the node in an action success state with no global expect, for a case that needs a missed node orchestration. Left as is.
| clusterName, secret := clusterNameAndSecret(fetchObjectConfigData[naming.Cluster]) | ||
| certNames := func(f string) (bool, error) { | ||
| return certFileNames(f, hostname.Hostname(), clusterName, secret) | ||
| } | ||
| file, timestamp = t.waitCertNamingNode(ctx, file, timestamp, fetch, certNames, certReissueInterval, certReissueTimeout) |
There was a problem hiding this comment.
🔵 Low - Incomplete fetched cluster configuration can crash the join
When the fetched cluster configuration lacks a non-empty cluster name or secret, the new certificate check passes empty values to the crypto implementation, whose validation panics instead of returning an error. A malformed certificate payload is likewise treated as "not named" and, after the bounded wait, installed anyway, so the join can terminate abnormally or leave the newly joined daemon with an unusable certificate.
Show fix
Validate the fetched cluster name and secret before constructing the alternate decrypter and return a normal join error for invalid configuration. Propagate certificate-decode errors separately from a valid certificate that does not name the node; do not install a malformed or undecryptable certificate after the timeout.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Partially fixed in commit 2dc398d.
This issue remains because core/omcmd/cluster_join.go:321-324 still treats an empty fetched cluster name or secret as a reason to install SecCert unchanged, while waitCertNamingNode in core/omcmd/cluster_join_cert.go:43-68 discards certificate-decryption/parse errors and installs the last malformed certificate after its timeout; the added guard prevents the empty-value panic but does not return a join error or reject unusable certificate data.
There was a problem hiding this comment.
Partially fixed in commit 2dc398d.
This issue remains because core/omcmd/cluster_join.go:320-324 still accepts an empty cluster name or secret and installs the fetched certificate, while lines 326-329 pass certFileNames errors through waitCertNamingNode, which ignores them and returns the malformed/undecryptable certificate after timeout; the empty-value panic is guarded, but invalid certificates are still installed.
There was a problem hiding this comment.
@AikidoSec ignore The panic is fixed. Installing the certificate as fetched, when it can not be read or does not name the node in time, is what the join did before this change: the wait only adds a chance to install the one reissued. A certificate this node can not read is not malformed for the cluster, which reads it with its own secret, and refusing the join for it would leave the node out of a cluster that would serve it the reissued one minutes later through discover; nmon warns of a certificate not naming the node meanwhile. Left as is.
There was a problem hiding this comment.
✅ Based on your feedback, we ignored this issue because of the following reason:
@AikidoSec ignore The panic is fixed. Installing the certificate as fetched, when it can not be read or does not name the node in time, is what the join did before this change: the wait only adds a chance to install the one reissued. A certificate this node can not read is not malformed for the cluster, which reads it with its own secret, and refusing the join for it would leave the node out of a cluster that would serve it the reissued one minutes later through discover; nmon warns of a certificate not naming the node meanwhile. Left as is.
2dc398d to
d1e0710
Compare
| speaker = local.Speaker | ||
| } | ||
| s, ok := collectors[speaker] | ||
| if speaker != localhost && (!ok || s.State == "") { |
There was a problem hiding this comment.
🟡 Medium - Collector jobs run while the selected speaker cannot send
When a node is a speaker-candidate following a peer whose collector state is warning or speaker-warning, the scheduler treats that non-empty state as sufficient knowledge and returns joinable unless UnreachableSince is set. Those states explicitly mean the peer has no usable client or credentials, so RequireCollector jobs run and fail instead of being held until a usable speaker exists.
| if speaker != localhost && (!ok || s.State == "") { | |
| if speaker != localhost && (!ok || s.State != "speaker") { |
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
@AikidoSec ignore Each node pushes its stats, inventories and checks with its own collector client and credentials: a speaker in warning, its client missing or its credentials refused, says nothing of the pushes of this node, which a node without a usable client of its own already holds through its own warning state. The hold added here is for the collector found unreachable by the speaker, which pings it; holding the jobs of every node on the state of the speaker client would stop pushes that work, and so would holding them while the speaker starts or stops. Left as is.
There was a problem hiding this comment.
✅ Based on your feedback, we ignored this issue because of the following reason:
@AikidoSec ignore Each node pushes its stats, inventories and checks with its own collector client and credentials: a speaker in warning, its client missing or its credentials refused, says nothing of the pushes of this node, which a node without a usable client of its own already holds through its own warning state. The hold added here is for the collector found unreachable by the speaker, which pings it; holding the jobs of every node on the state of the speaker client would stop pushes that work, and so would holding them while the speaker starts or stops. Left as is.
b800d5e to
d6dea41
Compare
…enew as the speaker only once rejoined
A joining node fetched system/sec/cert right after the join, before the
speaker reissued it for the cluster nodes: the node presented a
certificate not naming it, which its peers refused, until it fetched the
one reissued. It now fetches it last, again until it names the node, 15s
at most, unless the cluster does not issue it for its nodes (alt_names
without {clusternodes}, acme).
A daemon starting elects itself the speaker until it hears its peers, and
renewed the certificates then: a node just joined reissued the certificate
of the cluster. The speaker now renews once its node rejoined its peers,
the renewals asked meanwhile left pending.
…clock skew A node adopts the global expect of a peer stamped after its own, the stamp of the clock of the node accepting it: a request accepted on a node whose clock is behind the node that accepted the one before was stamped before it, the peers kept the one before, and the node adopted it back, its request accepted and never run, its --wait hanging. The instance and node monitors now stamp a global expect after every stamp they saw. The instance monitor checks its configuration against the local time it saw the global expect at, not the stamp.
…ailed heartbeats The daemon configures every heartbeat again at each retry of the failed ones, every 30s, and the relay, unicast and disk heartbeats warned each time about the timeout they raise to their floor: a relay with an interval of 60s and no timeout logged its readjust 120 times an hour. The warning is now logged once per heartbeat, again only when it changes, and at the debug level otherwise.
A request refused authentication was logged at the error level, every time: a metrics scraper left without credentials, polling every 15s, logged 240 errors an hour on each node. The refusal is the client's: it is now logged as a warning the first time per client address, the repeats at the debug level, counted, and summed up every 10 minutes. The reason is not of the key, a client changing its username at each request escaping the pacing otherwise, and the clients remembered are bounded.
…ds it unreachable The scheduled pushes, checks and sysreports each failed the same way while the collector was unreachable, each run logged as two errors. The collector speaker now records the time its daemon status and ping sends start failing before any answer, as on a name not resolved or a connection refused, in unreachable_since of its collector status, and clears it once the collector answers. The scheduler of every node holds the jobs requiring the collector while the speaker it follows finds it unreachable, and runs them again once it answers: the stats push sends every stat since the last one pushed.
…e other, and leave it on an abort
A node whose freeze is done waits in the frozen state for its
orchestration to end. A thaw adopted meanwhile, as one asked right after
the freeze --wait of the accepting node returned, found no way from there
("don't know how to orchestrate unfrozen from frozen"), and the node
stayed frozen for good, an abort not helping either.
A thaw asked in the frozen state, or a freeze in the unfrozen state, now
goes back to idle and on, and an abort leaves these states for idle.
…it, 30s at most Each node announced the end of a cluster orchestration as its own part was done, and the session table of the accepting node ended it on that: om cluster freeze --wait returned with the peers still freezing, and a thaw asked then reached them mid-way. The end of a freeze, a thaw or an abort of the cluster is now held back until no live peer carries the orchestration, each peer dropping it once its own part is done, as v2 waited for every live node to be frozen. The peers starting, rejoining, in maintenance, shutting down, gone or forgotten are not waited for. A peer failing its part ends it failed, naming it, and so does a peer not done within 30s. A new orchestration displacing it ends it aborted. A drain, and an upgrade the command waits for on its own, end as before.
…or rejoining, as v2 did A cluster freeze or thaw was refused while any node was not rankable: with a node stopped (maintenance), shutting down, or never heard of since this daemon started (init), and while this node rejoined, the cluster could not be frozen at all. The accepting node also waited for every peer to adopt the request, which a stopped node never does. The request now goes on without the peers in maintenance, shutting down, starting or rejoining, and is taken while this node starts or rejoins, to run once it rejoined. The accepting node does not wait for these peers to adopt it: one rejoining adopts it then, and one that missed a cluster freeze catches it as it rejoins. An upgrade still wants every node.
…mand The runner is running as it starts its source, the run counted as the source runs: the count read as the state turned running was 1 at times, failing the race test on a loaded runner.
The drain freezes the node, stops the object and ends idle, in 2s: short of it at times with -race on a loaded runner, as on the race-test runs of 2026-11-14 and 2026-11-16.
…y unprovisioned A volume purged waited for the services using it to be unprovisioned, and went on as one was deleting its instance: purged first, the volume was gone with the configuration of the service naming it still there, kept for good by a delete failing, and the order of the two raced (qatests 512.07.3, ubuntu 22.04). The volume now waits, in a purge, for the child instances still purged or deleted to be gone, their status deleted; a child whose purge or delete failed holds nothing. An unprovision still waits for the unprovision only.
| case v.GlobalExpect == nil, v.State != nil, v.LocalExpect != nil: | ||
| return false |
There was a problem hiding this comment.
🟡 Medium - Daemon shutdown can bypass a held object orchestration end
When an object orchestration has completed locally but a peer still carries its ID, the daemon shutdown path submits both State and LocalExpect, causing holdRequest to return false instead of queuing the request. The monitor then publishes the earlier orchestration's end before the peer clears it and accepts shutdown, so the earlier operation can be reported complete while it is still active on a peer.
Show fix
Apply the held-end barrier to shutdown requests too: queue a candidate request with State or LocalExpect while orchestrationEnding is present, then replay it only after the previous end is published. Keep only abort/cleanup updates that must cancel the held end outside this queue, and add a shutdown-after-local-completion/peer-still-active test.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
@AikidoSec ignore A daemon shutdown is not an orchestration request: it must not wait up to 10s per object behind a peer clearing the one before. The local part of that orchestration is done, and the monitor stopping announces its held end anyway (imon main.go, "its peers clearing it are not waited for any more"), so the shutdown announces it a moment earlier, as the stop would.
There was a problem hiding this comment.
✅ Based on your feedback, we ignored this issue because of the following reason:
@AikidoSec ignore A daemon shutdown is not an orchestration request: it must not wait up to 10s per object behind a peer clearing the one before. The local part of that orchestration is done, and the monitor stopping announces its held end anyway (imon main.go, "its peers clearing it are not waited for any more"), so the shutdown announces it a moment earlier, as the stop would.
| o.Error = errS | ||
| o.EndedAt = &now | ||
| delete(failures, id) | ||
| delete(awaitingGone, id) | ||
| notifyEnded() | ||
| purgeOrchestrations() | ||
| } |
There was a problem hiding this comment.
🟡 Medium - Purge completion events cancel the wait for leftover instances
When the last participating monitor leaves a delete or purge while a nonparticipant still retains instance data, leave correctly puts the orchestration in awaitingGone. The subsequent ObjectOrchestrationEnd emitted as the local object monitor is deleted unconditionally removes that wait and marks the orchestration ended, allowing --wait to succeed while the retained instance remains.
Show fix
| o.Error = errS | |
| o.EndedAt = &now | |
| delete(failures, id) | |
| delete(awaitingGone, id) | |
| notifyEnded() | |
| purgeOrchestrations() | |
| } | |
| if _, waiting := awaitingGone[id]; waiting && state == StateSucceeded { | |
| o.State = StateRunning | |
| return | |
| } | |
| o.Error = errS | |
| o.EndedAt = &now | |
| delete(failures, id) | |
| delete(awaitingGone, id) | |
| notifyEnded() |
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in 0655baf: on the node that accepted it, the success a delete or a purge announces no longer ends it while a participant is left or an instance of the object remains. It ends there as on the other nodes, once the participants left and the object is gone, failed past 10s. Covered by TestTheEndOfAPurgeAnnouncedDoesNotSkipTheWaitForTheObjectGone, the end announced both before and after the accepting monitor is seen deleted.
| if mon == nil { | ||
| return false |
There was a problem hiding this comment.
🟡 Medium - Config fetch can resurrect an object after its monitor is deleted
After a purge/delete removes the serving node's instance monitor, configBeingDeleted returns false even if the config file or config record is still present, so the instance-config endpoint can serve it and the object endpoint can proxy it. A peer fetch in the documented old-omon/recreated-config window can therefore reinstall an instance after the deletion orchestration has left, bypassing the gone check and resurrecting the object.
Show fix
Keep a deletion/purge tombstone or equivalent removal state until the object config and all in-flight fetches are settled, and reject both local and proxied config requests while that tombstone is active. Clear it only after the deletion session finishes; add a test where the local monitor is deleted, the config remains, and a peer fetch is refused rather than installed.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
@AikidoSec ignore The window does not occur on a delete or a purge: the instance monitor is deleted after its config file, the instance delete removing the file and the config manager cancelling the monitor once it sees it gone, so a node whose monitor is deleted has no config left to serve or to proxy to. The tombstone covering a node absent during a purge is a separate change, after this PR.
There was a problem hiding this comment.
✅ Based on your feedback, we ignored this issue because of the following reason:
@AikidoSec ignore The window does not occur on a delete or a purge: the instance monitor is deleted after its config file, the instance delete removing the file and the config manager cancelling the monitor once it sees it gone, so a node whose monitor is deleted has no config left to serve or to proxy to. The tombstone covering a node absent during a purge is a separate change, after this PR.
…t, the nodes down not waited for A delete or a purge ended as soon as the nodes seen running it had all left, their monitors deleted: a node whose instance came back just before, as one fetching the config again after a config update, was no participant yet, and purge --wait returned with its instance left, the object still listed (qatests 308.03.2, debian 12 and rhel 10, reproduced on main). The orchestration now ends once no instance of the object remains, its config, status or monitor, on the nodes but the ones in maintenance, starting, rejoining or shutting down, which keep the data of their instance and are not waited for. Past 10s it ends failed, naming the nodes an instance remains on. A participant failing still ends it failed at once.
…he config of an object being deleted A peer installing the config of an object has an instance coming, which the other nodes learn of only as its config reaches them by heartbeat: a purge accepted meanwhile went on without it, the leader first. The node serving a config to a peer now announces it, and its instance monitor counts the peer as starting until its monitor is seen out of init, 10s at most: a delete or a purge waits for it, and so does the leader of a purge in wait non-leader. A node whose monitor purges or deletes the object answers 404 to a request for its config, as v2 refused to fetch a config then: a peer installing it would bring back an instance the orchestration deletes.
…r a peer An orchestration done here, its id cleared, its end held back while a peer still runs it: a request received then was accepted at once, which published the end before, the --wait of its requester told it ended with the peer still running it, and the two orchestrations ran at once. The request is now held while that end is held back, and taken on once it is published, the peer clearing it or the wait for it timing out.
Fix card 308 and the races and noise the QA runs of #1228 showed: an orchestration refused right after the
--waitof the one before, a cluster freeze--waitreturning before the peers froze, a cluster that could not be frozen with a node down, a joining node presenting a certificate not naming it, requests ordered by clocks that may disagree, and three logs repeated as errors or warnings.Changes
om cluster joinwaits, 35s at most, for the daemon to list the cluster object before printing "Joined".--waitof the one before (qatests 411.03 on debian 13). The end of an orchestration was announced as the accepting node cleared it, its peers clearing their part a heartbeat later.om cluster freeze --waitreturned with the peers still freezing, and a thaw asked then reached a peer still in the frozen state its freeze left it in, which had no way out ("don't know how to orchestrate unfrozen from frozen"), the node frozen for good. The end is now held back until no live peer carries the orchestration, failed when a peer fails its part or is not done within 30s, aborted when a new orchestration displaces it; drain and upgrade end as before. A freeze or a thaw asked in the success state of the other goes back to idle first, and an abort leaves these states.purge --waitreturned with an instance left: a node whose instance came back just before, fetching the config again after a config update, was no participant yet as the others left, and the leader, not seeing it, purged first. A delete or a purge now ends once no instance of the object remains, the nodes in maintenance, starting, rejoining or shutting down not waited for, 10s at most then failed, naming the nodes. A node serving the config to a peer counts it as starting, 10s at most, for the purge to wait for it, and a node purging or deleting an object answers 404 to a request for its config.unreachable_sincein its collector status when its pings fail before any answer, and the scheduler of every node holds the jobs requiring the collector until it answers again: the stats push sends every stat since the last one pushed.Tests
om cluster config getrun 40 times right after each, 200/200; the api answers 503 "Starting" for 2.5s, then lists.cap --waitof a flex service, 30/30; 20 pairs alternating the instance nodes, 20/20; 10 provisions then purges of a failover service, 10/10.cluster freeze --wait/cluster thaw --wait: every node frozen or thawed as each wait returned, each request accepted after both peers ended the one before. With a node daemon stopped: the freeze freezes the live nodes in 1s, the stopped node freezes as it rejoins; a freeze asked while the accepting node rejoins runs once rejoined.Known left
A node absent during a purge or a delete (maintenance, down) brings the object back when it returns, its config fetched by the peers, as in v2: a catch-up blocking it is planned in its own PR.
A node down during a cluster thaw rejoins frozen: a missed freeze is caught up on rejoin, a missed thaw is not, as in v2.
After
purge --waitreturns, the object stays listed for about 0.3s on a node without an instance, where acreateof the same name asked at once is refused as already existing.