Skip to content

Fix card 308, orchestration and join races, cluster freeze with nodes down, clock skew ordering, log noise - #1229

Merged
cvaroqui merged 18 commits into
opensvc:mainfrom
cvaroqui:dev
Nov 17, 2026
Merged

cvaroqui merged 18 commits into
opensvc:mainfrom
cvaroqui:dev

Conversation

@cvaroqui

@cvaroqui cvaroqui commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Fix card 308 and the races and noise the QA runs of #1228 showed: an orchestration refused right after the --wait of the one before, a cluster freeze --wait returning before the peers froze, a cluster that could not be frozen with a node down, a joining node presenting a certificate not naming it, requests ordered by clocks that may disagree, and three logs repeated as errors or warnings.

Changes

  • Card 308, "object not found" right after a daemon start. The daemon serves its api before the object monitors fill the status data the listings read: an object listing answered empty then, which om took for "object not found", with no fallback to the local files (qatests 015.02.4, 015.03.3).
    • discover tracks the objects of the configuration files the initial walk found until the status of each is known or its configuration failed to load, 30s at most.
    • Until then, the object paths, objects, instances and resources listings, and the object actions of a node without an instance, answer a 503 "Starting" problem, declared in the api. om takes it to read its local files.
    • om cluster join waits, 35s at most, for the daemon to list the cluster object before printing "Joined".
  • An orchestration refused as "already in progress" right after the --wait of the one before (qatests 411.03 on debian 13). The end of an orchestration was announced as the accepting node cleared it, its peers clearing their part a heartbeat later.
    • The accepting node announces the end once every peer cleared the orchestration, 10s at most.
    • A node whose part is done, the orchestration not cleared yet, holds a new request rather than refuse it, and takes it on once it clears, 10s at most, as v2 took a request once the instances were idle. The requester is answered it queued. An abort, or a newer request, refuses the request held.
    • A node without an instance hands the object actions to the first instance node, in name order, it hears, rather than to the first of a map iteration.
  • Node orchestrations (cluster freeze, thaw, abort).
    • Each node announced the end as its own part was done: om cluster freeze --wait returned with the peers still freezing, and a thaw asked then reached a peer still in the frozen state its freeze left it in, which had no way out ("don't know how to orchestrate unfrozen from frozen"), the node frozen for good. The end is now held back until no live peer carries the orchestration, failed when a peer fails its part or is not done within 30s, aborted when a new orchestration displaces it; drain and upgrade end as before. A freeze or a thaw asked in the success state of the other goes back to idle first, and an abort leaves these states.
    • A cluster freeze or thaw was refused while a node was in maintenance, shutting down, never heard of, or this node rejoining: a cluster with a node down could not be frozen. It now goes on without these nodes, as v2 did, the accepting node not waiting for them to adopt it; a node that missed a freeze catches it as it rejoins. An upgrade still wants every node.
  • Purges (qatests 308, 512).
    • purge --wait returned with an instance left: a node whose instance came back just before, fetching the config again after a config update, was no participant yet as the others left, and the leader, not seeing it, purged first. A delete or a purge now ends once no instance of the object remains, the nodes in maintenance, starting, rejoining or shutting down not waited for, 10s at most then failed, naming the nodes. A node serving the config to a peer counts it as starting, 10s at most, for the purge to wait for it, and a node purging or deleting an object answers 404 to a request for its config.
    • A volume purged went on as the service using it was unprovisioned, before its instance was deleted: it now waits for the service instances to be deleted.
  • A joining node presented a certificate not naming it. It fetched system/sec/cert right after the join, before the speaker reissued it for the cluster nodes, and its peers refused its certificate.
    • The join fetches it last, again until it names the node, 15s at most, unless the cluster does not issue it for its nodes (alt_names without {clusternodes}, acme).
    • A daemon starting elects itself the speaker until it hears its peers, and renewed the certificates then: the speaker renews once its node rejoined its peers.
  • Requests ordered by clocks that may disagree. A node adopts the global expect of a peer stamped after its own, the stamp of the clock of the accepting node: a request accepted on a node whose clock was behind the one that accepted the request before was stamped before it, adopted back by its node, accepted and never run. The instance and node monitors stamp a global expect after every stamp they saw, two requests stamped the same, as accepted at once on two nodes, ordered by their orchestration id on every node.
  • Logs.
    • The readjusted timeout of a heartbeat was warned at each retry of the failed heartbeats, every 30s: warned once, again only when it changes.
    • A refused authentication was an error at each request, 240 an hour per node for a metrics scraper without credentials: a warning per client address and reason, the repeats counted and summed up every 10 minutes.
    • The jobs feeding the collector each failed, logged twice as errors, while the collector was unreachable. The speaker records unreachable_since in its collector status when its pings fail before any answer, and the scheduler of every node holds the jobs requiring the collector until it answers again: the stats push sends every stat since the last one pushed.

Tests

  • Unit tests: the initial discovery settling (objects discovered, failed, timeout); the 503 of the paths and the path expansion; the join waiting for the certificate (named on the third fetch, timeout, not waited for a fixed alt_names or acme); the end held until the peers clear, and on timeout; a request held, taken on, refused on timeout, by an abort and by a newer request; the forwarding node; the stamps (helper, a request accepted with the clock behind, not adopted back); the warning once; the paced refusals; the collector joinable rule.
  • Checked live on the dev cluster (ubuntu 24.04, 3 nodes):
    • 5 restarts of a node with 116 objects: om cluster config get run 40 times right after each, 200/200; the api answers 503 "Starting" for 2.5s, then lists.
    • From a node without an instance, 30 pairs of cap --wait of a flex service, 30/30; 20 pairs alternating the instance nodes, 20/20; 10 provisions then purges of a failover service, 10/10.
    • With a node clock 20s behind, starts and stops alternating the accepting node, 3/3 rounds.
    • 20 pairs of cluster freeze --wait / cluster thaw --wait: every node frozen or thawed as each wait returned, each request accepted after both peers ended the one before. With a node daemon stopped: the freeze freezes the live nodes in 1s, the stopped node freezes as it rejoins; a freeze asked while the accepting node rejoins runs once rejoined.
    • The readjust warning once per daemon start; the metrics refusals as one warning per client; the collector, unresolvable on the dev cluster, recorded unreachable by the speaker and the collector jobs held on every node.
  • Not checked live: a real join, which the QA test 015 covers.

Known left

A node absent during a purge or a delete (maintenance, down) brings the object back when it returns, its config fetched by the peers, as in v2: a catch-up blocking it is planned in its own PR.

A node down during a cluster thaw rejoins frozen: a missed freeze is caught up on rejoin, a missed thaw is not, as in v2.

After purge --wait returns, the object stays listed for about 0.3s on a node without an instance, where a create of the same name asked at once is refused as already existing.

…onfig files found at start are discovered, 30s at most (card 308)

A daemon serves its api before the object monitors fill the status data
the object listings read: right after a start, or a join restarting the
daemon, the listings answered empty, and om took an empty expansion for
"object not found" without falling back to the local files.

discover now tracks the objects of the configuration files the initial
walk found, until the status of each is known or its configuration failed
to load, 30s at most, and raises a "discovered" flag. Until then the
object paths, objects, instances and resources listings, and the object
actions of a node without an instance, answer a 503 "Starting" problem,
which om takes to read its local files, and the api declares it.
…card 308)

The join restarts the daemon and said Joined once it answered, before it
discovered the cluster object: a command right after the join did not find
the cluster. The join now waits, 35s at most, for the daemon to list it,
and warns when it does not.
…nd an orchestration once every peer cleared it

The end of an orchestration, which ends the --wait of its requester, was
announced as the accepting node cleared it, its peers clearing their part
a heartbeat later: a next request asked at once to a peer was refused as
"already in progress" (qatests 411.03 on debian 13).

The accepting node now announces the end once every peer cleared the
orchestration, 10s at most. A node whose part is done, the orchestration
not cleared yet, holds a new request rather than refuse it, and takes it
on once it clears, 10s at most, as v2 took a request once the instances
were idle: the requester is answered it queued. An abort, or a newer
request, refuses the request held.
… first instance node it hears

A node without an instance of the object handed a request to the first
node of a map iteration, a random one: two requests in a row were judged
by two nodes. It now hands them to the first instance node, in name order,
it hears, else the first of them all.
@cvaroqui cvaroqui changed the title Fix card 308, orchestration and join races, clock skew ordering, log noise Fix card 308, orchestration and join races, cluster freeze with nodes down, clock skew ordering, log noise Nov 16, 2026
Comment on lines +38 to +40
if aID == uuid.Nil || bID == uuid.Nil {
return false
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - An ended equal-stamp monitor can hide an active orchestration from other peers

When one node has completed its orchestration and advertises GlobalExpect=None with stamp S while another node concurrently carries an active request at the same S, expectstamp.Newer returns false whenever either ID is nil. A third monitor can therefore select the ended entry over the active one and converge to None; the active node also treats that peer as converged, so the request runs only on the accepting node instead of being adopted cluster-wide.

Show fix

Define a total ordering for ended versus active entries at an equal stamp, preserving the active orchestration as newer than an ended marker (or retain the ended marker's orchestration ID for tie-breaking). Apply that ordering both when selecting the most recent peer and when checking convergence, and add a three-node test with an ended equal-stamp peer, an active equal-stamp peer, and a lagging third monitor.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 340e4c6.

This issue remains because daemon/expectstamp/main.go:38-40 still makes equal-stamp comparisons with either nil ID return false, while daemon/imon/converge_global_expect.go:40 and :96 continue using Newer for adoption and convergence, so an active equal-stamp orchestration can still be treated as converged against an ended peer; only peer candidate selection was changed to NewerCandidate.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d33556e: the peer selection uses expectstamp.NewerCandidate, which prefers the orchestration running over the one ended at an equal stamp, the greater id between two running (TestALaggingNodeAdoptsTheRequestRunning, three nodes). expectstamp.Newer keeps a nil id as no newer and no older for the local comparisons only: a node never adopts back an orchestration it ended itself, and a peer having ended it is converged.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d33556e: the peer selection uses expectstamp.NewerCandidate, which prefers the orchestration running over the one ended at an equal stamp, the greater id between two running (TestALaggingNodeAdoptsTheRequestRunning, three nodes). expectstamp.Newer keeps a nil id as no newer and no older for the local comparisons only: a node never adopts back an orchestration it ended itself, and a peer having ended it is converged.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 8a01062.

This issue remains because daemon/expectstamp/main.go:38-40 still makes Newer return false whenever either ID is nil, and daemon/imon/converge_global_expect.go:40 uses it for adoption, so an equal-stamp ended local monitor can reject the active peer orchestration; NewerCandidate only fixes selecting the most recent peer.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AikidoSec ignore The case left needs two different requests accepted at once on two nodes with exactly the same stamp, one of them ended on a node before it sees the other. A node never adopts back an orchestration it ended itself, which Newer keeps by taking a nil id as no newer and no older; telling an ended orchestration from another one of the same stamp would mean keeping the id of ended orchestrations, which changes what an ended monitor carries across the cluster, for a case the stamps after every stamp seen and the deterministic forwarding make rare. Left as is.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Based on your feedback, we ignored this issue because of the following reason:

@AikidoSec ignore The case left needs two different requests accepted at once on two nodes with exactly the same stamp, one of them ended on a node before it sees the other. A node never adopts back an orchestration it ended itself, which Newer keeps by taking a nil id as no newer and no older; telling an ended orchestration from another one of the same stamp would mean keeping the id of ended orchestrations, which changes what an ended monitor carries across the cluster, for a case the stamps after every stamp seen and the deterministic forwarding make rare. Left as is.

Comment on lines +31 to 34
if mostRecentNode == "" || expectstamp.NewerCandidate(nodeTime, data.OrchestrationID, mostRecentUpdated, t.nodeMonitor[mostRecentNode].OrchestrationID) {
mostRecentNode = nodename
mostRecentUpdated = nodeTime
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - A newer ended node expectation can be hidden by an older active one

When one peer has cleared an orchestration and advertises GlobalExpect=None at a newer stamp while another still advertises the older active request, convergence considers only the active candidate and ignores the newer ended stamp. A lagging node can therefore keep or adopt the stale orchestration and run it after another node has already ended it, instead of converging to the newer ended state.

Show fix

Compare ended None markers and active expectations through the same total ordering, including timestamp precedence, rather than discarding the ended candidate whenever an active candidate exists. Ensure convergence and isConvergedGlobalExpect use that ordering and add a three-node test with an older active stamp, newer ended stamp, and lagging monitor.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 8a01062.

This issue remains because daemon/nmon/converge_global_expect.go:41 routes a newer ended marker to alignEndedGlobalExpect, but that function at lines 83-86 refuses to change any local active GlobalExpect, and isConvergedGlobalExpect at line 109 then considers the older local stamp converged against the newer peer marker, allowing the stale orchestration to continue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AikidoSec ignore A lagging node no longer adopts a request older than one a peer ended since. The case left is a node carrying such an older request already: it missed the whole newer node orchestration, accepted and ended while it did not see it, and runs its own to its end, a node freeze or thaw ending on its own. Abandoning a local orchestration mid-action on a newer ended marker risks leaving the node in an action success state with no global expect, for a case that needs a missed node orchestration. Left as is.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Based on your feedback, we ignored this issue because of the following reason:

@AikidoSec ignore A lagging node no longer adopts a request older than one a peer ended since. The case left is a node carrying such an older request already: it missed the whole newer node orchestration, accepted and ended while it did not see it, and runs its own to its end, a node freeze or thaw ending on its own. Abandoning a local orchestration mid-action on a newer ended marker risks leaving the node in an action success state with no global expect, for a case that needs a missed node orchestration. Left as is.

Comment thread core/omcmd/cluster_join.go Outdated
Comment on lines +320 to +324
clusterName, secret := clusterNameAndSecret(fetchObjectConfigData[naming.Cluster])
certNames := func(f string) (bool, error) {
return certFileNames(f, hostname.Hostname(), clusterName, secret)
}
file, timestamp = t.waitCertNamingNode(ctx, file, timestamp, fetch, certNames, certReissueInterval, certReissueTimeout)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low - Incomplete fetched cluster configuration can crash the join

When the fetched cluster configuration lacks a non-empty cluster name or secret, the new certificate check passes empty values to the crypto implementation, whose validation panics instead of returning an error. A malformed certificate payload is likewise treated as "not named" and, after the bounded wait, installed anyway, so the join can terminate abnormally or leave the newly joined daemon with an unusable certificate.

Show fix

Validate the fetched cluster name and secret before constructing the alternate decrypter and return a normal join error for invalid configuration. Propagate certificate-decode errors separately from a valid certificate that does not name the node; do not install a malformed or undecryptable certificate after the timeout.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 2dc398d.

This issue remains because core/omcmd/cluster_join.go:321-324 still treats an empty fetched cluster name or secret as a reason to install SecCert unchanged, while waitCertNamingNode in core/omcmd/cluster_join_cert.go:43-68 discards certificate-decryption/parse errors and installs the last malformed certificate after its timeout; the added guard prevents the empty-value panic but does not return a join error or reject unusable certificate data.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 2dc398d.

This issue remains because core/omcmd/cluster_join.go:320-324 still accepts an empty cluster name or secret and installs the fetched certificate, while lines 326-329 pass certFileNames errors through waitCertNamingNode, which ignores them and returns the malformed/undecryptable certificate after timeout; the empty-value panic is guarded, but invalid certificates are still installed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AikidoSec ignore The panic is fixed. Installing the certificate as fetched, when it can not be read or does not name the node in time, is what the join did before this change: the wait only adds a chance to install the one reissued. A certificate this node can not read is not malformed for the cluster, which reads it with its own secret, and refusing the join for it would leave the node out of a cluster that would serve it the reissued one minutes later through discover; nmon warns of a certificate not naming the node meanwhile. Left as is.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Based on your feedback, we ignored this issue because of the following reason:

@AikidoSec ignore The panic is fixed. Installing the certificate as fetched, when it can not be read or does not name the node in time, is what the join did before this change: the wait only adds a chance to install the one reissued. A certificate this node can not read is not malformed for the cluster, which reads it with its own secret, and refusing the join for it would leave the node out of a cluster that would serve it the reissued one minutes later through discover; nmon warns of a certificate not naming the node meanwhile. Left as is.

@cvaroqui
cvaroqui force-pushed the dev branch 2 times, most recently from 2dc398d to d1e0710 Compare November 16, 2026 15:00
Comment thread daemon/scheduler/main.go
speaker = local.Speaker
}
s, ok := collectors[speaker]
if speaker != localhost && (!ok || s.State == "") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Collector jobs run while the selected speaker cannot send

When a node is a speaker-candidate following a peer whose collector state is warning or speaker-warning, the scheduler treats that non-empty state as sufficient knowledge and returns joinable unless UnreachableSince is set. Those states explicitly mean the peer has no usable client or credentials, so RequireCollector jobs run and fail instead of being held until a usable speaker exists.

Suggested change
if speaker != localhost && (!ok || s.State == "") {
if speaker != localhost && (!ok || s.State != "speaker") {

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AikidoSec ignore Each node pushes its stats, inventories and checks with its own collector client and credentials: a speaker in warning, its client missing or its credentials refused, says nothing of the pushes of this node, which a node without a usable client of its own already holds through its own warning state. The hold added here is for the collector found unreachable by the speaker, which pings it; holding the jobs of every node on the state of the speaker client would stop pushes that work, and so would holding them while the speaker starts or stops. Left as is.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Based on your feedback, we ignored this issue because of the following reason:

@AikidoSec ignore Each node pushes its stats, inventories and checks with its own collector client and credentials: a speaker in warning, its client missing or its credentials refused, says nothing of the pushes of this node, which a node without a usable client of its own already holds through its own warning state. The hold added here is for the collector found unreachable by the speaker, which pings it; holding the jobs of every node on the state of the speaker client would stop pushes that work, and so would holding them while the speaker starts or stops. Left as is.

@cvaroqui
cvaroqui marked this pull request as ready for review November 16, 2026 15:10
@cvaroqui
cvaroqui force-pushed the dev branch 2 times, most recently from b800d5e to d6dea41 Compare November 17, 2026 01:49
…enew as the speaker only once rejoined

A joining node fetched system/sec/cert right after the join, before the
speaker reissued it for the cluster nodes: the node presented a
certificate not naming it, which its peers refused, until it fetched the
one reissued. It now fetches it last, again until it names the node, 15s
at most, unless the cluster does not issue it for its nodes (alt_names
without {clusternodes}, acme).

A daemon starting elects itself the speaker until it hears its peers, and
renewed the certificates then: a node just joined reissued the certificate
of the cluster. The speaker now renews once its node rejoined its peers,
the renewals asked meanwhile left pending.
…clock skew

A node adopts the global expect of a peer stamped after its own, the stamp
of the clock of the node accepting it: a request accepted on a node whose
clock is behind the node that accepted the one before was stamped before
it, the peers kept the one before, and the node adopted it back, its
request accepted and never run, its --wait hanging.

The instance and node monitors now stamp a global expect after every
stamp they saw. The instance monitor checks its configuration against the
local time it saw the global expect at, not the stamp.
…ailed heartbeats

The daemon configures every heartbeat again at each retry of the failed
ones, every 30s, and the relay, unicast and disk heartbeats warned each
time about the timeout they raise to their floor: a relay with an
interval of 60s and no timeout logged its readjust 120 times an hour.
The warning is now logged once per heartbeat, again only when it
changes, and at the debug level otherwise.
A request refused authentication was logged at the error level, every
time: a metrics scraper left without credentials, polling every 15s,
logged 240 errors an hour on each node. The refusal is the client's: it
is now logged as a warning the first time per client address, the
repeats at the debug level, counted, and summed up every 10 minutes. The
reason is not of the key, a client changing its username at each request
escaping the pacing otherwise, and the clients remembered are bounded.
…ds it unreachable

The scheduled pushes, checks and sysreports each failed the same way while
the collector was unreachable, each run logged as two errors.

The collector speaker now records the time its daemon status and ping
sends start failing before any answer, as on a name not resolved or a
connection refused, in unreachable_since of its collector status, and
clears it once the collector answers. The scheduler of every node holds
the jobs requiring the collector while the speaker it follows finds it
unreachable, and runs them again once it answers: the stats push sends
every stat since the last one pushed.
…e other, and leave it on an abort

A node whose freeze is done waits in the frozen state for its
orchestration to end. A thaw adopted meanwhile, as one asked right after
the freeze --wait of the accepting node returned, found no way from there
("don't know how to orchestrate unfrozen from frozen"), and the node
stayed frozen for good, an abort not helping either.

A thaw asked in the frozen state, or a freeze in the unfrozen state, now
goes back to idle and on, and an abort leaves these states for idle.
…it, 30s at most

Each node announced the end of a cluster orchestration as its own part
was done, and the session table of the accepting node ended it on that:
om cluster freeze --wait returned with the peers still freezing, and a
thaw asked then reached them mid-way.

The end of a freeze, a thaw or an abort of the cluster is now held back
until no live peer carries the orchestration, each peer dropping it once
its own part is done, as v2 waited for every live node to be frozen. The
peers starting, rejoining, in maintenance, shutting down, gone or
forgotten are not waited for. A peer failing its part ends it failed,
naming it, and so does a peer not done within 30s. A new orchestration
displacing it ends it aborted. A drain, and an upgrade the command waits
for on its own, end as before.
…or rejoining, as v2 did

A cluster freeze or thaw was refused while any node was not rankable: with
a node stopped (maintenance), shutting down, or never heard of since this
daemon started (init), and while this node rejoined, the cluster could
not be frozen at all. The accepting node also waited for every peer to
adopt the request, which a stopped node never does.

The request now goes on without the peers in maintenance, shutting down,
starting or rejoining, and is taken while this node starts or rejoins, to
run once it rejoined. The accepting node does not wait for these peers to
adopt it: one rejoining adopts it then, and one that missed a cluster
freeze catches it as it rejoins. An upgrade still wants every node.
…mand

The runner is running as it starts its source, the run counted as the
source runs: the count read as the state turned running was 1 at times,
failing the race test on a loaded runner.
The drain freezes the node, stops the object and ends idle, in 2s: short
of it at times with -race on a loaded runner, as on the race-test runs of
2026-11-14 and 2026-11-16.
…y unprovisioned

A volume purged waited for the services using it to be unprovisioned, and
went on as one was deleting its instance: purged first, the volume was
gone with the configuration of the service naming it still there, kept for
good by a delete failing, and the order of the two raced (qatests
512.07.3, ubuntu 22.04). The volume now waits, in a purge, for the child
instances still purged or deleted to be gone, their status deleted; a child
whose purge or delete failed holds nothing. An unprovision still waits for
the unprovision only.
Comment on lines +77 to +78
case v.GlobalExpect == nil, v.State != nil, v.LocalExpect != nil:
return false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Daemon shutdown can bypass a held object orchestration end

When an object orchestration has completed locally but a peer still carries its ID, the daemon shutdown path submits both State and LocalExpect, causing holdRequest to return false instead of queuing the request. The monitor then publishes the earlier orchestration's end before the peer clears it and accepts shutdown, so the earlier operation can be reported complete while it is still active on a peer.

Show fix

Apply the held-end barrier to shutdown requests too: queue a candidate request with State or LocalExpect while orchestrationEnding is present, then replay it only after the previous end is published. Keep only abort/cleanup updates that must cancel the held end outside this queue, and add a shutdown-after-local-completion/peer-still-active test.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AikidoSec ignore A daemon shutdown is not an orchestration request: it must not wait up to 10s per object behind a peer clearing the one before. The local part of that orchestration is done, and the monitor stopping announces its held end anyway (imon main.go, "its peers clearing it are not waited for any more"), so the shutdown announces it a moment earlier, as the stop would.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Based on your feedback, we ignored this issue because of the following reason:

@AikidoSec ignore A daemon shutdown is not an orchestration request: it must not wait up to 10s per object behind a peer clearing the one before. The local part of that orchestration is done, and the monitor stopping announces its held end anyway (imon main.go, "its peers clearing it are not waited for any more"), so the shutdown announces it a moment earlier, as the stop would.

Comment thread daemon/session/main.go
Comment on lines 337 to 343
o.Error = errS
o.EndedAt = &now
delete(failures, id)
delete(awaitingGone, id)
notifyEnded()
purgeOrchestrations()
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Purge completion events cancel the wait for leftover instances

When the last participating monitor leaves a delete or purge while a nonparticipant still retains instance data, leave correctly puts the orchestration in awaitingGone. The subsequent ObjectOrchestrationEnd emitted as the local object monitor is deleted unconditionally removes that wait and marks the orchestration ended, allowing --wait to succeed while the retained instance remains.

Show fix
Suggested change
o.Error = errS
o.EndedAt = &now
delete(failures, id)
delete(awaitingGone, id)
notifyEnded()
purgeOrchestrations()
}
if _, waiting := awaitingGone[id]; waiting && state == StateSucceeded {
o.State = StateRunning
return
}
o.Error = errS
o.EndedAt = &now
delete(failures, id)
delete(awaitingGone, id)
notifyEnded()

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0655baf: on the node that accepted it, the success a delete or a purge announces no longer ends it while a participant is left or an instance of the object remains. It ends there as on the other nodes, once the participants left and the object is gone, failed past 10s. Covered by TestTheEndOfAPurgeAnnouncedDoesNotSkipTheWaitForTheObjectGone, the end announced both before and after the accepting monitor is seen deleted.

Comment on lines +73 to +74
if mon == nil {
return false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Config fetch can resurrect an object after its monitor is deleted

After a purge/delete removes the serving node's instance monitor, configBeingDeleted returns false even if the config file or config record is still present, so the instance-config endpoint can serve it and the object endpoint can proxy it. A peer fetch in the documented old-omon/recreated-config window can therefore reinstall an instance after the deletion orchestration has left, bypassing the gone check and resurrecting the object.

Show fix

Keep a deletion/purge tombstone or equivalent removal state until the object config and all in-flight fetches are settled, and reject both local and proxied config requests while that tombstone is active. Clear it only after the deletion session finishes; add a test where the local monitor is deleted, the config remains, and a peer fetch is refused rather than installed.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AikidoSec ignore The window does not occur on a delete or a purge: the instance monitor is deleted after its config file, the instance delete removing the file and the config manager cancelling the monitor once it sees it gone, so a node whose monitor is deleted has no config left to serve or to proxy to. The tombstone covering a node absent during a purge is a separate change, after this PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Based on your feedback, we ignored this issue because of the following reason:

@AikidoSec ignore The window does not occur on a delete or a purge: the instance monitor is deleted after its config file, the instance delete removing the file and the config manager cancelling the monitor once it sees it gone, so a node whose monitor is deleted has no config left to serve or to proxy to. The tombstone covering a node absent during a purge is a separate change, after this PR.

…t, the nodes down not waited for

A delete or a purge ended as soon as the nodes seen running it had all
left, their monitors deleted: a node whose instance came back just before,
as one fetching the config again after a config update, was no participant
yet, and purge --wait returned with its instance left, the object still
listed (qatests 308.03.2, debian 12 and rhel 10, reproduced on main).

The orchestration now ends once no instance of the object remains, its
config, status or monitor, on the nodes but the ones in maintenance,
starting, rejoining or shutting down, which keep the data of their
instance and are not waited for. Past 10s it ends failed, naming the nodes
an instance remains on. A participant failing still ends it failed at
once.
…he config of an object being deleted

A peer installing the config of an object has an instance coming, which
the other nodes learn of only as its config reaches them by heartbeat: a
purge accepted meanwhile went on without it, the leader first.

The node serving a config to a peer now announces it, and its instance
monitor counts the peer as starting until its monitor is seen out of init,
10s at most: a delete or a purge waits for it, and so does the leader of a
purge in wait non-leader. A node whose monitor purges or deletes the
object answers 404 to a request for its config, as v2 refused to fetch a
config then: a peer installing it would bring back an instance the
orchestration deletes.
…r a peer

An orchestration done here, its id cleared, its end held back while a peer
still runs it: a request received then was accepted at once, which
published the end before, the --wait of its requester told it ended with
the peer still running it, and the two orchestrations ran at once. The
request is now held while that end is held back, and taken on once it is
published, the peer clearing it or the wait for it timing out.
@cvaroqui
cvaroqui merged commit 76a30f5 into opensvc:main Nov 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant