Skip to content

Tombstones for the purges and deletes some nodes missed, netmon address events of the container namespaces, apparmor_profile keyword - #1232

Draft
cvaroqui wants to merge 20 commits into
opensvc:mainfrom
cvaroqui:dev
Draft

cvaroqui wants to merge 20 commits into
opensvc:mainfrom
cvaroqui:dev

Conversation

@cvaroqui

Copy link
Copy Markdown
Member

Three topics, in this order: the address events of the container network namespaces (netmon), the apparmor_profile container keyword, and the tombstones of the purges and deletes some nodes of an object did not run.

netmon: address events of the other network namespaces

The address of an ip.netns or ip.cni resource lives in the network namespace of its container, which the netlink subscription of netmon did not hear: deleting or adding it there published no NetIPAddrDeleted or NetIPAddrAdded, and the instance status saw the change at its next scheduled refresh only.

  • [netmon] Subscribe again after a receive error. The netlink library ends a subscription on a receive error, as on a buffer overflow, closing its channel; the worker then read the closed channel in a busy loop, with no network event until the daemon restarted. It now logs it and subscribes again after a second.
  • [netmon] Publish the address events of the other network namespaces. A second netlink socket, with NETLINK_LISTEN_ALL_NSID, hears every namespace the host holds an id for, which moving a link into a namespace gives it (bridge/veth, macvlan, ipvlan and dedicated modes, and cni, checked on dev2).
    • Only the addresses of global scope of the other namespaces are published; the events of the host namespace, heard too, are dropped, the existing subscription publishing them unchanged.
    • The events get an nsid field, omitted for the host namespace. The link is named by its label, or nsid<N>-index-<I> for an ipv6 address, whose message names none.
    • The debounce state of a namespace is forgotten as its id is deleted, the next namespace possibly getting the same id.
    • Without the option (kernel older than 4.2, no CAP_NET_BROADCAST) the daemon logs it once and hears the host namespace only, as before.

Tests:

  • Unit: the receive returning on a closed channel and on stop; the parsing of hand-built messages (ipv4 with label, ipv6 without, host namespace without control message, with nsid -1 and with its own id, scope filter, namespace id deletions); the debounce and its purge.
  • dev2n1 (kernel 6.8), a service with a pause container and an ipv4 and an ipv6 ip.netns: an address deleted in the container is published at once, imon matches the resource, which reads down within 1 to 5s, and up again as it is added back; the events go on after a stop/start and after a docker kill, the new namespace getting the same id; the host address events are unchanged, published once.

container: the apparmor_profile keyword

On the Ubuntu 26.04 kernels (7.0, kernel.apparmor_restrict_unprivileged_unconfined=1) the containers are labeled docker-default//&unconfined, and the default profile of the engine denies the processes of a container signalling one another (moby/moby#54099): the haproxy master of the igw_haproxy gateway can not stop its old workers on a reload, and they go on serving the certificate loaded before it (qatests 440.04.6 failing on every u26 run).

  • apparmor_profile names the profile the processes of a docker or podman container run under: a profile loaded on the node, or unconfined. The engines get --security-opt apparmor=<profile>, the apparmor option of run_args dropped, its other security options kept.
  • The status warns when the profile the engine reports differs.
  • A keyword of its own rather than a list of engine options: the profile means the same for every engine, and the status can check it.
  • qatests 440/441 set it on u26 until the engines allow the signals of stacked labels (qatests commit bcbb082).

Tombstones: purges and deletes some nodes did not run

A node that missed a purge or a delete, down, in maintenance, or up with its instance not seen running it, brought the object back to the peers having deleted theirs: 208 to 440 of 850 secs came back on dev2 when created then deleted at once.

Owner decisions: block and tell, never purge on a return, as the node away may hold the only copy of the data; a delete is treated as a purge.

The record

  • [tombstone] A record per orchestration names the holders, the nodes still holding the instance, each pending until its instance is gone or kept, live (up) or away. Copies merge per holder, commutatively; kept under <var>/node/tombstone/<orchestration id>.json.
  • [daemondata] The records of a node are published with its data.
  • [tombmon] Loads them at start, before the rejoin, merges the peer copies, adopts the missed ones in a sweep, demotes the live holders seen away, evicts the holders no longer cluster nodes, and removes a record once all holders are cleared, every node is up and none publishes it pending. Records removed are remembered for an hour, and a stale copy naming the node pending is answered with its holder cleared. Writes are batched (one directory sync per 500ms), publications are made out of its lock, in order.
  • [imon] Before deleting its instance, the imon records the scope nodes never seen carrying the orchestration, or seen carrying it and still holding the instance (a node killed mid-delete).

What the holders do

  • [discover] No config of the path is fetched from a pending holder, nor by one, and a pending holder does not serve it. A live holder of the node is cleared once its instance is gone.
  • [imon] A live holder completes the delete or the purge on its own: it adopts the orchestration of the record. If that fails, the peers done, it turns away.
  • [imon] An away holder is flagged stopped once, never stopped nor frozen; its monitor carries the issue in a new issues field, shown under the instance in the status board, and a start, restart, placement or provision is refused with it. om <path> purge there clears it, an edit or a touch of the config keeps the object, which propagates again.
  • [cluster] om|ox cluster tombstone ls, GET /api/cluster/tombstone: a row per holder.

Breaking changes (CHANGELOG)

  • purge, delete, unprovision of an object whose instance on a node away is up or provisioned are refused, naming the node, without --force (new on delete and on ox; now visible on om purge and om unprovision).
  • create over a tombstone with pending holders is refused, naming them; --force drops the tombstone once every check of the create passed. In a rolling upgrade, a node on a former version never clears itself from a record.

Lag fixes found on the way, standalone commits

  • [discover] The fetched config is synced in the fetch, not in the loop starting the config managers.
  • [pubsub] Subscription.AddFilters adds a list of filters in one bus operation; used by the per-object managers at start, the imon relation, affinity and volume filters, the api waits of a create --wait, and the event stream.
  • [session] The deletes waiting for their object to be gone are checked for the object of the event only, reading the node names with GetNodesByPath, as do 12 api handlers and the imon hard affinity check, rather than copy every node's data.
  • [node], [imon] One list of the node states an orchestration does not wait for; an unprovision or a purge is not held for a node away, nor its convergence for a node in maintenance or shutting down.

Tests

  • Unit: merge, store, data; tombmon (stale copies, reconcile, flush retry, publish loop, demotion, eviction, removal, away holder detection); discover guards and live holder clearing; imon record, adoption, refusal, issue and failure demotion; api listing and create refusal; each commit builds and passes the tests of its packages on its own.
  • dev2 (3 nodes):
    • 850 secs created with --wait (34s) then deleted at once: gone on the three nodes in 64s, no object back, no queue warning, no restart; before the fixes of this PR, 180s, and daemon panics on full queues earlier on.
    • kill -9 of a node mid-delete: no config fetched back; the records heard after its rejoin completed, the ones loaded at restart held until a delete there.
    • an away holder after purge --force: issue shown, stopped flag, start refused; a touch keeps the object everywhere, a purge there removes the record; create refused over the record, create --force drops it.
    • purge, delete, unprovision and ox purge refused naming the node away; with --force the unprovision ends in 2.4s.
    • a child sees its parent go up then down (batched relation filters).

To check in QA: RHEL 7 (3.10 with the option backported) and the macvlan, ipvlan and dedicated modes there.

Comment thread core/tombstone/store.go
// SyncDir syncs the directory of the records to stable storage: the
// renames and removals made in it.
func SyncDir(dir string) error {
d, err := os.Open(dir)

@aikido-pr-checks aikido-pr-checks Bot Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential file inclusion attack via reading file - high severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.

Show fix
Suggested change
d, err := os.Open(dir)
for _, seg := range strings.Split(filepath.ToSlash(dir), "/") {
if seg == ".." {
return fmt.Errorf("invalid file path")
}
}
d, err := os.Open(dir)

Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not reachable by user input: dir is always tombstone.Dir(), <var>/node/tombstone under the node var directory of the node config, and no caller passes another. The files read are the entries os.ReadDir lists in that directory, each name then checked against the id of the record it holds, and the ids of the records written or removed go through recordFile, which refuses an empty id, ., .. and any / or \. Nothing an API client sends names a path here.

Comment thread core/tombstone/store.go
if e.IsDir() || strings.HasPrefix(name, ".") || !strings.HasSuffix(name, ".json") {
continue
}
b, err := os.ReadFile(filepath.Join(dir, name))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential file inclusion attack via reading file - medium severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.

Show fix

Remediation: Ignore this issue only after you've verified or sanitized the input going into this function.

Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not reachable by user input: dir is always tombstone.Dir(), <var>/node/tombstone under the node var directory of the node config, and no caller passes another. The files read are the entries os.ReadDir lists in that directory, each name then checked against the id of the record it holds, and the ids of the records written or removed go through recordFile, which refuses an empty id, ., .. and any / or \. Nothing an API client sends names a path here.

Comment thread daemon/tombmon/main.go Outdated
if len(pending) == 0 {
return nil, nil
}
return st.clear(path, pending, tombstone.ClearedByDropped), nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Forced recreation does not durably remove the old tombstone

A forced create clears the tombstone in memory immediately before committing the new config, but tombmon.Drop only queues the changed record and does not flush it to stable storage. If the daemon crashes after the new config is committed and before that flush, restart reloads the old pending tombstone and continues suppressing or adopting the newly recreated object.

Show fix

Make forced recreation wait for the tombstone update/removal to be durably flushed before acknowledging the new object, and use a transactional or recoverable ordering so a crash cannot leave a committed recreation paired with the old pending tombstone.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit fbdcbcb.

This issue remains because writeObjectConfigFile commits the new config at daemon/daemonapi/lib_object_config_file.go:129 before invoking check.afterCommit at line 134; the callback then calls tombmon.Drop at daemon/daemonapi/post_object_config_file.go:103, whose flush at daemon/tombmon/main.go:800 is therefore still reachable only after a crash window in which the committed config coexists with the old durable tombstone.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The window is real and the order is deliberate. Two files cannot be written atomically, so one side of a crash has to be chosen: here it leaves the new config with the old tombstone still pending, which only withholds. Peers fetch nothing from a pending holder, an away holder never deletes on its own, and a creating node that was itself a holder is demoted to away at restart, so it does not adopt the old delete. Running create --force again drops the tombstone; it is the same state and remedy as the "created, and its tombstone not dropped" answer when the flush fails. The reverse order leaves the old object unprotected on any failed commit (a write conflict is enough): its config propagates again from the holder, and a cleared holder cannot be restored. Closing the window needs an intent journal, which we judge not worth it for a millisecond crash window with a safe outcome and a documented retry (post_object_config_file.go, the comment above the tombstone check).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit c0bbe34.

This issue remains because PostObjectConfigFile commits the config before invoking afterCommit (daemon/daemonapi/lib_object_config_file.go:129-135), while tombmon.Drop only starts its durable tombstone flush afterward (daemon/tombmon/main.go:803-808); a crash in that interval still leaves the committed recreation paired with the old pending tombstone. The added flush is a partial fix, but it does not make the ordering crash-safe.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit d2e46c0.

This issue remains because lib_object_config_file.go:129-135 commits the recreated config before invoking tombmon.Drop, so a daemon crash in that gap restarts with the old pending tombstone; Drop's flush at daemon/tombmon/main.go:808 only makes removal durable after it is reached.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 787c9d6.

This issue remains because daemon/daemonapi/lib_object_config_file.go:129 commits the new config before invoking afterCommit at line 134; a crash in that gap leaves the old tombstone on disk, and tombmon.Drop's flush at daemon/tombmon/main.go:808 cannot run. The added synchronous flush only narrows the window after Drop starts.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 05be262.

This issue remains because writeObjectConfigFile commits the new config at daemon/daemonapi/lib_object_config_file.go:129 before invoking the forced-create callback at :134, so a crash before tombmon.Drop at daemon/daemonapi/post_object_config_file.go:106 starts still leaves the old pending tombstone despite Drop now flushing after it runs.

Comment thread daemon/imon/main.go Outdated
Comment on lines 573 to 577
t.tombstones = tombstone.Data.MergedOf(t.path.String())
t.applyTombstoneIssue()
for n, v := range instance.StatusData.GetByPath(t.path) {
t.instStatus[n] = *v
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Startup flags an up tombstone holder as stopped

When an imon starts on a node whose instance is actually up and has an away-holder tombstone, startup applies the tombstone issue before loading the local instance status cache. The availability check therefore sees the zero-value status and writes the stopped flag, suppressing later automatic starts for an instance that was never down until an operator clears it.

Show fix
Suggested change
t.tombstones = tombstone.Data.MergedOf(t.path.String())
t.applyTombstoneIssue()
for n, v := range instance.StatusData.GetByPath(t.path) {
t.instStatus[n] = *v
}
for n, v := range instance.StatusData.GetByPath(t.path) {
t.instStatus[n] = *v.Value
}
t.tombstones = tombstone.Data.MergedOf(t.path.String())
t.applyTombstoneIssue()

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c0bbe34: the statuses are read before the tombstones at start, and the stopped flag is decided only once the local instance status is known, re-checked when it arrives; an up instance shows the issue and is never flagged. Checked on dev2 with an away holder up on its return.

Comment on lines +95 to +105
case errors.Is(err, unix.ENOBUFS):
t.log.Warnf("address events of the other network namespaces lost: receive buffer overflow")
continue
default:
t.log.Warnf("address events of the other network namespaces: receive: %s", err)
select {
case <-t.ctx.Done():
return
case <-time.After(resubscribeDelay):
}
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - NSID event loss leaves stale deduplication state across namespace reuse

When the all-namespace netlink socket overflows or receives an error, nsidWorker keeps using the same socket and state map instead of recreating the subscription and resetting state. If the lost messages include RTM_DELNSID, a later namespace can reuse that numeric ID and have its first matching address-add event suppressed as a duplicate, so imon misses the address and may keep the instance status stale until a scheduled refresh.

Show fix
Suggested change
case errors.Is(err, unix.ENOBUFS):
t.log.Warnf("address events of the other network namespaces lost: receive buffer overflow")
continue
default:
t.log.Warnf("address events of the other network namespaces: receive: %s", err)
select {
case <-t.ctx.Done():
return
case <-time.After(resubscribeDelay):
}
continue
case errors.Is(err, unix.ENOBUFS):
t.log.Warnf("address events of the other network namespaces lost: receive buffer overflow")
state = make(map[int]map[string]bool)
continue
default:
t.log.Warnf("address events of the other network namespaces: receive: %s", err)
state = make(map[int]map[string]bool)
select {
case <-t.ctx.Done():
return
case <-time.After(resubscribeDelay):
}
continue

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c0bbe34: the per-namespace state is forgotten on a buffer overflow and on a receive error, the deletion of a namespace id possibly among the events lost, so the next namespace getting the id publishes its first address.

Comment thread daemon/tombmon/main.go
Comment on lines +354 to +356
if err := tombstone.RemoveFile(dir, id); err != nil {
log.Errorf("remove the tombstone %s: %s", id, err)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Failed tombstone removal is acknowledged as successful

When a forced recreation clears a tombstone, flush logs a RemoveFile or removal-only directory-sync failure but does not return or retry it, so Drop reports success after the new config is committed while the old pending tombstone can remain on disk. A restart then reloads and republishes that stale tombstone, blocking the recreated object from peers or causing it to be treated as still tombstoned.

Show fix

Record removal and directory-sync failures in the returned error map, preserve/requeue the corresponding removed entries for retry, and make Drop fail until the removal state is durably persisted.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c0bbe34: a removal or a directory sync failing is now an error of the flush, Drop's included, and the removal is queued again for the next flush.

func (t *Manager) hasNonLeaderProvisioned() bool {
for node, otherInstStatus := range t.instStatus {
if t.isUnprovisionLeaderNode(node) {
if t.isUnprovisionLeaderNode(node) || t.isNodeAway(node) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Unprovision releases shared resources after a holder goes away

After an unprovision request is accepted, a provisioned non-leader can enter maintenance, upgrade, or shutdown before the leader acts. The leader now skips that away peer when deciding whether to release shared resources, so it can unprovision while the peer still uses them; the request-time refusal is not rechecked and unprovision creates no tombstone to protect the missed holder.

Suggested change
if t.isUnprovisionLeaderNode(node) || t.isNodeAway(node) {
if t.isUnprovisionLeaderNode(node) {

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, fixed in c0bbe34. The refusal for an away node holding a provisioned instance is checked when the request is accepted, and the leader then skipped any away peer, so a peer entering maintenance or restarting its daemon after acceptance was not waited for. The leader now skips an away peer only when the request was forced; otherwise it holds for a provisioned one, as before the change, until the peer is back and runs its part or the maintenance grace period drops its data. The suggestion as written would also hold a forced unprovision, which the refusal sends there on purpose, so the skip is kept for that case.

Comment on lines +120 to +125
if bt.AppArmorProfile != "" {
// The --security-opt of run_args setting another profile is
// dropped, the others kept.
runArgs.DropOptionAndMatchingValue("--security-opt", "^apparmor[=:]")
a.Append("--security-opt", "apparmor="+bt.AppArmorProfile)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Equals-form run_args overrides apparmor_profile

When a root-managed container sets apparmor_profile and includes --security-opt=apparmor=other in run_args, the new filter misses the equals form and appends it after the explicit profile option. Docker and Podman apply the later single-valued AppArmor option, so the container runs under other rather than the profile named by the keyword and the keyword no longer reliably controls or describes the container.

Show fix
Suggested change
if bt.AppArmorProfile != "" {
// The --security-opt of run_args setting another profile is
// dropped, the others kept.
runArgs.DropOptionAndMatchingValue("--security-opt", "^apparmor[=:]")
a.Append("--security-opt", "apparmor="+bt.AppArmorProfile)
}
if bt.AppArmorProfile != "" {
// The --security-opt of run_args setting another profile is
// dropped, the others kept.
filtered := make([]string, 0, len(runArgs.Get()))
for _, arg := range runArgs.Get() {
if strings.HasPrefix(arg, "--security-opt=apparmor=") || strings.HasPrefix(arg, "--security-opt=apparmor:") {
continue
}
filtered = append(filtered, arg)
}
runArgs.Set(filtered)
runArgs.DropOptionAndMatchingValue("--security-opt", "^apparmor[=:]")
a.Append("--security-opt", "apparmor="+bt.AppArmorProfile)
}

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c0bbe34: the --security-opt=apparmor=... and apparmor:... equals forms of run_args are dropped too when apparmor_profile is set, the other security options kept.

Comment thread daemon/netmon/linux.go
if !ended {
t.log.Infof("context done, stopping netlink monitor")
return
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Resubscribe retains stale netlink deduplication state

When the host-only netlink subscription closes after a receive error, the worker resubscribes but keeps the address and link state accumulated before the outage. If the missed events remove and recreate an address or link with the same name and state, the first post-recovery event is classified as a duplicate and no refresh is published, leaving the matching instance resource stale until an unrelated scheduled refresh.

Suggested change
}
t.lastPublished = make(map[string]linkPublishState)

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d2e46c0: what was published is forgotten at each subscription, so an address or link removed and added back while the events were missed is published again.

Comment thread daemon/netmon/nsid_linux.go Outdated
Comment on lines +318 to +327
key := fmt.Sprintf("%d:%s", ev.linkIndex, ev.address)
last, ok := state[ev.nsid]
if !ok {
last = make(map[string]bool)
state[ev.nsid] = last
}
if added, ok := last[key]; ok && added == ev.added {
continue
}
last[key] = ev.added

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - All-namespace netlink deduplication grows with every address churn event

The all-namespace listener stores a permanent key for every unique link-index/address operation in each live namespace and never removes keys when an address is deleted. A process able to churn unique addresses in a monitored namespace can therefore grow this daemon-owned map without bound, eventually consuming memory and destabilizing netmon or the daemon.

Show fix

Bound the per-namespace deduplication history or delete an address key after its corresponding delete has been processed while retaining only the minimal state needed to suppress duplicate kernel notifications.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d2e46c0: only the addresses present are kept, a delete forgetting its address, so churned addresses no longer grow the state of a namespace; a delete repeated is published again, which only refreshes.

Comment thread daemon/imon/tombstone.go Outdated
Comment on lines +296 to +301
t.tombstoneStopped = id
if t.instStatus[t.localhost].Avail.Is(status.Up, status.Warn) {
return
}
t.log.Infof("flag the instance stopped: %s", issue)
_ = t.setStopped()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Tombstone stopped decision is lost when the flag write fails

When an away-holder tombstone is first applied to a locally non-up instance, the monitor records the tombstone ID before attempting to persist the stopped flag and discards the write error. If that write fails, later status or tombstone events skip the ID permanently, so clearing the tombstone can allow the instance to start without the operator's required stopped decision.

Show fix
Suggested change
t.tombstoneStopped = id
if t.instStatus[t.localhost].Avail.Is(status.Up, status.Warn) {
return
}
t.log.Infof("flag the instance stopped: %s", issue)
_ = t.setStopped()
if t.instStatus[t.localhost].Avail.Is(status.Up, status.Warn) {
return
}
t.log.Infof("flag the instance stopped: %s", issue)
if err := t.setStopped(); err != nil {
return
}
t.tombstoneStopped = id

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d2e46c0: the tombstone is taken as decided only once the stopped flag is written, or the instance seen up; a failed write is decided again at the next status or tombstone event.

Comment thread daemon/imon/tombstone.go Outdated
Comment on lines +163 to +164
if t.seenCarrying[n] == id && !holding[n] {
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Stale carrying state omits a peer from the delete tombstone

If a peer has announced that it is carrying the current delete orchestration and then disappears before the local tombstone is recorded, peer-loss handling removes its cached instance data but leaves its seen-carrying ID. The tombstone builder then interprets the missing cache entry as proof that the peer deleted its instance and omits that node, so the peer can return with the old instance and reintroduce the object without tombstone protection.

Show fix

Clear seenCarrying for a node when its instance monitor is deleted/forgotten, and treat a carrying peer whose data disappeared because the peer was lost as an away holder unless deletion is independently confirmed.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d2e46c0: a peer seen running the delete with no instance left counts as deleted only while its node is up. One away or lost, its data dropped with it, is an away holder, cleared on its return if its instance is gone there.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit d2e46c0.

This issue remains because daemon/imon/main_cmd.go:934-944 removes a peer's instMonitor without clearing seenCarrying (and onForgetPeer at :896-900 also leaves it), while daemon/imon/tombstone.go:184 still omits that peer whenever its nodeMonitor is momentarily still up during dropPeer event processing; the isNodeUp guard only handles an already-away node, not this race.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, fixed in 787c9d6. dropPeer now drops the node monitor of the peer before its instance data, and the holder check reads the node as up only when both the monitor of the imon and the node data say so: a reader finding the instance data of a dropped peer gone finds its node gone too, so a delete recording its tombstone during the drop names the peer an away holder. Checked on dev2 with a node frozen until dropped, then purged: recorded away, the issue shown on its return, and cleared by a purge there.

…ve error, rather than read the closed channel in a loop

The netlink library ends a subscription on a receive error, as on a receive
buffer overflow, closing its channel. The worker read the closed channel
for good: a busy loop, and no link or address event until the daemon
restarted.

The worker now sees the channel closed, logs it, and subscribes to both
again after a second, the events in between missed. The subscription errors
are logged. What was published before is forgotten at each subscription:
the events missed may have removed and added back the same address or link,
whose next event would else read as a repeat.
@cvaroqui
cvaroqui force-pushed the dev branch 2 times, most recently from d2e46c0 to 787c9d6 Compare November 18, 2026 12:21
Comment thread daemon/netmon/nsid_linux.go Outdated
Comment on lines +100 to +107
t.log.Warnf("address events of the other network namespaces: receive: %s", err)
forgetNsState(state)
select {
case <-t.ctx.Done():
return
case <-time.After(resubscribeDelay):
}
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Fatal NSID receive errors permanently disable container address monitoring

When the all-namespace netlink socket returns any error other than EAGAIN, EINTR, or ENOBUFS, the worker waits and then loops on the same descriptor instead of closing and recreating the subscription. If that descriptor is no longer usable, all subsequent address changes in container namespaces are silently lost until the daemon restarts, leaving matching instance resources stale.

Show fix

Treat a non-retryable receive error as subscription loss: close the descriptor, wait for the resubscribe delay, open a fresh all-NSID socket, and reset the per-NSID state before receiving again. Keep retrying transient open failures while the daemon is running.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 05be262: a receive error but a timeout, an interruption or an overflow closes the all-namespace socket and opens a new one after the resubscribe delay, again until it opens or the daemon stops, the per-namespace state forgotten.

Comment thread daemon/imon/tombstone.go Outdated
Comment on lines +360 to +367
if !t.state.State.IsOneOf(instance.MonitorStatesFailure...) {
return
}
switch t.state.GlobalExpect {
case instance.MonitorGlobalExpectPurged, instance.MonitorGlobalExpectDeleted:
default:
return
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Recoverable purge failures demote a live tombstone holder

demoteTombstoneIfFailed treats every generic instance failure state as a failed delete/purge, including freeze, provision, start, stop, and unfreeze failures. During a purge, several of those states are explicitly recovered by returning to idle and continuing, but this hook first converts the local live holder to away, causing a false tombstone issue and potentially setting the stopped flag or blocking later starts/provisions while the purge is still recoverable.

Show fix
Suggested change
if !t.state.State.IsOneOf(instance.MonitorStatesFailure...) {
return
}
switch t.state.GlobalExpect {
case instance.MonitorGlobalExpectPurged, instance.MonitorGlobalExpectDeleted:
default:
return
}
switch t.state.GlobalExpect {
case instance.MonitorGlobalExpectPurged:
if !t.state.State.IsOneOf(instance.MonitorStateStopFailure, instance.MonitorStatePurgeFailed) {
return
}
case instance.MonitorGlobalExpectDeleted:
if t.state.State != instance.MonitorStateDeleteFailure {
return
}
default:
return
}

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 05be262: the live holder turns away only on the failures ending the orchestration, delete failed for a delete, stop or purge failed for a purge; a delete going on after a stop, start, freeze or provision failure, as from idle, leaves it live.

Comment thread daemon/tombmon/main.go
Comment on lines +451 to +480
func (s *state) onPeerRecord(rec tombstone.Record) {
if _, ok := s.records[rec.OrchestrationID]; !ok {
if !rec.HasPending() {
return
}
if g, ok := s.gone[rec.OrchestrationID]; ok {
// Removed here: a stale copy, in flight or published
// again by a peer that missed the removal. Answered when
// it names the node pending: its holder was cleared,
// the node alone can say it.
if g.by == "" || !rec.IsPending(s.localhost) {
return
}
rec = rec.DeepCopy()
h := rec.Holders[s.localhost]
h.State = tombstone.HolderCleared
h.ClearedBy = g.by
rec.Holders[s.localhost] = h
s.log.Infof("tombstone %s of %s: a peer publishes it again, holder %s cleared, %s", rec.OrchestrationID, rec.Path, s.localhost, g.by)
s.upsert(rec)
return
}
rec = rec.DeepCopy()
if !s.localRejoined() {
// Adopted before the rejoin, as by a node that was down:
// its instance is not trusted to complete the delete.
demote(&rec, s.localhost)
}
}
s.upsert(rec)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Restart forgets removed tombstones and allows stale peer copies to resurrect

A node remembers removed orchestration IDs only in the in-memory gone map, which is empty after tombmon restarts and expires after one hour. When a down or evicted peer later republishes its durable pending tombstone, the restarted node adopts it as a new record; if the record names the node as a live holder, imon can adopt the old delete/purge orchestration and delete or block a current object with the same path.

Show fix

Persist removal generations/cleared markers alongside tombstone records, or otherwise retain durable suppression for stale orchestration IDs across restart and beyond the in-memory grace period. A returned peer's old pending copy must be merged with the known cleared state before it can be adopted.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 05be262, durably rather than by remembering removals: the instance config now carries the object id (DEFAULT.id) and the tombstone the id of the object it deletes. A live holder whose instance is of another object, created again under the path, never adopts the old delete, its holder cleared instead, as an away holder whose config is of another object is; so a stale copy coming back after a restart, or days later, cannot delete or block a recreated object. Unknown ids, as in a record of a former version, tell nothing, as before.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 05be262.

This issue remains because Start initializes the removal suppression only in the in-memory gone map at daemon/tombmon/main.go:154, while onPeerRecord at daemon/tombmon/main.go:452-480 adopts any pending peer record after restart when gone is empty and the local node has rejoined, so a stale copy naming this node as a live holder can still resurrect the orchestration.

// the configuration is written: a create refused or failed leaves it,
// and a crash between the two leaves it pending, the safe side, a
// create --force again dropping it.
pending := tombstone.Data.PendingHolders(p.String(), "")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Concurrent tombstone creation can leave a committed create blocked

The POST handler snapshots pending holders before validation and installs the post-commit Drop callback only when that snapshot was nonempty. If a delete or purge records a tombstone after the snapshot but before the configuration commit, the new configuration commits successfully without dropping that tombstone, so propagation and discovery continue withholding the recreated object until another forced create or manual cleanup.

Show fix

Serialize the tombstone check/commit/drop sequence, or recheck for newly recorded tombstones immediately after the config commit and drop only the tombstone state that was part of this create transaction. The operation should not acknowledge a create while a concurrent pending tombstone remains unhandled.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 05be262 for a forced create: the drop now runs after every forced commit, and Drop reads the pending holders then, so a tombstone recorded by a delete after the check is dropped too. An unforced create racing a delete of the same path stays as is: the outcome only withholds the new object, the safe side, and create --force resolves it.

objectaction.WithSort(t.Sort),
objectaction.WithObjectSelector(mergedSelector),
objectaction.WithAsyncTarget("deleted"),
objectaction.WithAsyncTargetOptions(instance.MonitorGlobalExpectOptionsDeleted{Force: t.Force}),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Node-selected delete ignores the new force flag

When om delete --node or ox delete runs through the node-selected remote path, the new --force value is stored only in the async target options, while the custom remote callback sends PostInstanceActionDelete with a params struct containing only the session id. The target node therefore applies the normal away-holder refusal instead of honoring --force, so a forced delete fails or waits when the peer instance is away.

Show fix

Forward the force intent through the remote action API, or route node-selected deletes through the object-action endpoint that accepts MonitorGlobalExpectOptionsDeleted{Force:true}. Add an integration test for --node <away-node> --force and verify the target receives the force option.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not reachable: with --node, delete is an instance action, PostInstanceActionDelete running the delete of the instance on that node, and no orchestration. The refusal of an away node holding its instance up or provisioned, and the force overriding it, are checked by the instance monitor when it accepts the deleted global expect of an orchestration (daemon/imon/main_cmd.go); the instance action never goes through it, so there is no refusal for --force to lift on that path.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 05be262.

This issue remains because core/oxcmd/object_delete.go:46-51 still builds api.PostInstanceActionDeleteParams with only SessionID and sends it through the node-selected remote callback, while the Force value is only applied to the local async target options at line 35; the delete API has no force field for this path.

Comment thread daemon/tombmon/main.go Outdated
Comment on lines +708 to +735
func DemoteLocal(path string) error {
mu.Lock()
defer mu.Unlock()
if st == nil {
return ErrNotStarted
}
st.adoptPeerCopies(path)
for _, rec := range st.recordsOf(path) {
rec = rec.DeepCopy()
if demote(&rec, st.localhost) {
st.log.Infof("tombstone %s of %s: holder %s away, its instance failed to complete the %s", rec.OrchestrationID, rec.Path, st.localhost, rec.Kind)
st.upsert(rec)
}
}
return nil
}

// adoptPeerCopies adopts the copies the peers publish of the records of
// path the node does not keep and that have a pending holder, even one it
// removed: an explicit clear of a stale copy is what ends it. The caller
// holds mu.
func (s *state) adoptPeerCopies(path string) {
for id, rec := range tombstone.Data.MergedOf(path) {
if _, ok := s.records[id]; ok || !rec.HasPending() {
continue
}
s.upsert(rec)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Purge failure re-adopts stale tombstones for unrelated orchestration IDs

When a delete or purge failure demotes the local live holder, DemoteLocal first adopts every pending peer record for the path, including records previously removed and records for older orchestration IDs, then demotes all local live holders. A stale peer copy can therefore be resurrected and surfaced as an away-holder issue for the current object even though the failure only concerns one orchestration, causing a false stopped/blocking decision and potentially interfering with the current recreation.

Show fix

Pass the failed orchestration ID to DemoteLocal and update only that record. Do not adopt peer copies marked recently removed by WasRemoved, and preserve the normal stale-copy suppression when importing records for failure handling.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 05be262: DemoteLocal takes the orchestration id of the failed delete or purge and turns away the holder of the node in that record only, other records of the path left alone, and a record the node removed is not taken in again for it.

Comment thread daemon/imon/main_cmd.go
Comment on lines 289 to 295
func (t *Manager) onMyInstanceStatusUpdated(srcNode string, srcCmd *msgbus.InstanceStatusUpdated) {
// The node publishes its status again: its last one, kept while it
// was away, is stale.
delete(t.awayStatus, srcCmd.Node)
if srcCmd.Node == t.localhost {
// Armed from the status kept, which may be newer than the one
// of the event: the first refresh stores its status before its

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Stale peer status clears the away-holder protection

When a peer's instance status is deleted while the node is away, the new handler saves the last protected status in awayStatus, but any later InstanceStatusUpdated immediately deletes that saved value before validating freshness or confirming the node has left its away/rejoin state. A queued stale down/unprovisioned status can therefore replace the protected up/provisioned state, allowing a non-forced delete, purge, or unprovision to be accepted while the peer may still hold or use the instance.

Show fix
Suggested change
func (t *Manager) onMyInstanceStatusUpdated(srcNode string, srcCmd *msgbus.InstanceStatusUpdated) {
// The node publishes its status again: its last one, kept while it
// was away, is stale.
delete(t.awayStatus, srcCmd.Node)
if srcCmd.Node == t.localhost {
// Armed from the status kept, which may be newer than the one
// of the event: the first refresh stores its status before its
func (t *Manager) onMyInstanceStatusUpdated(srcNode string, srcCmd *msgbus.InstanceStatusUpdated) {
if _, ok := t.awayStatus[srcCmd.Node]; ok {
// Keep the protected status until NodeMonitorUpdated confirms the node is back.
return
}
if srcCmd.Node == t.localhost {
// Armed from the status kept, which may be newer than the one
// of the event: the first refresh stores its status before its

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not reachable as described: a peer away publishes no status, its status being deleted is what saved awayStatus, and the next InstanceStatusUpdated of that peer can only come from the peer itself once its daemon runs again, rejoining: a fresh status of its instance, not a queued stale one, the events of a peer being delivered in order. That status is the truth the refusal is to judge by, so it replaces the saved one; the saved status is also forgotten when the node is seen back (onNodeMonitorUpdated).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 05be262.

This issue remains because daemon/imon/main_cmd.go:289-292 still unconditionally deletes awayStatus for every InstanceStatusUpdated before the UpdatedAt freshness check and without checking the peer's away/rejoin state; the added NodeMonitorUpdated cleanup does not prevent a queued stale update from clearing protection.

…as the ones of the ip.netns and ip.cni containers

The netlink subscription heard the namespace of the daemon only: an address
deleted or added in the namespace of a container, as the one of an ip.netns
or ip.cni resource, published no event, and the instance status saw it at
its next scheduled refresh only.

A second socket listens to every namespace the namespace of the daemon
holds an id for, which moving a link into a namespace gives it, as every
mode of the ip drivers does (NETLINK_LISTEN_ALL_NSID). The kernel says the
namespace of a message by that id. The listener publishes the addresses of
global scope of the other namespaces, with the id in a new nsid field of
NetIPAddrAdded and NetIPAddrDeleted, omitted for the namespace of the
daemon, and the link named by its label, or nsid<N>-index-<I> for an ipv6
address, whose message names no link. imon matches them to the resources by
address, as before.

The events of the namespace of the daemon, which the socket hears too, are
dropped: the subscription of the worker publishes them. It stays as it was,
a burst of container events overflowing the buffer of the second socket not
losing an event of the node. The last event of each address of a namespace
is forgotten as its id is deleted: a namespace going deletes its addresses
with no message, and the next one may get the same id. All of them are
forgotten as events are lost, a buffer overflow or a receive error, the
deletion of an id possibly among them. Only the addresses present are kept:
a delete forgets its address, for addresses churned in a namespace not to
grow the state.

Without the option, as on a kernel older than 4.2 or a daemon without
CAP_NET_BROADCAST, the other namespaces are not heard, as before.

A receive error but a timeout, an interruption or an overflow closes the
socket and opens a new one after a second, again until it opens, as the
host subscription is subscribed again.
…docker or podman container runs under

On the ubuntu 7.0 kernels, with kernel.apparmor_restrict_unprivileged_unconfined=1,
the containers are labeled docker-default//&unconfined, and the default
profile of the engine, allowing the signals to its own name only, denies the
processes of a container signalling one another (moby/moby#54099): the
haproxy master of the igw_haproxy gateway can not stop its old workers on a
reload, and they go on serving the certificate loaded before it.

apparmor_profile names the profile the processes of the container run
under, instead of the default of the engine: a profile loaded on the node,
or unconfined. Docker and podman are given --security-opt apparmor=<profile>,
the apparmor option of run_args dropped, in the --security-opt=<value> form
too, its other security options kept.
The status warns when the profile the engine reports differs. A keyword of
its own rather than a list of engine options, which run_args already
passes: the profile means the same for every engine, a containerd one too,
and the status can check it. Setting it requires the root grant, as
run_args and privileged do: it decides what the container reaches of the
node.
The session and nmon each listed the node monitor states an orchestration
does not wait for: a node down, starting, rejoining or shutting down,
which runs no orchestration and catches up as it rejoins. The list is now
node.MonitorStatesAway, with MonitorState.IsAway, for the tombstones and
the instance monitor to use the same. MonitorState.IsAwayFromObjects adds
the upgrade of the packages of the node, its daemon restarting: it runs no
object orchestration, while the upgrade, a node orchestration, does not
count it away. The session and the object code judge with it.
…de away

The leader of an unprovision or a purge waited for the non-leader instances
to be unprovisioned, reading the status a node in maintenance retains: the
orchestration hung until the maintenance grace period dropped that data.
The instance of a node away is not waited for any more.
…tarts the config managers

The discovery loop installed each fetched config with a sync of the file,
waiting on the disk, and the same loop starts the config manager of every
object. With 850 objects created then deleted at once, the disk busy with
the deletes, the managers started up to 84s after their config landed: the
delete ran meanwhile on the nodes whose instance had no manager yet, was not
seen there, and the accepting node fetched the configs back from them, 208
objects coming back.

The fetch syncs the file it writes beside the config file, and the loop only
renames it in place (xconfig.InstallSyncedFile), under the lock. A crash
before the rename is on disk leaves the former state, which the peer serves
again, never an empty file.
…s, for the managers of many objects not to wait on the busy bus once per filter

The bus serves the operations of every component of the daemon one at a
time, and starting the managers of an object made one operation per filter:
about 43, for its config manager, object monitor and instance monitor. With
850 objects created and deleted at once, those starts queued behind the
busy bus, which held up the discovery loop starting the config managers.

Subscription.AddFilters adds a list of filters, built with Filters.Add, in
one operation, each taking effect at once as AddFilter does. The config
manager, the object monitor and the instance monitor start with it. On dev2
the 850-object create then delete left 1 object back instead of 29 with the
sync fix alone, the managers starting 22s after their config landed on
average instead of 30s.

The instance monitor adds with it the filters of the relations, the
affinities and the volumes of its object, before it reads their statuses,
and the api the filters of its waits for a config to propagate and for an
object to be known, which every create --wait set up, and of an event
stream, one per filter the client asks for.
…e object of the event only, with no copy of its data

While a delete or a purge waited for its object to be gone, the session
checked every orchestration waiting at each event, each check copying the
config, status and monitor of every node of the object, the statuses with
all their resources, to read the node names. With 850 objects deleted at
once, the session fell behind the events, its subscription queue reaching
the limit past which the daemon panics (615 warnings on a node of dev2).

An event about an object checks the deletes of that object only; an event
about a node, and the ticker, check them all. remainingInstances reads the
node names with instance.Data.GetNodesByPath, which copies nothing. On dev2
the same deletes gave 0 to 2 warnings per node.

The api handlers reading the nodes of the data of an object, or their
number, and the hard affinity check of the instance monitor read them with
GetNodesByPath too, rather than copy every node's monitor, config or
status.
…bject did not run

A record names the nodes still holding the instance of an object a delete
or a purge did not reach: live, up but not seen running the orchestration,
as one whose config manager had not started yet, or away, down or in
maintenance. Each holder is pending until its instance is gone or kept.

The copies of a record merge per holder, a cleared holder winning, of two
cleared a fixed rank, away over live: commutative, associative and
idempotent, every node ends with the same record. The records are kept
under <var>/node/tombstone/<orchestration id>.json, written atomically, and
the package holds the records every node publishes, the pending holders of
a path read from all the copies merged.
The records a node keeps are published under its data,
cluster.node.<node>.tombstone.<id>, with the NodeTombstoneUpdated and
NodeTombstoneDeleted events, wired as the pool statuses are: forwarded to
the peers, applied from a peer patch, compared against a peer full data as
they are, a record carrying no update time, and dropped with a dropped peer.

The data of a peer dropped goes with its node monitor first, then its
instance data: a reader finding the instances of the peer gone finds its
node gone too, and does not take them for deleted by a node still up, as a
delete recording its tombstone.
The tombmon manager loads the records of the node at start and publishes
them before the node monitor leaves the rejoin; a live holder of the node
turns away there, its daemon having restarted. It merges the copies its
peers publish, adopting a record it does not keep only with a pending
holder, and takes in by a sweep every 10s the ones it missed as events. It
turns away the live holders whose node goes away, clears the holders no
longer cluster nodes as evicted, and removes a record once all its holders
are cleared, every cluster node is up, and no node publishes it with a
pending holder.

Once the node monitor is out of its start and rejoin, the away holder of
the node is cleared as purged or deleted when the config file and config
data of the instance are gone there, and as kept when its config file
changes while the instance is quiet: its monitor not purging, deleting or
unprovisioning, idle, and no action of it running but a status refresh,
which a touch itself triggers. A change made otherwise takes the baseline
again. The baseline is the modification time at the first look, in memory:
a restart takes it again, and no time of another node is compared. A live
holder is left to the discovery, the only installer of a config fetched, so
a fetch in flight does not race it.

It keeps up with a delete of many objects, 850 at once on dev2, where a
first version panicked on its full queue at each restart: the records are
indexed by path; the manager takes the config events of the node, not every
instance monitor update, and a node monitor update only when the node goes
away or comes back; a merge or a clear is written by a flush every 500ms,
one sync of the directory for the batch; the messages are published in
order by a publisher of their own, out of the lock the callers wait on. The
subscription takes a huge queue.

The ids of the records removed are remembered for an hour: a copy of one
still in flight from a peer is stale, and adopted it was removed again, the
peers doing the same with the copies of the node, round after round. A
stale copy naming the node a pending holder, as one a peer killed before it
wrote its last merge publishes again, is answered with the holder cleared
as it was: the node alone can say it, and nothing else would.

Its package API: Record, for the imon to delete an instance only once its
record is written and published; ClearLocal for a holder clearing itself;
DemoteLocal for the live holder of the node in the record of a delete that
failed there, no other; Drop for a create --force, saying the holders it
cleared once written and published, nothing to do without a pending holder;
List for the listing; WasRemoved for the imon not to adopt a stale copy.
…er tombstone ls and GET /api/cluster/tombstone

GET /api/cluster/tombstone returns the tombstones the nodes publish, the
copies merged per orchestration, with their holders, the state, mode and
clearing of each, and the nodes publishing them. Every node holds them all,
so the node asked answers. Root sees them all, another user the ones of the
namespaces it is guest of.

om and ox cluster tombstone ls list them, a row per holder.
…clear a live holder of the node once its instance is gone

A pending holder of a tombstone holds the instance of an object a delete
or a purge did not reach yet: a config fetched from it brings the object
back, as the node accepting a delete fetched the configs back from the
peers whose instance had no manager yet. The discovery fetches no config
of the path from a pending holder, and a pending holder fetches none:
checked on a remote config update, before the foreign config branch, on a
config for, at the fetch every path goes through, at the install, and in
the choice of a peer to recover a config from. A pending holder does not
serve the config either.

A live holder of the node is cleared once its instance is gone there,
decided by the discovery, the only installer of a config fetched: as a
config manager ends with no config file, and as a tombstone naming the
node lands, its fetch cancelled first. A holder cleared with a config
published is a source again.
…ll holding the instance or not seen running the delete

The imon remembers the orchestration id the monitor of each peer was seen
carrying. Before it deletes its instance, the point of no return of a
delete or a purge, it records the tombstone of the scope nodes, cluster
nodes, never seen carrying the orchestration, or seen carrying it and still
holding the instance, its config, status or monitor: away when their node
monitor is absent or away, live otherwise, as a node whose config manager
had not started as the delete ran. No holder, no record: the last node to
delete, its peers gone, writes none. A tombstone that can not be saved
fails the delete, nothing deleted without it.

A peer seen running it is a holder while it holds the instance: killed
before it deleted, as by a crash with its delete queued, it restarted with
no monitor carrying the orchestration any more, kept the object, and the
peers fetched it back from it, as 18 objects came back on dev2. On dev2,
with the daemon of a node killed with 157 deletes queued, no config was
fetched back; the records it heard after its rejoin it completed, the 5 it
loaded at its restart it held, block and tell, until a delete there.

A peer seen running it with no instance left counts as deleted only while
its node is up, as the monitor last heard it and as the node data says now:
one lost, its data dropped with it, is an away holder, cleared on its
return if its instance is gone there, including while its drop is under
way, the monitor not told yet.

A delete or a purge waits for the live holders of its tombstone too,
checked again on the tombstone events of the object.
…e a live holder

A live holder is a node up whose instance did not run the delete or the
purge, as one whose config manager had not started as it ran: the node
that accepted it gone with its monitor, no peer monitor carries it any
more. The instance monitor adopts the global expect of the tombstone, its
stamp and orchestration id, when no orchestration runs and its node is out
of the rejoin, and runs it as one adopted from a peer, writing no
tombstone of its own for it, nor adopting a stale copy of a record its node
removed. An away holder never adopts it: a node down, or restarted since,
may hold the only copy of the data.

On dev2, 850 secs created then deleted at once end gone on the three
nodes, with no config fetched back, where 208 to 440 came back before.
…e of a node away up or provisioned, without --force

An orchestration does not wait for a node away: a purge or a delete can not
release its instance, maybe the only copy of the data, and an unprovision
may release resources it still uses. The node accepting the request refuses
it, naming the nodes down, in maintenance or unknown whose instance was
last published up or provisioned, provisioned only for an unprovision. A
node stopping deletes the statuses of its instances once it announced its
shutdown: the last one is kept, and forgotten as the node publishes again.

--force goes on without them, a purge or a delete then leaving a tombstone
the node holds. Unforced, the leader of an unprovision or a purge waits for
a node that went away after the request with its instance provisioned, as
one restarting its daemon or entering a maintenance, until it is back or
the maintenance grace period drops its data: it releases no shared
resources under it. The force is an option of the global expect the peers
adopt, a force query parameter of the object purge, delete and unprovision
actions, and --force of om and ox purge, delete and unprovision: the one of
om purge and unprovision, hidden before and allowing the dangerous
operations of a --local action, is shown and keeps that meaning too.
… its node was away

An away holder of a tombstone holds an instance the other nodes purged or
deleted while its node was away, maybe the only copy of the data. Its
instance monitor carries the issue in its new issues field, shown under the
instance in the status board, saying to purge it there or to keep it by an
edit or a touch of its config. The instance is flagged stopped once per
tombstone, unless up, decided once its status is known, so the daemon does
not start it on its own, and a start, a restart, a placement or a provision
is refused with the issue, as the ha and boot decisions are; a purge, a
delete, a stop or an unprovision is accepted. The instance is never
stopped, nor frozen, for it. The issue goes with the holder cleared, purged
there or kept. A flag not written is decided again at the next event.
…drop it with --force

A tombstone with pending holders says nodes still hold the instance of the
object of that name, purged or deleted elsewhere: a new object of the name
would meet theirs. A create is refused naming them: a delete in progress on
live holders, to retry once done, without suggesting --force, as the new
instances would join that delete, or nodes away holding it, to purge there.
The --force of a create, which replaces the config of an object that
exists, also drops the tombstones of the path, its holders cleared as
dropped, as they are once the config is written, one recorded by a delete
since the check included, the record written and published before the
create answers: a create refused or failed leaves the tombstone, and a
crash between the write and the drop leaves it pending, the safe side, a
create --force again dropping it. A dry run drops nothing.
…iled, the peers done

A live holder adopts the delete or the purge of its tombstone once. Its
delete failing, the peers having deleted theirs, the object lived on that
node only, with no issue shown, until its daemon restarted. The live holder
of the node now turns away, once per orchestration, when its instance lands
in a failure state ending the delete or the purge of the tombstone, delete
failed, or stop or purge failed, not one the delete goes on from, as a stop
failure: the instance is flagged stopped, the issue shown, and the operator
purges it there or keeps it, as for a node that was away. The failure state
an action ends in turns it away as well as the one transitionTo sets.

An instance never adopts the delete or the purge of a tombstone it carried
itself, as one failed then aborted: adopting it again ran what the operator
stopped (qatests 308, the purge aborted run again, the purge asked next
refused as one in progress). Its live holder turns away instead.
…ed again under the path not to be deleted by it

A tombstone names the path, and an object created again under it, with
create --force or once the record is gone, had the same path: a live holder
adopting the delete of a stale copy of the record, as one a peer killed
before it wrote its last merge publishes again after the node restarted or
forgot the removal, deleted the new object.

The instance config carries the id of the object, its DEFAULT.id, and the
record the id of the object it deletes. A live holder whose instance is of
another object does not adopt the delete: its holder is cleared, as an
away holder whose config is of another object is. An id unknown on either
side, as of a record written by a former version, tells nothing, as
before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant