Repository navigation
Conversation
| // SyncDir syncs the directory of the records to stable storage: the | ||
| // renames and removals made in it. | ||
| func SyncDir(dir string) error { | ||
| d, err := os.Open(dir) |
There was a problem hiding this comment.
Potential file inclusion attack via reading file - high severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.
Show fix
| d, err := os.Open(dir) | |
| for _, seg := range strings.Split(filepath.ToSlash(dir), "/") { | |
| if seg == ".." { | |
| return fmt.Errorf("invalid file path") | |
| } | |
| } | |
| d, err := os.Open(dir) |
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info
There was a problem hiding this comment.
Not reachable by user input: dir is always tombstone.Dir(), <var>/node/tombstone under the node var directory of the node config, and no caller passes another. The files read are the entries os.ReadDir lists in that directory, each name then checked against the id of the record it holds, and the ids of the records written or removed go through recordFile, which refuses an empty id, ., .. and any / or \. Nothing an API client sends names a path here.
| if e.IsDir() || strings.HasPrefix(name, ".") || !strings.HasSuffix(name, ".json") { | ||
| continue | ||
| } | ||
| b, err := os.ReadFile(filepath.Join(dir, name)) |
There was a problem hiding this comment.
Potential file inclusion attack via reading file - medium severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.
Show fix
Remediation: Ignore this issue only after you've verified or sanitized the input going into this function.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info
There was a problem hiding this comment.
Not reachable by user input: dir is always tombstone.Dir(), <var>/node/tombstone under the node var directory of the node config, and no caller passes another. The files read are the entries os.ReadDir lists in that directory, each name then checked against the id of the record it holds, and the ids of the records written or removed go through recordFile, which refuses an empty id, ., .. and any / or \. Nothing an API client sends names a path here.
| if len(pending) == 0 { | ||
| return nil, nil | ||
| } | ||
| return st.clear(path, pending, tombstone.ClearedByDropped), nil |
There was a problem hiding this comment.
🟡 Medium - Forced recreation does not durably remove the old tombstone
A forced create clears the tombstone in memory immediately before committing the new config, but tombmon.Drop only queues the changed record and does not flush it to stable storage. If the daemon crashes after the new config is committed and before that flush, restart reloads the old pending tombstone and continues suppressing or adopting the newly recreated object.
Show fix
Make forced recreation wait for the tombstone update/removal to be durably flushed before acknowledging the new object, and use a transactional or recoverable ordering so a crash cannot leave a committed recreation paired with the old pending tombstone.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Partially fixed in commit fbdcbcb.
This issue remains because writeObjectConfigFile commits the new config at daemon/daemonapi/lib_object_config_file.go:129 before invoking check.afterCommit at line 134; the callback then calls tombmon.Drop at daemon/daemonapi/post_object_config_file.go:103, whose flush at daemon/tombmon/main.go:800 is therefore still reachable only after a crash window in which the committed config coexists with the old durable tombstone.
There was a problem hiding this comment.
The window is real and the order is deliberate. Two files cannot be written atomically, so one side of a crash has to be chosen: here it leaves the new config with the old tombstone still pending, which only withholds. Peers fetch nothing from a pending holder, an away holder never deletes on its own, and a creating node that was itself a holder is demoted to away at restart, so it does not adopt the old delete. Running create --force again drops the tombstone; it is the same state and remedy as the "created, and its tombstone not dropped" answer when the flush fails. The reverse order leaves the old object unprotected on any failed commit (a write conflict is enough): its config propagates again from the holder, and a cleared holder cannot be restored. Closing the window needs an intent journal, which we judge not worth it for a millisecond crash window with a safe outcome and a documented retry (post_object_config_file.go, the comment above the tombstone check).
There was a problem hiding this comment.
Partially fixed in commit c0bbe34.
This issue remains because PostObjectConfigFile commits the config before invoking afterCommit (daemon/daemonapi/lib_object_config_file.go:129-135), while tombmon.Drop only starts its durable tombstone flush afterward (daemon/tombmon/main.go:803-808); a crash in that interval still leaves the committed recreation paired with the old pending tombstone. The added flush is a partial fix, but it does not make the ordering crash-safe.
There was a problem hiding this comment.
Partially fixed in commit d2e46c0.
This issue remains because lib_object_config_file.go:129-135 commits the recreated config before invoking tombmon.Drop, so a daemon crash in that gap restarts with the old pending tombstone; Drop's flush at daemon/tombmon/main.go:808 only makes removal durable after it is reached.
There was a problem hiding this comment.
Partially fixed in commit 787c9d6.
This issue remains because daemon/daemonapi/lib_object_config_file.go:129 commits the new config before invoking afterCommit at line 134; a crash in that gap leaves the old tombstone on disk, and tombmon.Drop's flush at daemon/tombmon/main.go:808 cannot run. The added synchronous flush only narrows the window after Drop starts.
There was a problem hiding this comment.
Partially fixed in commit 05be262.
This issue remains because writeObjectConfigFile commits the new config at daemon/daemonapi/lib_object_config_file.go:129 before invoking the forced-create callback at :134, so a crash before tombmon.Drop at daemon/daemonapi/post_object_config_file.go:106 starts still leaves the old pending tombstone despite Drop now flushing after it runs.
| t.tombstones = tombstone.Data.MergedOf(t.path.String()) | ||
| t.applyTombstoneIssue() | ||
| for n, v := range instance.StatusData.GetByPath(t.path) { | ||
| t.instStatus[n] = *v | ||
| } |
There was a problem hiding this comment.
🟡 Medium - Startup flags an up tombstone holder as stopped
When an imon starts on a node whose instance is actually up and has an away-holder tombstone, startup applies the tombstone issue before loading the local instance status cache. The availability check therefore sees the zero-value status and writes the stopped flag, suppressing later automatic starts for an instance that was never down until an operator clears it.
Show fix
| t.tombstones = tombstone.Data.MergedOf(t.path.String()) | |
| t.applyTombstoneIssue() | |
| for n, v := range instance.StatusData.GetByPath(t.path) { | |
| t.instStatus[n] = *v | |
| } | |
| for n, v := range instance.StatusData.GetByPath(t.path) { | |
| t.instStatus[n] = *v.Value | |
| } | |
| t.tombstones = tombstone.Data.MergedOf(t.path.String()) | |
| t.applyTombstoneIssue() |
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in c0bbe34: the statuses are read before the tombstones at start, and the stopped flag is decided only once the local instance status is known, re-checked when it arrives; an up instance shows the issue and is never flagged. Checked on dev2 with an away holder up on its return.
| case errors.Is(err, unix.ENOBUFS): | ||
| t.log.Warnf("address events of the other network namespaces lost: receive buffer overflow") | ||
| continue | ||
| default: | ||
| t.log.Warnf("address events of the other network namespaces: receive: %s", err) | ||
| select { | ||
| case <-t.ctx.Done(): | ||
| return | ||
| case <-time.After(resubscribeDelay): | ||
| } | ||
| continue |
There was a problem hiding this comment.
🟡 Medium - NSID event loss leaves stale deduplication state across namespace reuse
When the all-namespace netlink socket overflows or receives an error, nsidWorker keeps using the same socket and state map instead of recreating the subscription and resetting state. If the lost messages include RTM_DELNSID, a later namespace can reuse that numeric ID and have its first matching address-add event suppressed as a duplicate, so imon misses the address and may keep the instance status stale until a scheduled refresh.
Show fix
| case errors.Is(err, unix.ENOBUFS): | |
| t.log.Warnf("address events of the other network namespaces lost: receive buffer overflow") | |
| continue | |
| default: | |
| t.log.Warnf("address events of the other network namespaces: receive: %s", err) | |
| select { | |
| case <-t.ctx.Done(): | |
| return | |
| case <-time.After(resubscribeDelay): | |
| } | |
| continue | |
| case errors.Is(err, unix.ENOBUFS): | |
| t.log.Warnf("address events of the other network namespaces lost: receive buffer overflow") | |
| state = make(map[int]map[string]bool) | |
| continue | |
| default: | |
| t.log.Warnf("address events of the other network namespaces: receive: %s", err) | |
| state = make(map[int]map[string]bool) | |
| select { | |
| case <-t.ctx.Done(): | |
| return | |
| case <-time.After(resubscribeDelay): | |
| } | |
| continue |
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in c0bbe34: the per-namespace state is forgotten on a buffer overflow and on a receive error, the deletion of a namespace id possibly among the events lost, so the next namespace getting the id publishes its first address.
| if err := tombstone.RemoveFile(dir, id); err != nil { | ||
| log.Errorf("remove the tombstone %s: %s", id, err) | ||
| } |
There was a problem hiding this comment.
🟡 Medium - Failed tombstone removal is acknowledged as successful
When a forced recreation clears a tombstone, flush logs a RemoveFile or removal-only directory-sync failure but does not return or retry it, so Drop reports success after the new config is committed while the old pending tombstone can remain on disk. A restart then reloads and republishes that stale tombstone, blocking the recreated object from peers or causing it to be treated as still tombstoned.
Show fix
Record removal and directory-sync failures in the returned error map, preserve/requeue the corresponding removed entries for retry, and make Drop fail until the removal state is durably persisted.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in c0bbe34: a removal or a directory sync failing is now an error of the flush, Drop's included, and the removal is queued again for the next flush.
| func (t *Manager) hasNonLeaderProvisioned() bool { | ||
| for node, otherInstStatus := range t.instStatus { | ||
| if t.isUnprovisionLeaderNode(node) { | ||
| if t.isUnprovisionLeaderNode(node) || t.isNodeAway(node) { |
There was a problem hiding this comment.
🟡 Medium - Unprovision releases shared resources after a holder goes away
After an unprovision request is accepted, a provisioned non-leader can enter maintenance, upgrade, or shutdown before the leader acts. The leader now skips that away peer when deciding whether to release shared resources, so it can unprovision while the peer still uses them; the request-time refusal is not rechecked and unprovision creates no tombstone to protect the missed holder.
| if t.isUnprovisionLeaderNode(node) || t.isNodeAway(node) { | |
| if t.isUnprovisionLeaderNode(node) { |
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Confirmed, fixed in c0bbe34. The refusal for an away node holding a provisioned instance is checked when the request is accepted, and the leader then skipped any away peer, so a peer entering maintenance or restarting its daemon after acceptance was not waited for. The leader now skips an away peer only when the request was forced; otherwise it holds for a provisioned one, as before the change, until the peer is back and runs its part or the maintenance grace period drops its data. The suggestion as written would also hold a forced unprovision, which the refusal sends there on purpose, so the skip is kept for that case.
| if bt.AppArmorProfile != "" { | ||
| // The --security-opt of run_args setting another profile is | ||
| // dropped, the others kept. | ||
| runArgs.DropOptionAndMatchingValue("--security-opt", "^apparmor[=:]") | ||
| a.Append("--security-opt", "apparmor="+bt.AppArmorProfile) | ||
| } |
There was a problem hiding this comment.
🟡 Medium - Equals-form run_args overrides apparmor_profile
When a root-managed container sets apparmor_profile and includes --security-opt=apparmor=other in run_args, the new filter misses the equals form and appends it after the explicit profile option. Docker and Podman apply the later single-valued AppArmor option, so the container runs under other rather than the profile named by the keyword and the keyword no longer reliably controls or describes the container.
Show fix
| if bt.AppArmorProfile != "" { | |
| // The --security-opt of run_args setting another profile is | |
| // dropped, the others kept. | |
| runArgs.DropOptionAndMatchingValue("--security-opt", "^apparmor[=:]") | |
| a.Append("--security-opt", "apparmor="+bt.AppArmorProfile) | |
| } | |
| if bt.AppArmorProfile != "" { | |
| // The --security-opt of run_args setting another profile is | |
| // dropped, the others kept. | |
| filtered := make([]string, 0, len(runArgs.Get())) | |
| for _, arg := range runArgs.Get() { | |
| if strings.HasPrefix(arg, "--security-opt=apparmor=") || strings.HasPrefix(arg, "--security-opt=apparmor:") { | |
| continue | |
| } | |
| filtered = append(filtered, arg) | |
| } | |
| runArgs.Set(filtered) | |
| runArgs.DropOptionAndMatchingValue("--security-opt", "^apparmor[=:]") | |
| a.Append("--security-opt", "apparmor="+bt.AppArmorProfile) | |
| } |
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in c0bbe34: the --security-opt=apparmor=... and apparmor:... equals forms of run_args are dropped too when apparmor_profile is set, the other security options kept.
| if !ended { | ||
| t.log.Infof("context done, stopping netlink monitor") | ||
| return | ||
| } |
There was a problem hiding this comment.
🟡 Medium - Resubscribe retains stale netlink deduplication state
When the host-only netlink subscription closes after a receive error, the worker resubscribes but keeps the address and link state accumulated before the outage. If the missed events remove and recreate an address or link with the same name and state, the first post-recovery event is classified as a duplicate and no refresh is published, leaving the matching instance resource stale until an unrelated scheduled refresh.
| } | |
| t.lastPublished = make(map[string]linkPublishState) |
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in d2e46c0: what was published is forgotten at each subscription, so an address or link removed and added back while the events were missed is published again.
| key := fmt.Sprintf("%d:%s", ev.linkIndex, ev.address) | ||
| last, ok := state[ev.nsid] | ||
| if !ok { | ||
| last = make(map[string]bool) | ||
| state[ev.nsid] = last | ||
| } | ||
| if added, ok := last[key]; ok && added == ev.added { | ||
| continue | ||
| } | ||
| last[key] = ev.added |
There was a problem hiding this comment.
🟡 Medium - All-namespace netlink deduplication grows with every address churn event
The all-namespace listener stores a permanent key for every unique link-index/address operation in each live namespace and never removes keys when an address is deleted. A process able to churn unique addresses in a monitored namespace can therefore grow this daemon-owned map without bound, eventually consuming memory and destabilizing netmon or the daemon.
Show fix
Bound the per-namespace deduplication history or delete an address key after its corresponding delete has been processed while retaining only the minimal state needed to suppress duplicate kernel notifications.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in d2e46c0: only the addresses present are kept, a delete forgetting its address, so churned addresses no longer grow the state of a namespace; a delete repeated is published again, which only refreshes.
| t.tombstoneStopped = id | ||
| if t.instStatus[t.localhost].Avail.Is(status.Up, status.Warn) { | ||
| return | ||
| } | ||
| t.log.Infof("flag the instance stopped: %s", issue) | ||
| _ = t.setStopped() |
There was a problem hiding this comment.
🟡 Medium - Tombstone stopped decision is lost when the flag write fails
When an away-holder tombstone is first applied to a locally non-up instance, the monitor records the tombstone ID before attempting to persist the stopped flag and discards the write error. If that write fails, later status or tombstone events skip the ID permanently, so clearing the tombstone can allow the instance to start without the operator's required stopped decision.
Show fix
| t.tombstoneStopped = id | |
| if t.instStatus[t.localhost].Avail.Is(status.Up, status.Warn) { | |
| return | |
| } | |
| t.log.Infof("flag the instance stopped: %s", issue) | |
| _ = t.setStopped() | |
| if t.instStatus[t.localhost].Avail.Is(status.Up, status.Warn) { | |
| return | |
| } | |
| t.log.Infof("flag the instance stopped: %s", issue) | |
| if err := t.setStopped(); err != nil { | |
| return | |
| } | |
| t.tombstoneStopped = id |
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in d2e46c0: the tombstone is taken as decided only once the stopped flag is written, or the instance seen up; a failed write is decided again at the next status or tombstone event.
| if t.seenCarrying[n] == id && !holding[n] { | ||
| continue |
There was a problem hiding this comment.
🟡 Medium - Stale carrying state omits a peer from the delete tombstone
If a peer has announced that it is carrying the current delete orchestration and then disappears before the local tombstone is recorded, peer-loss handling removes its cached instance data but leaves its seen-carrying ID. The tombstone builder then interprets the missing cache entry as proof that the peer deleted its instance and omits that node, so the peer can return with the old instance and reintroduce the object without tombstone protection.
Show fix
Clear seenCarrying for a node when its instance monitor is deleted/forgotten, and treat a carrying peer whose data disappeared because the peer was lost as an away holder unless deletion is independently confirmed.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in d2e46c0: a peer seen running the delete with no instance left counts as deleted only while its node is up. One away or lost, its data dropped with it, is an away holder, cleared on its return if its instance is gone there.
There was a problem hiding this comment.
Partially fixed in commit d2e46c0.
This issue remains because daemon/imon/main_cmd.go:934-944 removes a peer's instMonitor without clearing seenCarrying (and onForgetPeer at :896-900 also leaves it), while daemon/imon/tombstone.go:184 still omits that peer whenever its nodeMonitor is momentarily still up during dropPeer event processing; the isNodeUp guard only handles an already-away node, not this race.
There was a problem hiding this comment.
Confirmed, fixed in 787c9d6. dropPeer now drops the node monitor of the peer before its instance data, and the holder check reads the node as up only when both the monitor of the imon and the node data say so: a reader finding the instance data of a dropped peer gone finds its node gone too, so a delete recording its tombstone during the drop names the peer an away holder. Checked on dev2 with a node frozen until dropped, then purged: recorded away, the issue shown on its return, and cleared by a purge there.
…ve error, rather than read the closed channel in a loop The netlink library ends a subscription on a receive error, as on a receive buffer overflow, closing its channel. The worker read the closed channel for good: a busy loop, and no link or address event until the daemon restarted. The worker now sees the channel closed, logs it, and subscribes to both again after a second, the events in between missed. The subscription errors are logged. What was published before is forgotten at each subscription: the events missed may have removed and added back the same address or link, whose next event would else read as a repeat.
d2e46c0 to
787c9d6
Compare
| t.log.Warnf("address events of the other network namespaces: receive: %s", err) | ||
| forgetNsState(state) | ||
| select { | ||
| case <-t.ctx.Done(): | ||
| return | ||
| case <-time.After(resubscribeDelay): | ||
| } | ||
| continue |
There was a problem hiding this comment.
🟡 Medium - Fatal NSID receive errors permanently disable container address monitoring
When the all-namespace netlink socket returns any error other than EAGAIN, EINTR, or ENOBUFS, the worker waits and then loops on the same descriptor instead of closing and recreating the subscription. If that descriptor is no longer usable, all subsequent address changes in container namespaces are silently lost until the daemon restarts, leaving matching instance resources stale.
Show fix
Treat a non-retryable receive error as subscription loss: close the descriptor, wait for the resubscribe delay, open a fresh all-NSID socket, and reset the per-NSID state before receiving again. Keep retrying transient open failures while the daemon is running.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in 05be262: a receive error but a timeout, an interruption or an overflow closes the all-namespace socket and opens a new one after the resubscribe delay, again until it opens or the daemon stops, the per-namespace state forgotten.
| if !t.state.State.IsOneOf(instance.MonitorStatesFailure...) { | ||
| return | ||
| } | ||
| switch t.state.GlobalExpect { | ||
| case instance.MonitorGlobalExpectPurged, instance.MonitorGlobalExpectDeleted: | ||
| default: | ||
| return | ||
| } |
There was a problem hiding this comment.
🟡 Medium - Recoverable purge failures demote a live tombstone holder
demoteTombstoneIfFailed treats every generic instance failure state as a failed delete/purge, including freeze, provision, start, stop, and unfreeze failures. During a purge, several of those states are explicitly recovered by returning to idle and continuing, but this hook first converts the local live holder to away, causing a false tombstone issue and potentially setting the stopped flag or blocking later starts/provisions while the purge is still recoverable.
Show fix
| if !t.state.State.IsOneOf(instance.MonitorStatesFailure...) { | |
| return | |
| } | |
| switch t.state.GlobalExpect { | |
| case instance.MonitorGlobalExpectPurged, instance.MonitorGlobalExpectDeleted: | |
| default: | |
| return | |
| } | |
| switch t.state.GlobalExpect { | |
| case instance.MonitorGlobalExpectPurged: | |
| if !t.state.State.IsOneOf(instance.MonitorStateStopFailure, instance.MonitorStatePurgeFailed) { | |
| return | |
| } | |
| case instance.MonitorGlobalExpectDeleted: | |
| if t.state.State != instance.MonitorStateDeleteFailure { | |
| return | |
| } | |
| default: | |
| return | |
| } |
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in 05be262: the live holder turns away only on the failures ending the orchestration, delete failed for a delete, stop or purge failed for a purge; a delete going on after a stop, start, freeze or provision failure, as from idle, leaves it live.
| func (s *state) onPeerRecord(rec tombstone.Record) { | ||
| if _, ok := s.records[rec.OrchestrationID]; !ok { | ||
| if !rec.HasPending() { | ||
| return | ||
| } | ||
| if g, ok := s.gone[rec.OrchestrationID]; ok { | ||
| // Removed here: a stale copy, in flight or published | ||
| // again by a peer that missed the removal. Answered when | ||
| // it names the node pending: its holder was cleared, | ||
| // the node alone can say it. | ||
| if g.by == "" || !rec.IsPending(s.localhost) { | ||
| return | ||
| } | ||
| rec = rec.DeepCopy() | ||
| h := rec.Holders[s.localhost] | ||
| h.State = tombstone.HolderCleared | ||
| h.ClearedBy = g.by | ||
| rec.Holders[s.localhost] = h | ||
| s.log.Infof("tombstone %s of %s: a peer publishes it again, holder %s cleared, %s", rec.OrchestrationID, rec.Path, s.localhost, g.by) | ||
| s.upsert(rec) | ||
| return | ||
| } | ||
| rec = rec.DeepCopy() | ||
| if !s.localRejoined() { | ||
| // Adopted before the rejoin, as by a node that was down: | ||
| // its instance is not trusted to complete the delete. | ||
| demote(&rec, s.localhost) | ||
| } | ||
| } | ||
| s.upsert(rec) |
There was a problem hiding this comment.
🟡 Medium - Restart forgets removed tombstones and allows stale peer copies to resurrect
A node remembers removed orchestration IDs only in the in-memory gone map, which is empty after tombmon restarts and expires after one hour. When a down or evicted peer later republishes its durable pending tombstone, the restarted node adopts it as a new record; if the record names the node as a live holder, imon can adopt the old delete/purge orchestration and delete or block a current object with the same path.
Show fix
Persist removal generations/cleared markers alongside tombstone records, or otherwise retain durable suppression for stale orchestration IDs across restart and beyond the in-memory grace period. A returned peer's old pending copy must be merged with the known cleared state before it can be adopted.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in 05be262, durably rather than by remembering removals: the instance config now carries the object id (DEFAULT.id) and the tombstone the id of the object it deletes. A live holder whose instance is of another object, created again under the path, never adopts the old delete, its holder cleared instead, as an away holder whose config is of another object is; so a stale copy coming back after a restart, or days later, cannot delete or block a recreated object. Unknown ids, as in a record of a former version, tell nothing, as before.
There was a problem hiding this comment.
Partially fixed in commit 05be262.
This issue remains because Start initializes the removal suppression only in the in-memory gone map at daemon/tombmon/main.go:154, while onPeerRecord at daemon/tombmon/main.go:452-480 adopts any pending peer record after restart when gone is empty and the local node has rejoined, so a stale copy naming this node as a live holder can still resurrect the orchestration.
| // the configuration is written: a create refused or failed leaves it, | ||
| // and a crash between the two leaves it pending, the safe side, a | ||
| // create --force again dropping it. | ||
| pending := tombstone.Data.PendingHolders(p.String(), "") |
There was a problem hiding this comment.
🟡 Medium - Concurrent tombstone creation can leave a committed create blocked
The POST handler snapshots pending holders before validation and installs the post-commit Drop callback only when that snapshot was nonempty. If a delete or purge records a tombstone after the snapshot but before the configuration commit, the new configuration commits successfully without dropping that tombstone, so propagation and discovery continue withholding the recreated object until another forced create or manual cleanup.
Show fix
Serialize the tombstone check/commit/drop sequence, or recheck for newly recorded tombstones immediately after the config commit and drop only the tombstone state that was part of this create transaction. The operation should not acknowledge a create while a concurrent pending tombstone remains unhandled.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in 05be262 for a forced create: the drop now runs after every forced commit, and Drop reads the pending holders then, so a tombstone recorded by a delete after the check is dropped too. An unforced create racing a delete of the same path stays as is: the outcome only withholds the new object, the safe side, and create --force resolves it.
| objectaction.WithSort(t.Sort), | ||
| objectaction.WithObjectSelector(mergedSelector), | ||
| objectaction.WithAsyncTarget("deleted"), | ||
| objectaction.WithAsyncTargetOptions(instance.MonitorGlobalExpectOptionsDeleted{Force: t.Force}), |
There was a problem hiding this comment.
🟡 Medium - Node-selected delete ignores the new force flag
When om delete --node or ox delete runs through the node-selected remote path, the new --force value is stored only in the async target options, while the custom remote callback sends PostInstanceActionDelete with a params struct containing only the session id. The target node therefore applies the normal away-holder refusal instead of honoring --force, so a forced delete fails or waits when the peer instance is away.
Show fix
Forward the force intent through the remote action API, or route node-selected deletes through the object-action endpoint that accepts MonitorGlobalExpectOptionsDeleted{Force:true}. Add an integration test for --node <away-node> --force and verify the target receives the force option.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Not reachable: with --node, delete is an instance action, PostInstanceActionDelete running the delete of the instance on that node, and no orchestration. The refusal of an away node holding its instance up or provisioned, and the force overriding it, are checked by the instance monitor when it accepts the deleted global expect of an orchestration (daemon/imon/main_cmd.go); the instance action never goes through it, so there is no refusal for --force to lift on that path.
There was a problem hiding this comment.
Partially fixed in commit 05be262.
This issue remains because core/oxcmd/object_delete.go:46-51 still builds api.PostInstanceActionDeleteParams with only SessionID and sends it through the node-selected remote callback, while the Force value is only applied to the local async target options at line 35; the delete API has no force field for this path.
| func DemoteLocal(path string) error { | ||
| mu.Lock() | ||
| defer mu.Unlock() | ||
| if st == nil { | ||
| return ErrNotStarted | ||
| } | ||
| st.adoptPeerCopies(path) | ||
| for _, rec := range st.recordsOf(path) { | ||
| rec = rec.DeepCopy() | ||
| if demote(&rec, st.localhost) { | ||
| st.log.Infof("tombstone %s of %s: holder %s away, its instance failed to complete the %s", rec.OrchestrationID, rec.Path, st.localhost, rec.Kind) | ||
| st.upsert(rec) | ||
| } | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // adoptPeerCopies adopts the copies the peers publish of the records of | ||
| // path the node does not keep and that have a pending holder, even one it | ||
| // removed: an explicit clear of a stale copy is what ends it. The caller | ||
| // holds mu. | ||
| func (s *state) adoptPeerCopies(path string) { | ||
| for id, rec := range tombstone.Data.MergedOf(path) { | ||
| if _, ok := s.records[id]; ok || !rec.HasPending() { | ||
| continue | ||
| } | ||
| s.upsert(rec) | ||
| } |
There was a problem hiding this comment.
🟡 Medium - Purge failure re-adopts stale tombstones for unrelated orchestration IDs
When a delete or purge failure demotes the local live holder, DemoteLocal first adopts every pending peer record for the path, including records previously removed and records for older orchestration IDs, then demotes all local live holders. A stale peer copy can therefore be resurrected and surfaced as an away-holder issue for the current object even though the failure only concerns one orchestration, causing a false stopped/blocking decision and potentially interfering with the current recreation.
Show fix
Pass the failed orchestration ID to DemoteLocal and update only that record. Do not adopt peer copies marked recently removed by WasRemoved, and preserve the normal stale-copy suppression when importing records for failure handling.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Fixed in 05be262: DemoteLocal takes the orchestration id of the failed delete or purge and turns away the holder of the node in that record only, other records of the path left alone, and a record the node removed is not taken in again for it.
| func (t *Manager) onMyInstanceStatusUpdated(srcNode string, srcCmd *msgbus.InstanceStatusUpdated) { | ||
| // The node publishes its status again: its last one, kept while it | ||
| // was away, is stale. | ||
| delete(t.awayStatus, srcCmd.Node) | ||
| if srcCmd.Node == t.localhost { | ||
| // Armed from the status kept, which may be newer than the one | ||
| // of the event: the first refresh stores its status before its |
There was a problem hiding this comment.
🟡 Medium - Stale peer status clears the away-holder protection
When a peer's instance status is deleted while the node is away, the new handler saves the last protected status in awayStatus, but any later InstanceStatusUpdated immediately deletes that saved value before validating freshness or confirming the node has left its away/rejoin state. A queued stale down/unprovisioned status can therefore replace the protected up/provisioned state, allowing a non-forced delete, purge, or unprovision to be accepted while the peer may still hold or use the instance.
Show fix
| func (t *Manager) onMyInstanceStatusUpdated(srcNode string, srcCmd *msgbus.InstanceStatusUpdated) { | |
| // The node publishes its status again: its last one, kept while it | |
| // was away, is stale. | |
| delete(t.awayStatus, srcCmd.Node) | |
| if srcCmd.Node == t.localhost { | |
| // Armed from the status kept, which may be newer than the one | |
| // of the event: the first refresh stores its status before its | |
| func (t *Manager) onMyInstanceStatusUpdated(srcNode string, srcCmd *msgbus.InstanceStatusUpdated) { | |
| if _, ok := t.awayStatus[srcCmd.Node]; ok { | |
| // Keep the protected status until NodeMonitorUpdated confirms the node is back. | |
| return | |
| } | |
| if srcCmd.Node == t.localhost { | |
| // Armed from the status kept, which may be newer than the one | |
| // of the event: the first refresh stores its status before its |
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
Not reachable as described: a peer away publishes no status, its status being deleted is what saved awayStatus, and the next InstanceStatusUpdated of that peer can only come from the peer itself once its daemon runs again, rejoining: a fresh status of its instance, not a queued stale one, the events of a peer being delivered in order. That status is the truth the refusal is to judge by, so it replaces the saved one; the saved status is also forgotten when the node is seen back (onNodeMonitorUpdated).
There was a problem hiding this comment.
Partially fixed in commit 05be262.
This issue remains because daemon/imon/main_cmd.go:289-292 still unconditionally deletes awayStatus for every InstanceStatusUpdated before the UpdatedAt freshness check and without checking the peer's away/rejoin state; the added NodeMonitorUpdated cleanup does not prevent a queued stale update from clearing protection.
…as the ones of the ip.netns and ip.cni containers The netlink subscription heard the namespace of the daemon only: an address deleted or added in the namespace of a container, as the one of an ip.netns or ip.cni resource, published no event, and the instance status saw it at its next scheduled refresh only. A second socket listens to every namespace the namespace of the daemon holds an id for, which moving a link into a namespace gives it, as every mode of the ip drivers does (NETLINK_LISTEN_ALL_NSID). The kernel says the namespace of a message by that id. The listener publishes the addresses of global scope of the other namespaces, with the id in a new nsid field of NetIPAddrAdded and NetIPAddrDeleted, omitted for the namespace of the daemon, and the link named by its label, or nsid<N>-index-<I> for an ipv6 address, whose message names no link. imon matches them to the resources by address, as before. The events of the namespace of the daemon, which the socket hears too, are dropped: the subscription of the worker publishes them. It stays as it was, a burst of container events overflowing the buffer of the second socket not losing an event of the node. The last event of each address of a namespace is forgotten as its id is deleted: a namespace going deletes its addresses with no message, and the next one may get the same id. All of them are forgotten as events are lost, a buffer overflow or a receive error, the deletion of an id possibly among them. Only the addresses present are kept: a delete forgets its address, for addresses churned in a namespace not to grow the state. Without the option, as on a kernel older than 4.2 or a daemon without CAP_NET_BROADCAST, the other namespaces are not heard, as before. A receive error but a timeout, an interruption or an overflow closes the socket and opens a new one after a second, again until it opens, as the host subscription is subscribed again.
…docker or podman container runs under On the ubuntu 7.0 kernels, with kernel.apparmor_restrict_unprivileged_unconfined=1, the containers are labeled docker-default//&unconfined, and the default profile of the engine, allowing the signals to its own name only, denies the processes of a container signalling one another (moby/moby#54099): the haproxy master of the igw_haproxy gateway can not stop its old workers on a reload, and they go on serving the certificate loaded before it. apparmor_profile names the profile the processes of the container run under, instead of the default of the engine: a profile loaded on the node, or unconfined. Docker and podman are given --security-opt apparmor=<profile>, the apparmor option of run_args dropped, in the --security-opt=<value> form too, its other security options kept. The status warns when the profile the engine reports differs. A keyword of its own rather than a list of engine options, which run_args already passes: the profile means the same for every engine, a containerd one too, and the status can check it. Setting it requires the root grant, as run_args and privileged do: it decides what the container reaches of the node.
The session and nmon each listed the node monitor states an orchestration does not wait for: a node down, starting, rejoining or shutting down, which runs no orchestration and catches up as it rejoins. The list is now node.MonitorStatesAway, with MonitorState.IsAway, for the tombstones and the instance monitor to use the same. MonitorState.IsAwayFromObjects adds the upgrade of the packages of the node, its daemon restarting: it runs no object orchestration, while the upgrade, a node orchestration, does not count it away. The session and the object code judge with it.
…de away The leader of an unprovision or a purge waited for the non-leader instances to be unprovisioned, reading the status a node in maintenance retains: the orchestration hung until the maintenance grace period dropped that data. The instance of a node away is not waited for any more.
…tarts the config managers The discovery loop installed each fetched config with a sync of the file, waiting on the disk, and the same loop starts the config manager of every object. With 850 objects created then deleted at once, the disk busy with the deletes, the managers started up to 84s after their config landed: the delete ran meanwhile on the nodes whose instance had no manager yet, was not seen there, and the accepting node fetched the configs back from them, 208 objects coming back. The fetch syncs the file it writes beside the config file, and the loop only renames it in place (xconfig.InstallSyncedFile), under the lock. A crash before the rename is on disk leaves the former state, which the peer serves again, never an empty file.
…s, for the managers of many objects not to wait on the busy bus once per filter The bus serves the operations of every component of the daemon one at a time, and starting the managers of an object made one operation per filter: about 43, for its config manager, object monitor and instance monitor. With 850 objects created and deleted at once, those starts queued behind the busy bus, which held up the discovery loop starting the config managers. Subscription.AddFilters adds a list of filters, built with Filters.Add, in one operation, each taking effect at once as AddFilter does. The config manager, the object monitor and the instance monitor start with it. On dev2 the 850-object create then delete left 1 object back instead of 29 with the sync fix alone, the managers starting 22s after their config landed on average instead of 30s. The instance monitor adds with it the filters of the relations, the affinities and the volumes of its object, before it reads their statuses, and the api the filters of its waits for a config to propagate and for an object to be known, which every create --wait set up, and of an event stream, one per filter the client asks for.
…e object of the event only, with no copy of its data While a delete or a purge waited for its object to be gone, the session checked every orchestration waiting at each event, each check copying the config, status and monitor of every node of the object, the statuses with all their resources, to read the node names. With 850 objects deleted at once, the session fell behind the events, its subscription queue reaching the limit past which the daemon panics (615 warnings on a node of dev2). An event about an object checks the deletes of that object only; an event about a node, and the ticker, check them all. remainingInstances reads the node names with instance.Data.GetNodesByPath, which copies nothing. On dev2 the same deletes gave 0 to 2 warnings per node. The api handlers reading the nodes of the data of an object, or their number, and the hard affinity check of the instance monitor read them with GetNodesByPath too, rather than copy every node's monitor, config or status.
…bject did not run A record names the nodes still holding the instance of an object a delete or a purge did not reach: live, up but not seen running the orchestration, as one whose config manager had not started yet, or away, down or in maintenance. Each holder is pending until its instance is gone or kept. The copies of a record merge per holder, a cleared holder winning, of two cleared a fixed rank, away over live: commutative, associative and idempotent, every node ends with the same record. The records are kept under <var>/node/tombstone/<orchestration id>.json, written atomically, and the package holds the records every node publishes, the pending holders of a path read from all the copies merged.
The records a node keeps are published under its data, cluster.node.<node>.tombstone.<id>, with the NodeTombstoneUpdated and NodeTombstoneDeleted events, wired as the pool statuses are: forwarded to the peers, applied from a peer patch, compared against a peer full data as they are, a record carrying no update time, and dropped with a dropped peer. The data of a peer dropped goes with its node monitor first, then its instance data: a reader finding the instances of the peer gone finds its node gone too, and does not take them for deleted by a node still up, as a delete recording its tombstone.
The tombmon manager loads the records of the node at start and publishes them before the node monitor leaves the rejoin; a live holder of the node turns away there, its daemon having restarted. It merges the copies its peers publish, adopting a record it does not keep only with a pending holder, and takes in by a sweep every 10s the ones it missed as events. It turns away the live holders whose node goes away, clears the holders no longer cluster nodes as evicted, and removes a record once all its holders are cleared, every cluster node is up, and no node publishes it with a pending holder. Once the node monitor is out of its start and rejoin, the away holder of the node is cleared as purged or deleted when the config file and config data of the instance are gone there, and as kept when its config file changes while the instance is quiet: its monitor not purging, deleting or unprovisioning, idle, and no action of it running but a status refresh, which a touch itself triggers. A change made otherwise takes the baseline again. The baseline is the modification time at the first look, in memory: a restart takes it again, and no time of another node is compared. A live holder is left to the discovery, the only installer of a config fetched, so a fetch in flight does not race it. It keeps up with a delete of many objects, 850 at once on dev2, where a first version panicked on its full queue at each restart: the records are indexed by path; the manager takes the config events of the node, not every instance monitor update, and a node monitor update only when the node goes away or comes back; a merge or a clear is written by a flush every 500ms, one sync of the directory for the batch; the messages are published in order by a publisher of their own, out of the lock the callers wait on. The subscription takes a huge queue. The ids of the records removed are remembered for an hour: a copy of one still in flight from a peer is stale, and adopted it was removed again, the peers doing the same with the copies of the node, round after round. A stale copy naming the node a pending holder, as one a peer killed before it wrote its last merge publishes again, is answered with the holder cleared as it was: the node alone can say it, and nothing else would. Its package API: Record, for the imon to delete an instance only once its record is written and published; ClearLocal for a holder clearing itself; DemoteLocal for the live holder of the node in the record of a delete that failed there, no other; Drop for a create --force, saying the holders it cleared once written and published, nothing to do without a pending holder; List for the listing; WasRemoved for the imon not to adopt a stale copy.
…er tombstone ls and GET /api/cluster/tombstone GET /api/cluster/tombstone returns the tombstones the nodes publish, the copies merged per orchestration, with their holders, the state, mode and clearing of each, and the nodes publishing them. Every node holds them all, so the node asked answers. Root sees them all, another user the ones of the namespaces it is guest of. om and ox cluster tombstone ls list them, a row per holder.
…clear a live holder of the node once its instance is gone A pending holder of a tombstone holds the instance of an object a delete or a purge did not reach yet: a config fetched from it brings the object back, as the node accepting a delete fetched the configs back from the peers whose instance had no manager yet. The discovery fetches no config of the path from a pending holder, and a pending holder fetches none: checked on a remote config update, before the foreign config branch, on a config for, at the fetch every path goes through, at the install, and in the choice of a peer to recover a config from. A pending holder does not serve the config either. A live holder of the node is cleared once its instance is gone there, decided by the discovery, the only installer of a config fetched: as a config manager ends with no config file, and as a tombstone naming the node lands, its fetch cancelled first. A holder cleared with a config published is a source again.
…ll holding the instance or not seen running the delete The imon remembers the orchestration id the monitor of each peer was seen carrying. Before it deletes its instance, the point of no return of a delete or a purge, it records the tombstone of the scope nodes, cluster nodes, never seen carrying the orchestration, or seen carrying it and still holding the instance, its config, status or monitor: away when their node monitor is absent or away, live otherwise, as a node whose config manager had not started as the delete ran. No holder, no record: the last node to delete, its peers gone, writes none. A tombstone that can not be saved fails the delete, nothing deleted without it. A peer seen running it is a holder while it holds the instance: killed before it deleted, as by a crash with its delete queued, it restarted with no monitor carrying the orchestration any more, kept the object, and the peers fetched it back from it, as 18 objects came back on dev2. On dev2, with the daemon of a node killed with 157 deletes queued, no config was fetched back; the records it heard after its rejoin it completed, the 5 it loaded at its restart it held, block and tell, until a delete there. A peer seen running it with no instance left counts as deleted only while its node is up, as the monitor last heard it and as the node data says now: one lost, its data dropped with it, is an away holder, cleared on its return if its instance is gone there, including while its drop is under way, the monitor not told yet. A delete or a purge waits for the live holders of its tombstone too, checked again on the tombstone events of the object.
…e a live holder A live holder is a node up whose instance did not run the delete or the purge, as one whose config manager had not started as it ran: the node that accepted it gone with its monitor, no peer monitor carries it any more. The instance monitor adopts the global expect of the tombstone, its stamp and orchestration id, when no orchestration runs and its node is out of the rejoin, and runs it as one adopted from a peer, writing no tombstone of its own for it, nor adopting a stale copy of a record its node removed. An away holder never adopts it: a node down, or restarted since, may hold the only copy of the data. On dev2, 850 secs created then deleted at once end gone on the three nodes, with no config fetched back, where 208 to 440 came back before.
…e of a node away up or provisioned, without --force An orchestration does not wait for a node away: a purge or a delete can not release its instance, maybe the only copy of the data, and an unprovision may release resources it still uses. The node accepting the request refuses it, naming the nodes down, in maintenance or unknown whose instance was last published up or provisioned, provisioned only for an unprovision. A node stopping deletes the statuses of its instances once it announced its shutdown: the last one is kept, and forgotten as the node publishes again. --force goes on without them, a purge or a delete then leaving a tombstone the node holds. Unforced, the leader of an unprovision or a purge waits for a node that went away after the request with its instance provisioned, as one restarting its daemon or entering a maintenance, until it is back or the maintenance grace period drops its data: it releases no shared resources under it. The force is an option of the global expect the peers adopt, a force query parameter of the object purge, delete and unprovision actions, and --force of om and ox purge, delete and unprovision: the one of om purge and unprovision, hidden before and allowing the dangerous operations of a --local action, is shown and keeps that meaning too.
… its node was away An away holder of a tombstone holds an instance the other nodes purged or deleted while its node was away, maybe the only copy of the data. Its instance monitor carries the issue in its new issues field, shown under the instance in the status board, saying to purge it there or to keep it by an edit or a touch of its config. The instance is flagged stopped once per tombstone, unless up, decided once its status is known, so the daemon does not start it on its own, and a start, a restart, a placement or a provision is refused with the issue, as the ha and boot decisions are; a purge, a delete, a stop or an unprovision is accepted. The instance is never stopped, nor frozen, for it. The issue goes with the holder cleared, purged there or kept. A flag not written is decided again at the next event.
…drop it with --force A tombstone with pending holders says nodes still hold the instance of the object of that name, purged or deleted elsewhere: a new object of the name would meet theirs. A create is refused naming them: a delete in progress on live holders, to retry once done, without suggesting --force, as the new instances would join that delete, or nodes away holding it, to purge there. The --force of a create, which replaces the config of an object that exists, also drops the tombstones of the path, its holders cleared as dropped, as they are once the config is written, one recorded by a delete since the check included, the record written and published before the create answers: a create refused or failed leaves the tombstone, and a crash between the write and the drop leaves it pending, the safe side, a create --force again dropping it. A dry run drops nothing.
…iled, the peers done A live holder adopts the delete or the purge of its tombstone once. Its delete failing, the peers having deleted theirs, the object lived on that node only, with no issue shown, until its daemon restarted. The live holder of the node now turns away, once per orchestration, when its instance lands in a failure state ending the delete or the purge of the tombstone, delete failed, or stop or purge failed, not one the delete goes on from, as a stop failure: the instance is flagged stopped, the issue shown, and the operator purges it there or keeps it, as for a node that was away. The failure state an action ends in turns it away as well as the one transitionTo sets. An instance never adopts the delete or the purge of a tombstone it carried itself, as one failed then aborted: adopting it again ran what the operator stopped (qatests 308, the purge aborted run again, the purge asked next refused as one in progress). Its live holder turns away instead.
…ed again under the path not to be deleted by it A tombstone names the path, and an object created again under it, with create --force or once the record is gone, had the same path: a live holder adopting the delete of a stale copy of the record, as one a peer killed before it wrote its last merge publishes again after the node restarted or forgot the removal, deleted the new object. The instance config carries the id of the object, its DEFAULT.id, and the record the id of the object it deletes. A live holder whose instance is of another object does not adopt the delete: its holder is cleared, as an away holder whose config is of another object is. An id unknown on either side, as of a record written by a former version, tells nothing, as before.
Three topics, in this order: the address events of the container network namespaces (netmon), the
apparmor_profilecontainer keyword, and the tombstones of the purges and deletes some nodes of an object did not run.netmon: address events of the other network namespaces
The address of an ip.netns or ip.cni resource lives in the network namespace of its container, which the netlink subscription of netmon did not hear: deleting or adding it there published no NetIPAddrDeleted or NetIPAddrAdded, and the instance status saw the change at its next scheduled refresh only.
nsidfield, omitted for the host namespace. The link is named by its label, ornsid<N>-index-<I>for an ipv6 address, whose message names none.Tests:
docker kill, the new namespace getting the same id; the host address events are unchanged, published once.container: the apparmor_profile keyword
On the Ubuntu 26.04 kernels (7.0,
kernel.apparmor_restrict_unprivileged_unconfined=1) the containers are labeleddocker-default//&unconfined, and the default profile of the engine denies the processes of a container signalling one another (moby/moby#54099): the haproxy master of the igw_haproxy gateway can not stop its old workers on a reload, and they go on serving the certificate loaded before it (qatests 440.04.6 failing on every u26 run).apparmor_profilenames the profile the processes of a docker or podman container run under: a profile loaded on the node, orunconfined. The engines get--security-opt apparmor=<profile>, the apparmor option ofrun_argsdropped, its other security options kept.Tombstones: purges and deletes some nodes did not run
A node that missed a purge or a delete, down, in maintenance, or up with its instance not seen running it, brought the object back to the peers having deleted theirs: 208 to 440 of 850 secs came back on dev2 when created then deleted at once.
Owner decisions: block and tell, never purge on a return, as the node away may hold the only copy of the data; a delete is treated as a purge.
The record
<var>/node/tombstone/<orchestration id>.json.What the holders do
issuesfield, shown under the instance in the status board, and a start, restart, placement or provision is refused with it.om <path> purgethere clears it, an edit or a touch of the config keeps the object, which propagates again.om|ox cluster tombstone ls,GET /api/cluster/tombstone: a row per holder.Breaking changes (CHANGELOG)
--force(new ondeleteand on ox; now visible onom purgeandom unprovision).--forcedrops the tombstone once every check of the create passed. In a rolling upgrade, a node on a former version never clears itself from a record.Lag fixes found on the way, standalone commits
Subscription.AddFiltersadds a list of filters in one bus operation; used by the per-object managers at start, the imon relation, affinity and volume filters, the api waits of a create--wait, and the event stream.GetNodesByPath, as do 12 api handlers and the imon hard affinity check, rather than copy every node's data.Tests
--wait(34s) then deleted at once: gone on the three nodes in 64s, no object back, no queue warning, no restart; before the fixes of this PR, 180s, and daemon panics on full queues earlier on.purge --force: issue shown, stopped flag, start refused; atouchkeeps the object everywhere, apurgethere removes the record;createrefused over the record,create --forcedrops it.--forcethe unprovision ends in 2.4s.To check in QA: RHEL 7 (3.10 with the option backported) and the macvlan, ipvlan and dedicated modes there.