Skip to content

Publish ports of the node to the address of ip.netns and ip.cni resources - #1230

Merged
cvaroqui merged 11 commits into
opensvc:mainfrom
TorisutanKholwes:dev
Nov 17, 2026
Merged

cvaroqui merged 11 commits into
opensvc:mainfrom
TorisutanKholwes:dev

Conversation

@TorisutanKholwes

@TorisutanKholwes TorisutanKholwes commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a publish keyword to the ip.netns and ip.cni resources: the node forwards a port of its own to the address the resource plumbed, with nftables rules of their own. No container engine option is involved, so it works the same whatever runs in the namespace (docker, podman, lxc, a containerd/runc container joining an ip.cni namespace, an app resource).

[ip#0]
type = netns
netns = container#0
network = default
publish = 25489:80/tcp      # node port 25489 -> port 80 of the address
# publish = 80/tcp          # a free port of the node, drawn at start
# publish = 10.0.0.5:8443:443/tcp   only on one address of the node

Behaviour

  • Syntax [<host addr>:][<host port>:]<port>[/<proto>], tcp by default, ipv6 host addresses in brackets.
  • Without host port, a free port of the node is drawn at start (the kernel picks it), kept by a start of the resource already up, released at stop.
  • Rules live in a dedicated osvc-publish table (the osvc table is rebuilt by every network setup), rendered in one transaction from a state file per resource under <run>/publish/<namespace>/<kind>/<name>/<rid>.json, emptied at boot with the rules. nft 0.8 (el7) syntax.
  • Covered: clients from outside (prerouting), the node itself to one of its non-loopback addresses (output), hairpin clients on the same network (masquerade).
  • A start is refused when the node does not forward the family (net.ipv4.ip_forward, net.ipv6.conf.all.forwarding; om never sets them), when another resource of the node publishes the port, or in a mode the node can not reach the address in (macvlan, ipvlan-*, dedicated).
  • On a lan network the publishes are ignored whatever the mode, the address being reached directly on the segment: the resource starts, an info line of its status says so.
  • The status warns when the rules are expected and not recorded or not loaded, or when the node no longer forwards.
  • Every published port is exposed by default (DNS SRV, as the matching expose declaration). An expose keyword set replaces them; expose = none exposes nothing (also accepted by ip.host).
  • Rights: a declaration naming a port or an address of the node (25489:80, 10.0.0.5::80) requires the root grant, the ports and addresses of the node being the node administrator's. A declaration naming neither (80/tcp), its node port drawn free by the kernel, is open to namespace administrators, as a NodePort is to the users of a Kubernetes namespace. Only the strict <port>[/<proto>] form passes; a reference that could hide a port still requires the grant.

expose

  • expose now takes the publish form [<frontend port>:]<port>[/<proto>]; the former <port>/<proto>[:<frontend port>] is still read and makes the same record.
  • Fix: a frontend port given by service name (8080/tcp:http) was refused; it now resolves.
  • The ip.cni documentation no longer claims expose configures port mappings.

Compatibility

Purely additive: a configuration without publish behaves as before. Nodes running an older agent ignore publish, so every node of a service is to be upgraded before setting it.

Commits

  1. [dns] expose read in the publish form, former form still read
  2. [network] publish core: parsing, port drawing, state, nftables rendering, forwarding check
  3. [ip] shared keyword, group, documentation and driver helpers
  4. [rbac] root grant for publish
  5. [ip] wiring in ip.netns, ip.cni (and expose = none in ip.host), lan handling, docs
  6. [ip] publish is StopToChange: a change or removal is refused while the resource is up
  7. [network] the status counts the forwarding rules of the resource, not the table only
  8. [network] the status warns of a published port a process of the node also holds
  9. [network] that check compares sockets of the family of the published address only
  10. [rbac] publish open to namespace administrators for a drawn node port (<port>[/<proto>])

Testing

  • Unit tests: parser (fixed, random, addresses, duplicates), rendering, nft --check of the ruleset (in a user/net namespace when unprivileged), start/stop/conflicts/rollback, port drawing and reuse, state layout, forwarding check, status info, lan handling, rbac guards, expose forms.
  • Each commit builds and passes the tests of the packages it touches.
  • Manual: forwarding verified on a dev node (docker engine, default bridge network): curl <node ip>:<port> reaches the nginx of the service.

…former form still read

An expose was written <port>/<proto>[:<frontend port>], the order of the
ports the reverse of the docker form the publish keyword takes. It is now
read as [<frontend port>:]<port>[/<proto>], the protocol tcp when not set.
The former form is still read, a colon after the slash saying it, and
makes the same record: 443/tcp:8443 and 8443:443/tcp name _8443._tcp
pointing to the port 443.

A frontend port given by its service name was looked up with the protocol
and the name as the network, and refused: it is looked up with the
protocol, in both forms.
… of their own

A port of the node can be forwarded to the address an ip resource plumbed,
by the node, so whatever runs in the namespace holding the address is
reached alike: no option of a container engine is involved.

A publish is written [<host addr>:][<host port>:]<port>[/<proto>]. One
naming no host port is given a free port of the node, the kernel picking
it, drawn again when another resource of the node publishes it, and kept by
a start of the resource already up.

The rules go in an osvc-publish table, the osvc one being deleted and
defined again by every network setup. The table is rendered in one
transaction from a state file per resource, under
<run>/publish/<namespace>/<kind>/<name>/<rid>.json, emptied at boot as the
rules are. A start is refused a port of the node another resource
publishes. The connections of the node itself to one of its addresses are
forwarded too, the loopback ones excepted, and the ones of the network of
the address masqueraded, for a client on the same bridge to get its
answers.

CheckForwarding reports the node not forwarding the packets of a family,
as an ErrForwardingOff: om does not set the sysctl.
The publish keyword is declared once, for the ip drivers plumbing an
address in a network namespace, in a group of its own, and validated as
the network package parses it.

StartPublish forwards the ports the keyword lists to the address, refused
when the node does not forward, with a rollback removing them. StatusPublish
warns when the rules are expected and not recorded or not loaded, or the
node no longer forwards. The messages say the fact only: the keyword
documents the fix.

PublishInfo returns the publish and expose values a status info reports:
the publishes with their drawn host ports, and every published port exposed
unless an expose is set, which replaces them, or set to none, which exposes
nothing.

ResolvedAllocation gives a test resource a network the node does not have.
A published port is a port of the node, forwarded on every address of the
node unless one is named: publishing 22 would take the ssh of the node. The
ports of the node are the node administrator's to hand out, so the publish
keyword of the ip group requires the root grant.
…i addresses

The ip.netns and ip.cni resources take the publish keyword. A start
forwards the ports once the address is plumbed, a stop removes them before
the address is released, and the status warns when they are not forwarded.
The status info reports the publishes with their drawn host ports, and the
exposes the cluster DNS makes the SRV records of.

ip.netns refuses a publish in the modes the node does not reach the address
in, macvlan, ipvlan and dedicated, except on a lan network, whose clients
reach the address directly: there the publishes are ignored, the resource
starting all the same, an info line of its status saying so.

The expose keyword of ip.netns, ip.cni and ip.host takes none, and its
documentation the publish form. The ip.cni one no longer says it configures
port mappings, which it never did.

The guard of the ip.cni keywords open to every user says the publish one
is refused on purpose, and the one of the ip.netns keywords deciding what
the node gives counts it.
@TorisutanKholwes
TorisutanKholwes marked this pull request as draft November 16, 2026 15:21
// is the administrator's decision.
func CheckForwarding(ip net.IP) error {
family := ipFamily(ip)
b, err := os.ReadFile(forwardingFiles[family])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential file inclusion attack via reading file - high severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.

Show fix

Remediation: Ignore this issue only after you've verified or sanitized the input going into this function.

Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info

}
entries := make([]publishEntry, 0, len(files))
for _, file := range files {
b, err := os.ReadFile(file)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential file inclusion attack via reading file - low severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.

Show fix

Remediation: Ignore this issue only after you've verified or sanitized the input going into this function.

Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info

Comment thread drivers/resip/manifest.go
Comment on lines +29 to +38
KeywordPublish = keywords.Keyword{
Attr: "Publish",
Converter: converters.List,
Example: "25489:80/tcp 192.168.1.10:8443:443/tcp",
Option: "publish",
Group: GroupPublish.ID,
Scopable: true,
Validate: network.ValidatePublish,
Text: keywords.NewText(fs, "text/kw/publish"),
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Removing publish leaves the old port forwarding active

A running ip.netns or ip.cni resource can change its non-empty publish value to empty because the new keyword is not marked StopToChange. The old state file and nft DNAT rules are then left in place while status stops checking them, so the node port continues forwarding traffic to the resource despite the current configuration no longer publishing it.

Show fix
Suggested change
KeywordPublish = keywords.Keyword{
Attr: "Publish",
Converter: converters.List,
Example: "25489:80/tcp 192.168.1.10:8443:443/tcp",
Option: "publish",
Group: GroupPublish.ID,
Scopable: true,
Validate: network.ValidatePublish,
Text: keywords.NewText(fs, "text/kw/publish"),
}
KeywordPublish = keywords.Keyword{
Attr: "Publish",
Converter: converters.List,
Example: "25489:80/tcp 192.168.1.10:8443:443/tcp",
Option: "publish",
Group: GroupPublish.ID,
Scopable: true,
StopToChange: true,
Validate: network.ValidatePublish,

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix in 3a72cec: the publish keyword is now StopToChange, so a change or a removal of its value is refused while the resource is up, as for the address keywords. Setting a value where there was none stays allowed, the status saying the ports are not forwarded until a start.

Comment on lines +119 to +131
c, err := net.ListenPacket("udp", hostport)
if err != nil {
return 0, err
}
defer c.Close()
return c.LocalAddr().(*net.UDPAddr).Port, nil
}
l, err := net.Listen("tcp", hostport)
if err != nil {
return 0, err
}
defer l.Close()
return l.Addr().(*net.TCPAddr).Port, nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Auto-assigned publish ports are released before the mapping is installed

For a publish without a host port, the allocator binds port 0 only long enough to read the selected port and closes the socket before nftables is applied. A host process can claim that port in the gap, or after the resource starts because no reservation remains; nft then redirects clients for that port to the namespace, causing a collision and denying the host process the port it successfully bound.

Show fix

Keep an appropriate reservation through the nftables installation and for the lifetime of the dynamically assigned mapping, or use a node-wide allocator/reservation mechanism that prevents other services from claiming the selected host port and releases it on stop.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix in 372649b: the status now reads the socket tables of the node and warns of a published port a socket of the node also holds (tcp listening, udp bound, on the published address or all of them; loopback-only sockets excepted, their connections not being forwarded). This covers the drawn ports and the fixed ones alike. Holding the port for the lifetime of the mapping would need a process living as long, which the actions of a resource are not, so the conflict is reported rather than prevented; the publish keyword documentation says so.

Comment on lines +68 to +78
recorded, loaded, err := network.PublishStatus(owner)
switch {
case err != nil:
slog.Warn("publish: %s", err)
return status.Warn
case !recorded:
slog.Warn("publish: %s not forwarded", strings.Join(publish, " "))
return status.Warn
case !loaded:
slog.Warn("publish: table %s not loaded", network.PublishTableName)
return status.Warn

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Status treats an empty publish table as a working forwarding

When an administrator removes this resource's DNAT rule but leaves another resource's osvc-publish table loaded, PublishStatus reports the owner as forwarded because it checks only the table family and name. The resource remains Up and continues advertising the published SRV port even though its forwarding rule is absent.

Show fix

Have publish status verify that the owner's expected rules/chains are present, for example by listing the dedicated table and matching the owner comment and rule parameters, and warn or reconcile when they are missing.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix in ff1bdcd: the status now reads the rules of the prerouting chain of the osvc-publish table and counts the ones whose comment names the resource. Fewer rules than publishes recorded is a forwarding missing, and the status warns, whatever the rules of the other resources keeping the table loaded.

The publish keyword was not StopToChange: a value removed while the
resource ran left its forwardings in place, the node port still forwarding
to the address, and the status no longer checked them; a value changed left
the former forwardings unchecked too. A change or a removal is now refused
while the resource is up, as for the address keywords. Setting a value
where there was none stays allowed, the status saying the ports are not
forwarded until a start.
…y, in its status

The status of a resource publishing ports checked the osvc-publish table was
loaded, which the rules of the other resources keep loaded: the rules of a
resource deleted by hand left it up, its SRV record still served. The rules
of the prerouting chain are now read through netlink, and the ones whose
comment names the resource counted: fewer than its publishes recorded is a
forwarding missing, and the status warns.
Nothing keeps a process of the node from binding a published port: the
drawn port is free when drawn, the socket the kernel picked it with closed
before the rules are loaded, and a fixed one may be bound at any time. The
forwarding then takes the connections that process waits for. Holding the
port for as long as it is published needs a process living as long, which
the actions of a resource are not.

The status of a resource now reads the socket tables of the node, and warns
of a port of the node it publishes a socket holds too: a tcp one listening,
an udp one bound, on the address published or on all of them. A socket of a
loopback address alone is no conflict, its connections not forwarded.

func readPublishEntry(owner PublishOwner) (publishEntry, error) {
var entry publishEntry
b, err := os.ReadFile(owner.file())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential file inclusion attack via reading file - high severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.

Show fix

Remediation: Ignore this issue only after you've verified or sanitized the input going into this function.

Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info

if s.port != p.HostPort || s.addr.IsLoopback() {
continue
}
if p.HostAddr == nil || s.addr.IsUnspecified() || s.addr.Equal(p.HostAddr) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low - IPv6-only sockets falsely mark IPv4 publishes as conflicted

When a published port is used by an IPv6-only wildcard socket, PublishesBound scans that socket for an IPv4 publish and treats the unspecified address as an overlap. The IPv4 forwarding remains independent of the IPv6 socket, but StatusPublish returns Warn, causing a healthy resource to be reported degraded whenever the same protocol/port is used across address families.

Show fix

Carry the publish/target address family into the socket-conflict check and only compare sockets from that family; for wildcard sockets, account for whether an IPv6 socket is dual-stack before treating it as an IPv4 conflict (or avoid claiming cross-family wildcard conflicts).

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix in 6f3e701: the socket check now compares sockets of the family of the published address only: an IPv6 publish ignores IPv4 sockets, an IPv4 publish ignores sockets of a specific IPv6 address. An IPv6 wildcard socket is still reported for an IPv4 publish: it is dual-stack by default on Linux and then receives the IPv4 connections the forwarding takes, and /proc/net/tcp6 does not tell a v6-only one apart.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 6f3e701.

This issue remains because core/network/publish_bound.go:79 makes every IPv6 wildcard socket receive IPv4 traffic, even when it is IPv6-only; PublishesBound then treats that socket as a conflict at line 56/59, so an IPv4 publish is still reported bound. The family filtering added since the review base is only partial because /proc/net/tcp6 does not identify the socket's IPV6_V6ONLY setting.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix in 4a2143e: the ipv6 sockets are now read from the sock_diag netlink interface of the kernel, which reports the IPV6_V6ONLY option of each socket, so a v6only ipv6 wildcard socket no longer counts as a conflict for an ipv4 publish. /proc/net/{tcp,udp}6 is only read as a fallback when the kernel does not answer.

…its address only

The check of the published ports a socket of the node also holds compared
every socket with every publish, whatever their families: an ipv6 publish
was reported held by a socket of 0.0.0.0, and an ipv4 one by a socket of a
specific ipv6 address, neither receiving the connections the forwarding
takes. A socket now counts for the family it receives the connections of:
an ipv4 one, or of an ipv4 mapped address, for an ipv4 publish, an ipv6 one
for an ipv6 publish. The ipv6 wildcard counts for both, its sockets being
dual stack by default on linux; the socket tables do not tell a v6only one
apart.
… of the node drawn

The publish keyword required the root grant whatever its value. A
declaration naming neither a port nor an address of the node, as 80/tcp,
has its port of the node drawn free by the kernel, never one a process of
the node or another publish holds: it takes nothing the node administrator
hands out, as a NodePort is open to the users of a namespace. It is now
open to the users with no grant.

Only the form <port>[/<proto>] passes, every declaration of the value in
that form: a port or an address of the node named, or a reference whose
value could name one, still requires the root grant.
… the publish bound check

The socket tables of /proc do not say the IPV6_V6ONLY option of a socket:
every ipv6 wildcard socket was taken for a dual stack one, and an ipv4
publish of a port a v6only socket holds was reported bound, though the
forwarding of ipv4 takes none of its connections.

The ipv6 sockets are now read from the sock_diag netlink interface of the
kernel, which sends the v6only option of every ipv6 socket: a v6only
wildcard counts for the ipv6 publishes only. The ipv4 sockets are still
read from /proc, and the ipv6 ones too when the kernel does not answer,
then taken for dual stack ones as before.
// readProcNet reads the sockets of a protocol holding a port from a socket
// table of /proc.
func readProcNet(file, proto string) ([]boundSocket, error) {
f, err := os.Open(file)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential file inclusion attack via reading file - high severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.

Show fix

Remediation: Ignore this issue only after you've verified or sanitized the input going into this function.

Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info

@cvaroqui
cvaroqui marked this pull request as ready for review November 17, 2026 14:24
@cvaroqui
cvaroqui merged commit 1e402f9 into opensvc:main Nov 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants