Repository navigation
Publish ports of the node to the address of ip.netns and ip.cni resources - #1230
Conversation
…former form still read An expose was written <port>/<proto>[:<frontend port>], the order of the ports the reverse of the docker form the publish keyword takes. It is now read as [<frontend port>:]<port>[/<proto>], the protocol tcp when not set. The former form is still read, a colon after the slash saying it, and makes the same record: 443/tcp:8443 and 8443:443/tcp name _8443._tcp pointing to the port 443. A frontend port given by its service name was looked up with the protocol and the name as the network, and refused: it is looked up with the protocol, in both forms.
… of their own A port of the node can be forwarded to the address an ip resource plumbed, by the node, so whatever runs in the namespace holding the address is reached alike: no option of a container engine is involved. A publish is written [<host addr>:][<host port>:]<port>[/<proto>]. One naming no host port is given a free port of the node, the kernel picking it, drawn again when another resource of the node publishes it, and kept by a start of the resource already up. The rules go in an osvc-publish table, the osvc one being deleted and defined again by every network setup. The table is rendered in one transaction from a state file per resource, under <run>/publish/<namespace>/<kind>/<name>/<rid>.json, emptied at boot as the rules are. A start is refused a port of the node another resource publishes. The connections of the node itself to one of its addresses are forwarded too, the loopback ones excepted, and the ones of the network of the address masqueraded, for a client on the same bridge to get its answers. CheckForwarding reports the node not forwarding the packets of a family, as an ErrForwardingOff: om does not set the sysctl.
The publish keyword is declared once, for the ip drivers plumbing an address in a network namespace, in a group of its own, and validated as the network package parses it. StartPublish forwards the ports the keyword lists to the address, refused when the node does not forward, with a rollback removing them. StatusPublish warns when the rules are expected and not recorded or not loaded, or the node no longer forwards. The messages say the fact only: the keyword documents the fix. PublishInfo returns the publish and expose values a status info reports: the publishes with their drawn host ports, and every published port exposed unless an expose is set, which replaces them, or set to none, which exposes nothing. ResolvedAllocation gives a test resource a network the node does not have.
A published port is a port of the node, forwarded on every address of the node unless one is named: publishing 22 would take the ssh of the node. The ports of the node are the node administrator's to hand out, so the publish keyword of the ip group requires the root grant.
…i addresses The ip.netns and ip.cni resources take the publish keyword. A start forwards the ports once the address is plumbed, a stop removes them before the address is released, and the status warns when they are not forwarded. The status info reports the publishes with their drawn host ports, and the exposes the cluster DNS makes the SRV records of. ip.netns refuses a publish in the modes the node does not reach the address in, macvlan, ipvlan and dedicated, except on a lan network, whose clients reach the address directly: there the publishes are ignored, the resource starting all the same, an info line of its status saying so. The expose keyword of ip.netns, ip.cni and ip.host takes none, and its documentation the publish form. The ip.cni one no longer says it configures port mappings, which it never did. The guard of the ip.cni keywords open to every user says the publish one is refused on purpose, and the one of the ip.netns keywords deciding what the node gives counts it.
| // is the administrator's decision. | ||
| func CheckForwarding(ip net.IP) error { | ||
| family := ipFamily(ip) | ||
| b, err := os.ReadFile(forwardingFiles[family]) |
There was a problem hiding this comment.
Potential file inclusion attack via reading file - high severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.
Show fix
Remediation: Ignore this issue only after you've verified or sanitized the input going into this function.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info
| } | ||
| entries := make([]publishEntry, 0, len(files)) | ||
| for _, file := range files { | ||
| b, err := os.ReadFile(file) |
There was a problem hiding this comment.
Potential file inclusion attack via reading file - low severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.
Show fix
Remediation: Ignore this issue only after you've verified or sanitized the input going into this function.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info
| KeywordPublish = keywords.Keyword{ | ||
| Attr: "Publish", | ||
| Converter: converters.List, | ||
| Example: "25489:80/tcp 192.168.1.10:8443:443/tcp", | ||
| Option: "publish", | ||
| Group: GroupPublish.ID, | ||
| Scopable: true, | ||
| Validate: network.ValidatePublish, | ||
| Text: keywords.NewText(fs, "text/kw/publish"), | ||
| } |
There was a problem hiding this comment.
🟡 Medium - Removing publish leaves the old port forwarding active
A running ip.netns or ip.cni resource can change its non-empty publish value to empty because the new keyword is not marked StopToChange. The old state file and nft DNAT rules are then left in place while status stops checking them, so the node port continues forwarding traffic to the resource despite the current configuration no longer publishing it.
Show fix
| KeywordPublish = keywords.Keyword{ | |
| Attr: "Publish", | |
| Converter: converters.List, | |
| Example: "25489:80/tcp 192.168.1.10:8443:443/tcp", | |
| Option: "publish", | |
| Group: GroupPublish.ID, | |
| Scopable: true, | |
| Validate: network.ValidatePublish, | |
| Text: keywords.NewText(fs, "text/kw/publish"), | |
| } | |
| KeywordPublish = keywords.Keyword{ | |
| Attr: "Publish", | |
| Converter: converters.List, | |
| Example: "25489:80/tcp 192.168.1.10:8443:443/tcp", | |
| Option: "publish", | |
| Group: GroupPublish.ID, | |
| Scopable: true, | |
| StopToChange: true, | |
| Validate: network.ValidatePublish, |
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
fix in 3a72cec: the publish keyword is now StopToChange, so a change or a removal of its value is refused while the resource is up, as for the address keywords. Setting a value where there was none stays allowed, the status saying the ports are not forwarded until a start.
| c, err := net.ListenPacket("udp", hostport) | ||
| if err != nil { | ||
| return 0, err | ||
| } | ||
| defer c.Close() | ||
| return c.LocalAddr().(*net.UDPAddr).Port, nil | ||
| } | ||
| l, err := net.Listen("tcp", hostport) | ||
| if err != nil { | ||
| return 0, err | ||
| } | ||
| defer l.Close() | ||
| return l.Addr().(*net.TCPAddr).Port, nil |
There was a problem hiding this comment.
🟡 Medium - Auto-assigned publish ports are released before the mapping is installed
For a publish without a host port, the allocator binds port 0 only long enough to read the selected port and closes the socket before nftables is applied. A host process can claim that port in the gap, or after the resource starts because no reservation remains; nft then redirects clients for that port to the namespace, causing a collision and denying the host process the port it successfully bound.
Show fix
Keep an appropriate reservation through the nftables installation and for the lifetime of the dynamically assigned mapping, or use a node-wide allocator/reservation mechanism that prevents other services from claiming the selected host port and releases it on stop.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
fix in 372649b: the status now reads the socket tables of the node and warns of a published port a socket of the node also holds (tcp listening, udp bound, on the published address or all of them; loopback-only sockets excepted, their connections not being forwarded). This covers the drawn ports and the fixed ones alike. Holding the port for the lifetime of the mapping would need a process living as long, which the actions of a resource are not, so the conflict is reported rather than prevented; the publish keyword documentation says so.
| recorded, loaded, err := network.PublishStatus(owner) | ||
| switch { | ||
| case err != nil: | ||
| slog.Warn("publish: %s", err) | ||
| return status.Warn | ||
| case !recorded: | ||
| slog.Warn("publish: %s not forwarded", strings.Join(publish, " ")) | ||
| return status.Warn | ||
| case !loaded: | ||
| slog.Warn("publish: table %s not loaded", network.PublishTableName) | ||
| return status.Warn |
There was a problem hiding this comment.
🟡 Medium - Status treats an empty publish table as a working forwarding
When an administrator removes this resource's DNAT rule but leaves another resource's osvc-publish table loaded, PublishStatus reports the owner as forwarded because it checks only the table family and name. The resource remains Up and continues advertising the published SRV port even though its forwarding rule is absent.
Show fix
Have publish status verify that the owner's expected rules/chains are present, for example by listing the dedicated table and matching the owner comment and rule parameters, and warn or reconcile when they are missing.
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
fix in ff1bdcd: the status now reads the rules of the prerouting chain of the osvc-publish table and counts the ones whose comment names the resource. Fewer rules than publishes recorded is a forwarding missing, and the status warns, whatever the rules of the other resources keeping the table loaded.
The publish keyword was not StopToChange: a value removed while the resource ran left its forwardings in place, the node port still forwarding to the address, and the status no longer checked them; a value changed left the former forwardings unchecked too. A change or a removal is now refused while the resource is up, as for the address keywords. Setting a value where there was none stays allowed, the status saying the ports are not forwarded until a start.
…y, in its status The status of a resource publishing ports checked the osvc-publish table was loaded, which the rules of the other resources keep loaded: the rules of a resource deleted by hand left it up, its SRV record still served. The rules of the prerouting chain are now read through netlink, and the ones whose comment names the resource counted: fewer than its publishes recorded is a forwarding missing, and the status warns.
Nothing keeps a process of the node from binding a published port: the drawn port is free when drawn, the socket the kernel picked it with closed before the rules are loaded, and a fixed one may be bound at any time. The forwarding then takes the connections that process waits for. Holding the port for as long as it is published needs a process living as long, which the actions of a resource are not. The status of a resource now reads the socket tables of the node, and warns of a port of the node it publishes a socket holds too: a tcp one listening, an udp one bound, on the address published or on all of them. A socket of a loopback address alone is no conflict, its connections not forwarded.
|
|
||
| func readPublishEntry(owner PublishOwner) (publishEntry, error) { | ||
| var entry publishEntry | ||
| b, err := os.ReadFile(owner.file()) |
There was a problem hiding this comment.
Potential file inclusion attack via reading file - high severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.
Show fix
Remediation: Ignore this issue only after you've verified or sanitized the input going into this function.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info
| if s.port != p.HostPort || s.addr.IsLoopback() { | ||
| continue | ||
| } | ||
| if p.HostAddr == nil || s.addr.IsUnspecified() || s.addr.Equal(p.HostAddr) { |
There was a problem hiding this comment.
🔵 Low - IPv6-only sockets falsely mark IPv4 publishes as conflicted
When a published port is used by an IPv6-only wildcard socket, PublishesBound scans that socket for an IPv4 publish and treats the unspecified address as an overlap. The IPv4 forwarding remains independent of the IPv6 socket, but StatusPublish returns Warn, causing a healthy resource to be reported degraded whenever the same protocol/port is used across address families.
Show fix
Carry the publish/target address family into the socket-conflict check and only compare sockets from that family; for wildcard sockets, account for whether an IPv6 socket is dual-stack before treating it as an IPv4 conflict (or avoid claiming cross-family wildcard conflicts).
More info - Reply on this comment to give feedback or ignore the issue.
There was a problem hiding this comment.
fix in 6f3e701: the socket check now compares sockets of the family of the published address only: an IPv6 publish ignores IPv4 sockets, an IPv4 publish ignores sockets of a specific IPv6 address. An IPv6 wildcard socket is still reported for an IPv4 publish: it is dual-stack by default on Linux and then receives the IPv4 connections the forwarding takes, and /proc/net/tcp6 does not tell a v6-only one apart.
There was a problem hiding this comment.
Partially fixed in commit 6f3e701.
This issue remains because core/network/publish_bound.go:79 makes every IPv6 wildcard socket receive IPv4 traffic, even when it is IPv6-only; PublishesBound then treats that socket as a conflict at line 56/59, so an IPv4 publish is still reported bound. The family filtering added since the review base is only partial because /proc/net/tcp6 does not identify the socket's IPV6_V6ONLY setting.
There was a problem hiding this comment.
fix in 4a2143e: the ipv6 sockets are now read from the sock_diag netlink interface of the kernel, which reports the IPV6_V6ONLY option of each socket, so a v6only ipv6 wildcard socket no longer counts as a conflict for an ipv4 publish. /proc/net/{tcp,udp}6 is only read as a fallback when the kernel does not answer.
…its address only The check of the published ports a socket of the node also holds compared every socket with every publish, whatever their families: an ipv6 publish was reported held by a socket of 0.0.0.0, and an ipv4 one by a socket of a specific ipv6 address, neither receiving the connections the forwarding takes. A socket now counts for the family it receives the connections of: an ipv4 one, or of an ipv4 mapped address, for an ipv4 publish, an ipv6 one for an ipv6 publish. The ipv6 wildcard counts for both, its sockets being dual stack by default on linux; the socket tables do not tell a v6only one apart.
… of the node drawn The publish keyword required the root grant whatever its value. A declaration naming neither a port nor an address of the node, as 80/tcp, has its port of the node drawn free by the kernel, never one a process of the node or another publish holds: it takes nothing the node administrator hands out, as a NodePort is open to the users of a namespace. It is now open to the users with no grant. Only the form <port>[/<proto>] passes, every declaration of the value in that form: a port or an address of the node named, or a reference whose value could name one, still requires the root grant.
… the publish bound check The socket tables of /proc do not say the IPV6_V6ONLY option of a socket: every ipv6 wildcard socket was taken for a dual stack one, and an ipv4 publish of a port a v6only socket holds was reported bound, though the forwarding of ipv4 takes none of its connections. The ipv6 sockets are now read from the sock_diag netlink interface of the kernel, which sends the v6only option of every ipv6 socket: a v6only wildcard counts for the ipv6 publishes only. The ipv4 sockets are still read from /proc, and the ipv6 ones too when the kernel does not answer, then taken for dual stack ones as before.
| // readProcNet reads the sockets of a protocol holding a port from a socket | ||
| // table of /proc. | ||
| func readProcNet(file, proto string) ([]boundSocket, error) { | ||
| f, err := os.Open(file) |
There was a problem hiding this comment.
Potential file inclusion attack via reading file - high severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.
Show fix
Remediation: Ignore this issue only after you've verified or sanitized the input going into this function.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info
Summary
Adds a
publishkeyword to theip.netnsandip.cniresources: the node forwards a port of its own to the address the resource plumbed, with nftables rules of their own. No container engine option is involved, so it works the same whatever runs in the namespace (docker, podman, lxc, a containerd/runc container joining anip.cninamespace, anappresource).Behaviour
[<host addr>:][<host port>:]<port>[/<proto>], tcp by default, ipv6 host addresses in brackets.osvc-publishtable (theosvctable is rebuilt by every network setup), rendered in one transaction from a state file per resource under<run>/publish/<namespace>/<kind>/<name>/<rid>.json, emptied at boot with the rules. nft 0.8 (el7) syntax.net.ipv4.ip_forward,net.ipv6.conf.all.forwarding; om never sets them), when another resource of the node publishes the port, or in a mode the node can not reach the address in (macvlan,ipvlan-*,dedicated).lannetwork the publishes are ignored whatever the mode, the address being reached directly on the segment: the resource starts, an info line of its status says so.exposedeclaration). Anexposekeyword set replaces them;expose = noneexposes nothing (also accepted byip.host).25489:80,10.0.0.5::80) requires the root grant, the ports and addresses of the node being the node administrator's. A declaration naming neither (80/tcp), its node port drawn free by the kernel, is open to namespace administrators, as a NodePort is to the users of a Kubernetes namespace. Only the strict<port>[/<proto>]form passes; a reference that could hide a port still requires the grant.expose
exposenow takes the publish form[<frontend port>:]<port>[/<proto>]; the former<port>/<proto>[:<frontend port>]is still read and makes the same record.8080/tcp:http) was refused; it now resolves.ip.cnidocumentation no longer claimsexposeconfigures port mappings.Compatibility
Purely additive: a configuration without
publishbehaves as before. Nodes running an older agent ignorepublish, so every node of a service is to be upgraded before setting it.Commits
[dns]expose read in the publish form, former form still read[network]publish core: parsing, port drawing, state, nftables rendering, forwarding check[ip]shared keyword, group, documentation and driver helpers[rbac]root grant forpublish[ip]wiring in ip.netns, ip.cni (andexpose = nonein ip.host), lan handling, docs[ip]publishis StopToChange: a change or removal is refused while the resource is up[network]the status counts the forwarding rules of the resource, not the table only[network]the status warns of a published port a process of the node also holds[network]that check compares sockets of the family of the published address only[rbac]publishopen to namespace administrators for a drawn node port (<port>[/<proto>])Testing
nft --checkof the ruleset (in a user/net namespace when unprivileged), start/stop/conflicts/rollback, port drawing and reuse, state layout, forwarding check, status info, lan handling, rbac guards, expose forms.defaultbridge network):curl <node ip>:<port>reaches the nginx of the service.