From cac18515c423c1d1bd18cc90715c66f42d694631 Mon Sep 17 00:00:00 2001 From: Jim Lin Date: Sat, 17 Oct 2026 22:52:00 +0800 Subject: [PATCH 1/2] feat(sbom): extend the Go re-check to +dirty builds, other module paths and the project's own advisory A scan of cubecos's rebuilt binaries showed three cases the Go pass missed. A binary built from a modified tree stamps its pseudo-version with +dirty (beats' builds: v7.0.0-20260930214605-b80bf8464fba+dirty), which the pass did not read as a pseudo-version. It does now. An advisory can give semantic ranges under one module path of a repository and only a fixed pseudo-version under another. GitHub's advisory for a Packetbeat flaw fixes github.com/elastic/beats in 8.19.9, 9.1.9 and 9.2.3, but github.com/elastic/beats/v7 only at a commit on the 9.2 branch. The pass now reads the ranges of the repository's other module paths too. When the linked commit is itself a release (v9.5.5 here), a fixed pseudo-version it does not contain no longer keeps the match, because the pseudo-version only names that release. The Go vulnerability database can list a project's older module path with no affected range, meaning every version and no fix, where the project's own advisory names only the newer path. Examples are containerd v1, for two CRI checkpoint flaws in a feature only v2 has, and chi v1 to v4, for a RedirectSlashes flaw introduced in v5.2.2. grype reports such a match with no constraint. It now moves to ignoredMatches when the GitHub advisory the repository itself published for the flaw names only the repository's other module paths. Verified with the script as committed. test_recheckvulns.sh passes 126 checks, 6 of them new. They cover a +dirty build, a release judged by the other path's ranges, the same advisory on a commit that is no release, an advisory naming only the other path, and the project's advisory deferred to. They also cover three cases that must stay: a database record with a range, an advisory without the project's own page, and an advisory naming the module itself. test_projsbom.sh passes its 12. In the jim build jail, on grype 0.119.0's reports, both beats matches on the rebuilt filebeat and auditbeat are ignored as release v9.5.5. On the 3.2.0 rootfs report, the pseudo-version counts are unchanged, and the new check ignores exactly seven matches: the two containerd v1 Criticals and a Medium, and four chi v4 matches. Refs bigstack-oss/cubecos#578 Signed-off-by: Jim Lin Co-authored-by: Eandalf Co-authored-by: Claude Opus 5.5 (1M context) --- scripts/recheckvulns.py | 115 ++++++++++++++++++++++++----- scripts/tests/test_recheckvulns.sh | 42 ++++++++++- 2 files changed, 135 insertions(+), 22 deletions(-) diff --git a/scripts/recheckvulns.py b/scripts/recheckvulns.py index 9d74ade..e695d58 100755 --- a/scripts/recheckvulns.py +++ b/scripts/recheckvulns.py @@ -75,9 +75,15 @@ from is outside every affected range, at or above every fixed release, and was released after all of them, the fixed releases being tags of the repository. That last condition keeps a renumbered project's old commits: prometheus's 2021 commit descends from v2.26.0, which sorts above its 3.x -module versions (v0.311.3) but came long before them. A fixed pseudo-version whose commit it does -not contain keeps it. The clones hold commits only and are brought up to date on later scans, the -report's descriptor lists them, and GOMODULE_GIT in the environment points at a mirror. +module versions (v0.311.3) but came long before them. The ranges GitHub gives under the +repository's other module paths count too (github.com/elastic/beats for github.com/elastic/beats/v7). +A fixed pseudo-version whose commit it does not contain keeps the match, unless the commit is itself +a release, which the pseudo-version then only names. The clones hold commits only and are brought +up to date on later scans, the report's descriptor lists them, and GOMODULE_GIT in the environment +points at a mirror. With the same option, a match from the Go vulnerability database that names a +module with no affected range at all, every version and no fix, moves to "ignoredMatches" when the +project's own advisory for the flaw (a GitHub advisory the repository published) names only the +repository's other module paths: containerd v1 for a flaw in a CRI feature v2 added. """ import argparse @@ -718,8 +724,9 @@ def recheck_ossa(rep, path): # A Go pseudo-version names a commit: v0.0.0-20250227174106-7829dbe62d7f, -# v1.5.1-0.20260427112133-525d1bab07e0, v1.2.3-pre.0.20200101000000-0123456789ab. -PSEUDO = re.compile(r'^v?\d+\.\d+\.\d+-(?:[0-9A-Za-z.]+\.)?(\d{14})-([0-9a-f]{12})$') +# v1.5.1-0.20260427112133-525d1bab07e0, v1.2.3-pre.0.20200101000000-0123456789ab. A build from a +# modified tree carries build metadata after it: v7.0.0-20260930214605-b80bf8464fba+dirty. +PSEUDO = re.compile(r'^v?\d+\.\d+\.\d+-(?:[0-9A-Za-z.]+\.)?(\d{14})-([0-9a-f]{12})(?:\+[0-9A-Za-z.-]+)?$') RELEASE_TAG = re.compile(r'^v(\d+)\.(\d+)\.(\d+)$') GO_COND = re.compile(r'^\s*(<=|>=|<|>|=)\s*v?(\S+?)\s*$') @@ -746,9 +753,12 @@ def go_in_range(ver, conds): _ADV = {} -def advisory(vid, module, repo): - """The affected ranges [(conditions, fix or None)] and fix commits grype's own database holds - for vid on module, from GitHub's advisory; None if it has none for that module.""" +def same_repo(name, repo): + """True if name is repo's module path, or the same path with a major-version suffix.""" + return re.fullmatch(rf'github\.com/{re.escape(repo)}(/v\d+)?', name) is not None + + +def db_search(vid): if vid not in _ADV: out = subprocess.run( [GRYPE, 'db', 'search', '--vuln', vid, '-o', 'json'], @@ -757,13 +767,23 @@ def advisory(vid, module, repo): if out.returncode: sys.exit(f'recheckvulns: cannot read {vid} from grype\'s database') _ADV[vid] = json.loads(out.stdout or '[]') - ranges, commits = [], set() - for rec in _ADV[vid]: + return _ADV[vid] + + +def advisory(vid, module, repo): + """The affected ranges [(conditions, fix or None)] and fix commits grype's own database holds + for vid, from GitHub's advisory, on module and on the repository's other module paths (the + same project, released under another major version: github.com/elastic/beats for + github.com/elastic/beats/v7); None if it names the module under neither.""" + ranges, commits, named = [], set(), False + for rec in db_search(vid): if rec['vulnerability'].get('provider') != 'github': continue for p in rec.get('packages') or []: - if p['package'].get('ecosystem') != 'go-module' or p['package'].get('name') != module: + pkg = p.get('package') or {} + if pkg.get('ecosystem') != 'go-module' or not same_repo(pkg.get('name', ''), repo): continue + named |= pkg['name'] == module for r in p['detail'].get('ranges') or []: conds = [GO_COND.match(c) for c in r['version']['constraint'].split(',')] if conds and all(conds): @@ -772,7 +792,7 @@ def advisory(vid, module, repo): m = re.match(rf'https://github\.com/{re.escape(repo)}/commit/([0-9a-f]{{7,40}})\b', ref['url']) if m: commits.add(m.group(1)) - return (ranges, commits) if ranges else None + return (ranges, commits) if ranges and named else None def fetch_repo(url, path): @@ -796,6 +816,29 @@ def ancestor(path, a, b): stderr=subprocess.DEVNULL).returncode == 0 +def project_advisory(m, repo): + """(GHSA, module paths) when the Go vulnerability database's record behind m names its module + with no affected range at all, while the project's own GitHub advisory for the same flaw names + only the repository's other module paths; else None.""" + vul, art = m['vulnerability'], m['artifact'] + rec = next((r for r in db_search(vul['id']) if r['vulnerability'].get('provider') == 'govulndb'), None) + pkgs = [p for p in (rec or {}).get('packages') or [] if (p.get('package') or {}).get('name') == art['name']] + if not pkgs or any(p['detail'].get('ranges') for p in pkgs): + return None + for g in [x['id'] for x in m.get('relatedVulnerabilities') or [] if x['id'].startswith('GHSA-')]: + for r in db_search(g): + v = r['vulnerability'] + if v.get('provider') != 'github' or not any( + u['url'].startswith(f'https://github.com/{repo}/security/advisories/') for u in v.get('refs') or [] + ): + continue + names = {p['package']['name'] for p in r.get('packages') or [] + if (p.get('package') or {}).get('ecosystem') == 'go-module'} + if names and art['name'] not in names and all(same_repo(n, repo) for n in names): + return g, sorted(names) + return None + + def recheck_gopseudo(rep, path): """Re-judge matches on Go modules at a pseudo-version by the module's own history in place; a summary, or None if the report has no such match.""" @@ -806,7 +849,18 @@ def recheck_gopseudo(rep, path): parts = art['name'].split('/') if pm and len(parts) >= 3 and parts[0] == 'github.com': todo.append((m, f'{parts[1]}/{parts[2]}', pm.group(2))) - if not todo: + # The Go vulnerability database can list a project's older module path with no range at all + # (every version, no fix) where the project's own advisory names only its newer one: containerd + # v1 for a flaw in a CRI feature v2 added. grype reports such a match without a constraint. + blanket = [ + (m, '/'.join(m['artifact']['name'].split('/')[1:3])) for m in rep['matches'] + if m['artifact']['type'] == 'go-module' and m['artifact']['name'].startswith('github.com/') + and len(m['artifact']['name'].split('/')) >= 3 and not PSEUDO.match(m['artifact']['version']) + and m['vulnerability'].get('namespace', '').startswith('govulndb') + and any((d.get('found') or {}).get('versionConstraint', '').startswith('none') + for d in m.get('matchDetails') or []) + ] + if not todo and not blanket: return None repos = {} @@ -823,8 +877,18 @@ def recheck_gopseudo(rep, path): tags[name] = int(date or 0) repos[repo] = (clone, tags) - ignored = {'fix commit': [], 'release': []} + ignored = {'fix commit': [], 'release': [], 'project advisory': []} kept = 0 + deferred = 0 + for m, repo in blanket: + said = project_advisory(m, repo) + if not said: + continue + deferred += 1 + g, names = said + ignore(m, f'Go vulnerability database: {m["vulnerability"]["id"]} names {m["artifact"]["name"]} ' + f'with no affected range; the project\'s own advisory {g} names only {", ".join(names)}') + ignored['project advisory'].append(m) for m, repo, short in todo: vul, art = m['vulnerability'], m['artifact'] clone, tags = repos[repo] @@ -843,14 +907,22 @@ def recheck_gopseudo(rep, path): # The module path has no major-version suffix, so the project's releases cannot be # named: every version a Go build can link is a pseudo-version, which sorts below # every fixed release. Judge by the newest release the commit descends from instead, - # when the fixes are releases of this repository and that one came after them all. + # when the fixes are releases of this repository and that one came after them all. A + # fixed pseudo-version the commit does not contain keeps the match, unless the commit + # is itself a release: the pseudo-version then only names that release, and a fix on + # another branch says nothing about it (beats/v7 at v9.5.5, its fix commit on 9.2). fixes = [f for _, f in ranges if f and not PSEUDO.match(f)] merged = [t for t in (git_out(clone, 'tag', '--merged', sha, '--list', 'v*') or '').split() if RELEASE_TAG.match(t)] - if pseudo_fixes or not fixes or not merged or any('v' + f not in tags for f in fixes): + exact = [t for t in merged + if (git_out(clone, 'rev-parse', '-q', '--verify', t + '^{commit}') or '').strip() == sha] + if ( + (pseudo_fixes and not exact) or not fixes or not merged + or any('v' + f not in tags for f in fixes) + ): kept += 1 continue - base = max(merged, key=semkey) + base = max(exact or merged, key=semkey) if ( any(go_in_range(base, conds) for conds, _ in ranges) or semkey(base) < max(semkey(f) for f in fixes) @@ -859,8 +931,8 @@ def recheck_gopseudo(rep, path): kept += 1 continue why = ( - f'descends from {base}, released after every fixed release ' - f'({", ".join(sorted(set(fixes), key=semkey))}) and outside every affected range' + f'{"is release" if exact else "descends from"} {base}, released after every fixed ' + f'release ({", ".join(sorted(set(fixes), key=semkey))}) and outside every affected range' ) by = 'release' ignore(m, f'go pseudo-version: {art["name"]} {art["version"]} is commit {sha[:12]}, which {why}') @@ -870,6 +942,7 @@ def recheck_gopseudo(rep, path): rep['matches'] = [m for m in rep['matches'] if id(m) not in drop] rep['ignoredMatches'] = ( (rep.get('ignoredMatches') or []) + ignored['fix commit'] + ignored['release'] + + ignored['project advisory'] ) rep.setdefault('descriptor', {})['goPseudoRecheck'] = { 'source': GOMODULE_GIT, @@ -877,10 +950,12 @@ def recheck_gopseudo(rep, path): 'judged': len(todo), 'ignored': {by: len(ms) for by, ms in ignored.items()}, 'kept': kept, + 'blanketJudged': len(blanket), } return ( f'Go pseudo-version matches: {len(todo)} judged; {len(ignored["fix commit"])} ignored by a ' - f'fix commit, {len(ignored["release"])} by a release; {kept} kept' + f'fix commit, {len(ignored["release"])} by a release; {kept} kept. Go vulnerability database ' + f'matches with no range: {len(blanket)} judged; {deferred} ignored by the project\'s own advisory' ) diff --git a/scripts/tests/test_recheckvulns.sh b/scripts/tests/test_recheckvulns.sh index 2f5f0c7..5f0e9c0 100755 --- a/scripts/tests/test_recheckvulns.sh +++ b/scripts/tests/test_recheckvulns.sh @@ -339,10 +339,48 @@ chk " kept" "$(q "$gkept")" "GHSA-k001@${LINKED:0:12} GHSA-k002@${LINKED:0:12} chk " a fix commit the advisory names is an ancestor" "$(gwhy GHSA-f001)" "go pseudo-version: github.com/acme/widget $PL is commit ${LINKED:0:12}, which contains the fix commit ${FIX:0:12}" chk " so is a fixed pseudo-version's commit" "$(gwhy GHSA-f002)" "go pseudo-version: github.com/acme/widget $PL is commit ${LINKED:0:12}, which contains the fix commit ${FIX:0:12}" chk " descends from a release after every fix" "$(gwhy GHSA-r001)" "go pseudo-version: github.com/acme/widget $PL is commit ${LINKED:0:12}, which descends from v2.6.3, released after every fixed release (1.6.27, 2.2.4) and outside every affected range" -chk " descriptor" "$(q "r['descriptor']['goPseudoRecheck']['repositories'], r['descriptor']['goPseudoRecheck']['judged'], r['descriptor']['goPseudoRecheck']['ignored'], r['descriptor']['goPseudoRecheck']['kept']")" "['acme/widget'] 10 {'fix commit': 2, 'release': 1} 7" -chk " summary" "$OUT" "Go pseudo-version matches: 10 judged; 2 ignored by a fix commit, 1 by a release; 7 kept" +chk " descriptor" "$(q "r['descriptor']['goPseudoRecheck']['repositories'], r['descriptor']['goPseudoRecheck']['judged'], r['descriptor']['goPseudoRecheck']['ignored'], r['descriptor']['goPseudoRecheck']['kept']")" "['acme/widget'] 10 {'fix commit': 2, 'release': 1, 'project advisory': 0} 7" +chk " summary" "$OUT" "Go pseudo-version matches: 10 judged; 2 ignored by a fix commit, 1 by a release; 7 kept. Go vulnerability database matches with no range: 0 judged; 0 ignored by the project's own advisory" chk " the clone holds no trees" "$(git -C "$T/gomodules/acme/widget.git" rev-parse --is-bare-repository)" "true" +# A build from a modified tree (+dirty); a pseudo-version that names a release, under the module path +# with a major-version suffix, whose fix commit is on another branch while the repository's other +# module path gives semantic ranges; the same advisory on a commit that is no release; an advisory +# naming only the other path. And the Go vulnerability database naming the old module path with no +# range, where the project's own advisory names only the new one. +PT=$(pseudo $(git -C "$G" rev-parse v2.6.3)) +graw(){ echo "$2" > "$T/gadv/$1.json"; } +pk(){ echo "{\"package\": {\"name\": \"$1\", \"ecosystem\": \"go-module\"}, \"detail\": {\"ranges\": [$2]}}"; } +gadv GHSA-r002 "$(rg '<1.6.27' '"1.6.27"'), $(rg '>=2.0.0,<2.2.4' '"2.2.4"')" "" +graw GHSA-s001 "[{\"vulnerability\": {\"id\": \"GHSA-s001\", \"provider\": \"github\", \"refs\": []}, \"packages\": [$(pk github.com/acme/widget "$(rg '<1.6.27' '"1.6.27"'), $(rg '>=2.0.0,<2.2.4' '"2.2.4"')"), $(pk github.com/acme/widget/v7 "$(rg "<0.0.0-20260129092249-${SIDE:0:12}" "\"0.0.0-20260129092249-${SIDE:0:12}\"")")]}]" +graw GHSA-s002 "[{\"vulnerability\": {\"id\": \"GHSA-s002\", \"provider\": \"github\", \"refs\": []}, \"packages\": [$(pk github.com/acme/widget "$(rg '<2.2.4' '"2.2.4"')")]}]" +own='{"url": "https://github.com/acme/widget/security/advisories/GHSA-p00X"}' +govdb(){ graw $1 "[{\"vulnerability\": {\"id\": \"$1\", \"provider\": \"govulndb\"}, \"packages\": [$(pk github.com/acme/widget "$2")]}]"; } +ghsa(){ graw $1 "[{\"vulnerability\": {\"id\": \"$1\", \"provider\": \"github\", \"refs\": [$3]}, \"packages\": [$2]}]"; } +govdb GO-2099-0001 ""; ghsa GHSA-p001 "$(pk github.com/acme/widget/v2 "$(rg '>=2.1.0,<2.1.9' '"2.1.9"')")" "${own/p00X/p001}" +govdb GO-2099-0002 ""; ghsa GHSA-p002 "$(pk github.com/acme/widget/v2 "$(rg '>=2.1.0,<2.1.9' '"2.1.9"')")" "" +govdb GO-2099-0003 "$(rg '<1.7.40' '"1.7.40"')"; ghsa GHSA-p003 "$(pk github.com/acme/widget/v2 "$(rg '>=2.1.0,<2.1.9' '"2.1.9"')")" "${own/p00X/p003}" +govdb GO-2099-0004 ""; ghsa GHSA-p004 "$(pk github.com/acme/widget "$(rg '<1.7.40' '"1.7.40"')"), $(pk github.com/acme/widget/v2 "$(rg '>=2.1.0,<2.1.9' '"2.1.9"')")" "${own/p00X/p004}" +govdb GO-2099-0005 ""; ghsa GHSA-p005 "$(pk github.com/acme/widget/v2 "$(rg '>=2.1.0,<2.1.9' '"2.1.9"')")" "${own/p00X/p005}" +gv(){ echo '{"vulnerability": {"id": "'$1'", "namespace": "govulndb:language:go", "severity": "Critical", "fix": {"versions": [], "state": "not-fixed"}}, "relatedVulnerabilities": [{"id": "CVE-2099-'${1: -4}'"}, {"id": "'$2'"}], "matchDetails": [{"type": "exact-direct-match", "found": {"versionConstraint": "'$3'"}}], "artifact": {"name": "github.com/acme/widget", "version": "v1.7.36", "type": "go-module"}}'; } +goreport2(){ cat > "$T/vuln.json" < Date: Sat, 17 Oct 2026 23:48:43 +0800 Subject: [PATCH 2/2] fix(rootfs): read the kernel's ixgbe file directly in the E610 check The E610 check looked ixgbe up by name in the jail's module tree (modinfo -k). The jail installs its kernel packages with rpm -i --nodeps, which never runs depmod on that tree, so the tree has no modules.dep or modules.alias to look a name up in. The check therefore failed on the 7.2.11 kernel although its ixgbe carries the E610 (jim pipeline #162). The check now finds the kernel's own ixgbe.ko in that tree and reads its aliases. The build still fails when the kernel has no ixgbe, or one without the E610 10GBASE-T id. Verified in the rebuilt jim build jail (7.2.11) by running the rule as committed through make. It passes for 7.2.11-1.el9.x86_64, whose ixgbe.ko.xz carries v00008086d000057B0, and fails for a kernel with no ixgbe in the tree. Refs bigstack-oss/cubecos#578 Signed-off-by: Jim Lin Co-authored-by: Eandalf Co-authored-by: Claude Opus 5.5 (1M context) --- rootfs/Makefile | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/rootfs/Makefile b/rootfs/Makefile index 486a9e4..0cc5dd0 100755 --- a/rootfs/Makefile +++ b/rootfs/Makefile @@ -110,8 +110,11 @@ base_rootfs_install: $(Q)$(INSTALL_MODULES) $(ROOTDIR) $(HEX_DATADIR)/os/base.modules @# The Intel E610 (8086:57AE-57B2) is known to the kernel's own ixgbe from Linux 6.14. Every @# image that carries NIC drivers takes ixgbe from the kernel's module tree, so check that tree: - @# fail the build on a kernel whose ixgbe lacks the E610 10GBASE-T id. - $(Q)modinfo -k $(KERNEL_VERS) -F alias ixgbe | grep -qi 'v00008086d000057B0' + @# fail the build on a kernel whose ixgbe lacks the E610 10GBASE-T id, or that has no ixgbe. The + @# module file is read directly: the jail installs its kernel packages without running depmod, + @# so its tree has no index to look ixgbe up by name in. + $(Q)f=$$(find /lib/modules/$(KERNEL_VERS)/kernel -name 'ixgbe.ko*' | head -1) && [ -n "$$f" ] && \ + modinfo -F alias $$f | grep -qi 'v00008086d000057B0' $(Q)$(INSTALL_DATA) $(ROOTDIR) $(HEX_DATADIR)/os/systemd-hexctl-sys.service ./lib/systemd/system $(Q)$(INSTALL_DATA) $(ROOTDIR) $(HEX_DATADIR)/os/passwd ./etc $(Q)$(INSTALL_DATA) $(ROOTDIR) $(HEX_DATADIR)/os/shadow ./etc