Repository navigation
perf(rootfs): compress the split .cgz files and rootfs.cgz concurrently - #209
Merged
Merged
Conversation
mountrootfs compressed a rootfs over 8 GiB one .cgz at a time: one per second-level folder over 512 MiB, opensearch, the two registry halves, the remainder in rootfs_0.cgz, and then the whole rootfs.cgz. The eleven jobs read the same tree and write different files, so they now run concurrently. Each job's pid is kept and waited on, and the script fails once all of them have finished if any failed, because a bare wait would return 0 and let a broken .cgz into the pkg. A job fails on the same condition as before, the exit status of the pigz at the end of its pipeline. A rootfs under 8 GiB still produces only rootfs.cgz. Measured on bldsrv-200-13 (80 cores) by rerunning these commands in a jail container on the rootfs of CUBE build #105 (21.5 GB as cpio). One at a time took 165-181 s and all at once 112-146 s; run back to back, the two pairs were 165 against 112 s and 170 against 146 s. The eleven files are byte-identical either way, 19263215190 bytes in total. CPU time rose 2-4% (about 5520 to 5640 CPU-s). The average number of busy cores, 39-50, stays below the 52 that the single pigz on the whole rootfs already uses today. The block from the old and new scripts, run with stubs for du, xargs -i and pigz, produces the same files with the same cpio member lists, in 1 s instead of 8 s. With one output unwritable, the new script exits 1 with the error, as the old one did. The same script ending in a bare wait exits 0. In CUBE build #107 on bldsrv-200-13, the GEN rootfs.cgz step, which runs this script, took 548 s against 596-608 s in builds #102-#105. Its ten split .cgz files pass pigz -t. Their cpio member lists together match rootfs.cgz, except for seven parent directories under opt/docker, which the registry and remainder slices already leave out without this change. Signed-off-by: Seki Xu <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
`umount proc` names no path, so umount takes it as a source and unmounts
the most recently mounted proc on the system. When another mountrootfs is
running, that is the other one's proc: a cubecos build lost heavy_base's
/proc this way to the install/pxe/pxe_server/iso initramfs runs, and
every rpm scriptlet after that took about 2 minutes. Unmount
${ROOTDIR}/proc, like the trap and the two later umounts already do.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Seki Xu <[email protected]>
Eandalf-Bigstack
force-pushed
the
perf/parallel-rootfs-split
branch
from
October 19, 2026 07:39
77b3c7a to
3424e48
Compare
Eandalf-Bigstack
approved these changes
Oct 19, 2026
Eandalf-Bigstack
added a commit
to bigstack-oss/cubecos
that referenced
this pull request
Oct 19, 2026
The rootfs ships its container images in a registry under /opt/docker/registry, and the SBOM's syft run over the rootfs sees none of their packages. hex now has an opt-in sbom-images target that scans each of those images on its own and writes a second SBOM and report, <release>_sbom_images.json and <release>_vuln_images.json, that add them to copies of the rootfs pair. core/main names the registry for it in PROJ_SBOM_REGISTRY. Nothing in the pipeline runs the target, so the rootfs SBOM and report, which the DefectDojo import and the issue filing read, stay as they are. Running make -C core/main sbom-images after sbom produces the image pair. The hex bump also picks up what hex develop gained since the last bump: the concurrent compression of the split .cgz files and rootfs.cgz (bigstack-oss/hex#209), the fix that unmounts this rootfs's proc rather than the last one mounted, and two GitHub Actions bumps. Verified in hex with scripts/tests/test_scanimages.sh (26 checks) and test_projsbom.sh (16), and with a full run of the scan on this registry from a develop rootfs (092b08e): 56 unique images across 58 tags, about 13 minutes, the rootfs report carried through unchanged, and the registry's files left unchanged. Refs #578 Signed-off-by: Jim Lin <[email protected]> Co-authored-by: Eandalf <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Eandalf-Bigstack
added a commit
to bigstack-oss/cubecos
that referenced
this pull request
Oct 20, 2026
The rootfs ships its container images in a registry under /opt/docker/registry, and the SBOM's syft run over the rootfs sees none of their packages. hex now has an opt-in sbom-images target that scans each of those images on its own and writes a second SBOM and report, <release>_sbom_images.json and <release>_vuln_images.json, that add them to copies of the rootfs pair. It applies no VEX document to the images: the rootfs's statements name packages by a purl with no version and would hide image matches judged for other binaries. core/main names the registry for it in PROJ_SBOM_REGISTRY. Nothing in the pipeline runs the target, so the rootfs SBOM and report, which the DefectDojo import and the issue filing read, stay as they are. Running make -C core/main sbom-images after sbom produces the image pair. The hex bump also picks up what hex develop gained since the last bump: the installdnf fix that drops a cached rpm older than the build the rootfs already has, the concurrent compression of the split .cgz files and rootfs.cgz (bigstack-oss/hex#209), the fix that unmounts this rootfs's proc rather than the last one mounted, and two GitHub Actions bumps. Verified in hex with scripts/tests/test_scanimages.sh (26 checks) and test_projsbom.sh (17), and with a full run of the scan on this registry from a develop rootfs (092b08e): 56 unique images across 58 tags, about 13 minutes, the rootfs report carried through unchanged, and the registry's files left unchanged. Refs #578 Signed-off-by: Jim Lin <[email protected]> Co-authored-by: Eandalf <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Eandalf-Bigstack
added a commit
to bigstack-oss/cubecos
that referenced
this pull request
Oct 20, 2026
The rootfs ships its container images in a registry under /opt/docker/registry, and the SBOM's syft run over the rootfs sees none of their packages. hex now has an opt-in sbom-images target that scans each of those images on its own and writes a second SBOM and report, <release>_sbom_images.json and <release>_vuln_images.json, that add them to copies of the rootfs pair. It applies no VEX document to the images: the rootfs's statements name packages by a purl with no version and would hide image matches judged for other binaries. core/main names the registry for it in PROJ_SBOM_REGISTRY. Nothing in the pipeline runs the target, so the rootfs SBOM and report, which the DefectDojo import and the issue filing read, stay as they are. Running make -C core/main sbom-images after sbom produces the image pair. The hex bump also picks up what hex develop gained since the last bump: the installdnf fix that drops a cached rpm older than the build the rootfs already has, the concurrent compression of the split .cgz files and rootfs.cgz (bigstack-oss/hex#209), the fix that unmounts this rootfs's proc rather than the last one mounted, and two GitHub Actions bumps. Verified in hex with scripts/tests/test_scanimages.sh (26 checks) and test_projsbom.sh (17), and with a full run of the scan on this registry from a develop rootfs (092b08e): 56 unique images across 58 tags, about 13 minutes, the rootfs report carried through unchanged, and the registry's files left unchanged. Refs #578 Signed-off-by: Jim Lin <[email protected]> Co-authored-by: Eandalf <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind feature
What this PR does / why we need it
mountrootfstakes about 600 s in a CUBE build, and 165–181 s of that is compression. For a rootfs over 8 GiB, the script writes eleven .cgz files one at a time: one for each second-level folder over 512 MiB, one for opensearch, two for the registry halves,rootfs_0.cgzfor the rest, and then the wholerootfs.cgz. All eleven read the same tree and write different files, so this PR starts them together.The script keeps each job's pid and waits on every one of them. If any job fails, it calls
Errorafter all of them finish. A barewaitwould return 0 and let a broken .cgz into the pkg. A job fails under the same condition as before: the exit status of the pigz at the end of its pipeline. The file contents andpigz -9stay the same, so the ISO does not grow. A rootfs under 8 GiB still produces onlyrootfs.cgz.Validation
GEN rootfs.cgztook 548 s. Builds chore(deps): bump actions/upload-artifact from 6.0.0 to 7.0.0 #102–chore(deps): bump github/codeql-action from 4.32.6 to 4.35.1 #105 on the same node without this change took 596–608 s, so the step is 48–60 s (about 9 %) shorter.pigz -t. Their cpio member lists together matchrootfs.cgz, except for seven parent directories underopt/docker. The registry and remainder slices already leave those out today, andcpio -drecreates them on install.du,xargs -iandpigz. With one output that could not be written, both exited 1. The same block ending in a barewaitexited 0.Which issue(s) this PR fixes
None. This is a build-time improvement with no ticket.
Special notes for your reviewer
cubecos picks this up with a hex submodule bump, which will go out together with other hex changes.
Additional documentation
None.
🤖 Generated with Claude Code