Repository navigation
Conversation
Node-local packages belong to the node owner, not to the shared boot root: the engine builds every SCI extension as an EROFS-root GPT image, standalone on the pinned published base, so a package change rebuilds exactly its own extension and the base image stops churning for every host. Seven extensions ship: base identities, compute, network, storage, libvirt and the two hypervisor flavours. Builds are byte-reproducible, so a produced raw traces to its inputs: version-keyed timestamps, per-image seeds and three conscious-bump pins (apt suite, toolchain manifest, resolved configuration). The tools tree is bootstrapped from the pinned suite and accepted only when its manifest digest matches the committed pin. The numeric name prefixes state the merge order systemd-sysext applies, the virt drivers exclude libvirt-owned paths through RemoveFiles= lists because the overlay merge provides them at runtime, and every path has exactly one owning extension. The qemu extension carries the VNC port policy that the legacy image kept in /etc/libvirt/qemu.conf, as a template with a tmpfiles C rule, because /etc is outside the merge. `mkosi/build.sh --bootstrap` prepares sandbox, tools tree and base without building images, for a caller that only needs the toolchain. Signed-off-by: Fabian Wiesel <[email protected]>
A delivery is only safe to apply when its raws are intact and were built for the host's base version. The step verifies SHA256SUMS before anything lands, refuses a base-version mismatch through the manifest's TARGET_BASE gate, reconciles idempotently, activates crash-safely and runs as a privileged pod with the host root mounted. Image mode takes the staging the image carries. Fetch mode materializes the same staging from a published build: it verifies the detached inventory signature, downloads only the files whose staged hash differs, proves the complete set and prunes files the inventory no longer lists. The manifest lists every deployable, and the step never deploys a raw it does not list. The host tests pin the build, staging, fetch and activation contracts. Signed-off-by: Fabian Wiesel <[email protected]>
An unverified layer silently widens the boot root's trust boundary: an unsigned or tampered extension could change what a host runs. Enable the signed build: every raw gets a dm-verity hash tree and a signature partition, the committed certificate is the anchor a merging host checks, and the private key never enters the repository. A three-scenario VM boot test closes the loop: the signed set must boot the merged system, and tampered or unsigned layers must be refused. An artifact is trusted only after the merged system has booted, so the boot evidence is part of the delivery proof. Signed-off-by: Fabian Wiesel <[email protected]>
fwiesel
force-pushed
the
sysext-virt
branch
13 times, most recently
from
November 3, 2026 07:35
2626df8 to
814415e
Compare
Local-only proof dies with the machine that ran it. The workflow runs the quick suite on every change and, on push, builds all extensions, runs the full suite with the VM boot test and publishes one test artifact: the raws staging as plain files, including the delivery step and the run certificate. The integration job consumes that artifact the way a consumer would: it restores the run certificate, bootstraps the pinned toolchain, rebuilds the delivery image from the verified staging, boots two Garden Linux hypervisors and proves the sysext refresh and the QEMU and Cloud-Hypervisor live migrations. The consumer workflow stays callable standalone, so any prior run's raws can be re-tested by run ID. The adapter tests pin every workflow contract structurally, on the parsed workflow. The publish job derives the one product from the same artifact: one release per build, tagged <Release>-g<short sha>, carrying the raws, the shims, the manifest, the run certificate and the signed inventory. It drops the delivery step and fails closed if it is still present, so the signature covers exactly what is uploaded. It runs only when the signing key is configured; without it the release steps are skipped. The hosted runners need explicit accommodations: mkosi's unprivileged sandbox needs the AppArmor user-namespace restriction lifted, qemu-system-x86 provides the virtual qemu-kvm package, /dev/kvm must be accessible for the VM boot test, the finished qemu scenario's VM is stopped before the cloud-hypervisor stage because its nested I/O starves the staging copy, and the debug collection includes the HV serial consoles. Signed-off-by: Fabian Wiesel <[email protected]>
Write down the single-owner layout, the delivery contract and the operating procedures while they are current, so the design doc is the single source of truth. The doc states the design as it is: every invariant, gotcha and open item in Simplified Technical English, no change log, no facts restated from tracked files. Signed-off-by: Fabian Wiesel <[email protected]>
fwiesel
force-pushed
the
sysext-virt
branch
5 times, most recently
from
November 3, 2026 13:01
2cf3634 to
00a1df9
Compare
fwiesel
force-pushed
the
sysext-virt
branch
4 times, most recently
from
November 4, 2026 08:55
5965725 to
5556255
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Moves the SCI virtualization stack out of the base image into seven
systemd-sysext extensions — EROFS-root GPT disk images with a signed
dm-verity pair — so each component owner ships and updates independently of
the base image, and a stock Garden Linux worker becomes a KVM hypervisor host
by merging extensions at runtime instead of re-imaging.
base; builds are byte-reproducible and pinned three ways (apt suite, tools
tree, resolved mkosi summary)
TARGET_BASE), staged-raw integrity,idempotent reconcile, crash-safe activation, runs without privilege
boot test proves the signed set boots and tampered/unsigned layers are
refused
tests on push. Two outputs with different status: the
sysext-rawsartifact is a test artifact (the staging plus
sci-delivery.sh, used by theintegration job to rebuild the delivery image), and
publishcreates theproduct — a GitHub release tagged
<Release>-g<short sha>(
2150.9.0-g5556255for this tip) that carries data only: seven raws, fouractivation shims,
extensions.manifest,verity-run.crt,SHA256SUMSandSHA256SUMS.sig(15 files)SHA256SUMSinventory verified againstFETCH_PUBKEY,only changed files fetched,
REMOVE=tombstones for dropped extensions, anda
publishjob that signs the release withSYSEXT_UNHARDENED_SIGNING_KEY; until that secret exists, the release stepsskip with a notice and no release is created (the run-local CA stays inside
the test run)
docs/sysext-virt.md— every invariant, gotcha and openitem; start there
The delivery contract in one paragraph
One manifest lists every extension in merge order; which extensions a
deployment runs is the consumer's choice (trim the manifest — the delivery
step never deploys a raw the manifest does not list). The release is the
product: a GitHub release tagged
<Release>-g<short sha>(today2150.9.0-g5556255) that carries data only — seven raws, four activationshims,
extensions.manifest,verity-run.crt,SHA256SUMSandSHA256SUMS.sig, 15 files.publishremovessci-delivery.shfrom theupload directory before it regenerates the inventory, and a fail-closed
test ! -eguard stops the job if the file survives, so the signature coversexactly the uploaded set. The
sysext-rawsartifact is the test vehicle, nota second product: it still carries
sci-delivery.shbecause the integrationjob rebuilds the delivery image from it.
/etccarrier: the libvirt VNC port range (32200–32299) ships as atemplate in the qemu extension and is materialized by a tmpfiles
Crule(copy-once) at activation, so no confext and no read-only
/etc.is the product. A reuser owns selection, staging persistence, key and
anchor provisioning, and the restart window. Fetch mode has no selection,
and its convergence needs a persistent mounted staging directory.
Series (5 commits)
feat: sysext build engine, seven extensions, byte-reproducible buildsfeat: sysext delivery step with version gatefeat: dm-verity signing and boot testci: prove the sysext artifacts on GitHubdocs: document the sysext build as it standsEach commit is a self-contained layer: the quick suite passes at every
stop (7 → 69 → 103 → 140 → 140 items), and every message states what the
commit introduces and why, without narrating how it evolved.
Deliberately out of scope
SYSEXT_UNHARDENED_SIGNING_KEYwhen the repository secret exists andotherwise with a run-local CA (test-only by construction);
publishsignsthe release only when the secret exists and skips with a notice otherwise.
The private key never travels in an artifact. Key generation, storage and
rotation are an open decision recorded in the doc.
rel-2150-dev— needs a re-pin to the current 2150.x and arebase; the branch pins 2150.9.0. This PR targets
mainfor review.Verification
extensions/tests/— 152 items (140 quick, 11 build, 1 vm). The quick tierruns on this PR; the push run builds all seven extensions, boot-tests the
merged system and runs the integration landscape on GitHub runners.
The full pipeline at
5556255is green end to end:quick suite (25 s), producer (full suite
152 passed in 251.60 s, VM boottest, the one raws artifact), publish (10 s, all three release steps skipped
with the notice until the signing secret exists), and the integration
landscape (11m34 s) — consumer-side toolchain bootstrap and delivery rebuild
from the verified raws, sysext refresh, and the QEMU and Cloud-Hypervisor live
migrations, all verified on runner hardware (24.04, following the repo's
runner policy). The PR-event sysext run 41803253878 is green. The repo's
own dev workflow also runs on this branch: its PR path builds the legacy
flavors and then runs the legacy integration through this PR's modified
test.yml(run 41803254637; it was still in progress at the time of writing),which is the backward-compatibility check for the sysext input gating.
Draft until: the deferred items above have owners (the green-run condition is
met).