Skip to content

sci: ship the SCI layer as verity-signed systemd-sysext extensions - #401

Draft
fwiesel wants to merge 5 commits into
mainfrom
sysext-virt
Draft

fwiesel wants to merge 5 commits into
mainfrom
sysext-virt

Conversation

@fwiesel

@fwiesel fwiesel commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

What this does

Moves the SCI virtualization stack out of the base image into seven
systemd-sysext extensions
— EROFS-root GPT disk images with a signed
dm-verity pair — so each component owner ships and updates independently of
the base image, and a stock Garden Linux worker becomes a KVM hypervisor host
by merging extensions at runtime instead of re-imaging.

  • build engine: every extension builds standalone on the pinned published
    base; builds are byte-reproducible and pinned three ways (apt suite, tools
    tree, resolved mkosi summary)
  • delivery step: version-gated (TARGET_BASE), staged-raw integrity,
    idempotent reconcile, crash-safe activation, runs without privilege
  • verity signing + boot test: every raw is signed; a three-scenario VM
    boot test proves the signed set boots and tampered/unsigned layers are
    refused
  • CI: quick suite on every change, full suite plus hypervisor integration
    tests on push. Two outputs with different status: the sysext-raws
    artifact is a test artifact (the staging plus sci-delivery.sh, used by the
    integration job to rebuild the delivery image), and publish creates the
    product — a GitHub release tagged <Release>-g<short sha>
    (2150.9.0-g5556255 for this tip) that carries data only: seven raws, four
    activation shims, extensions.manifest, verity-run.crt, SHA256SUMS and
    SHA256SUMS.sig (15 files)
  • fetch: a signed SHA256SUMS inventory verified against FETCH_PUBKEY,
    only changed files fetched, REMOVE= tombstones for dropped extensions, and
    a publish job that signs the release with
    SYSEXT_UNHARDENED_SIGNING_KEY; until that secret exists, the release steps
    skip with a notice and no release is created (the run-local CA stays inside
    the test run)
  • design doc: docs/sysext-virt.md — every invariant, gotcha and open
    item; start there

The delivery contract in one paragraph

One manifest lists every extension in merge order; which extensions a
deployment runs is the consumer's choice (trim the manifest — the delivery
step never deploys a raw the manifest does not list). The release is the
product: a GitHub release tagged <Release>-g<short sha> (today
2150.9.0-g5556255) that carries data only — seven raws, four activation
shims, extensions.manifest, verity-run.crt, SHA256SUMS and
SHA256SUMS.sig, 15 files. publish removes sci-delivery.sh from the
upload directory before it regenerates the inventory, and a fail-closed
test ! -e guard stops the job if the file survives, so the signature covers
exactly the uploaded set. The sysext-raws artifact is the test vehicle, not
a second product: it still carries sci-delivery.sh because the integration
job rebuilds the delivery image from it.

  • /etc carrier: the libvirt VNC port range (32200–32299) ships as a
    template in the qemu extension and is materialized by a tmpfiles C rule
    (copy-once) at activation, so no confext and no read-only /etc.
  • Scope: the delivery step is the boot-test vehicle; the signed release
    is the product. A reuser owns selection, staging persistence, key and
    anchor provisioning, and the restart window. Fetch mode has no selection,
    and its convergence needs a persistent mounted staging directory.

Series (5 commits)

  1. feat: sysext build engine, seven extensions, byte-reproducible builds
  2. feat: sysext delivery step with version gate
  3. feat: dm-verity signing and boot test
  4. ci: prove the sysext artifacts on GitHub
  5. docs: document the sysext build as it stands

Each commit is a self-contained layer: the quick suite passes at every
stop (7 → 69 → 103 → 140 → 140 items), and every message states what the
commit introduces and why, without narrating how it evolved.

Deliberately out of scope

  • Production signing-key custody — the producer signs with
    SYSEXT_UNHARDENED_SIGNING_KEY when the repository secret exists and
    otherwise with a run-local CA (test-only by construction); publish signs
    the release only when the secret exists and skips with a notice otherwise.
    The private key never travels in an artifact. Key generation, storage and
    rotation are an open decision recorded in the doc.
  • Landing on rel-2150-dev — needs a re-pin to the current 2150.x and a
    rebase; the branch pins 2150.9.0. This PR targets main for review.

Verification

extensions/tests/ — 152 items (140 quick, 11 build, 1 vm). The quick tier
runs on this PR; the push run builds all seven extensions, boot-tests the
merged system and runs the integration landscape on GitHub runners.

The full pipeline at 5556255 is green end to end:
quick suite (25 s), producer (full suite 152 passed in 251.60 s, VM boot
test, the one raws artifact), publish (10 s, all three release steps skipped
with the notice until the signing secret exists), and the integration
landscape (11m34 s) — consumer-side toolchain bootstrap and delivery rebuild
from the verified raws, sysext refresh, and the QEMU and Cloud-Hypervisor live
migrations, all verified on runner hardware (24.04, following the repo's
runner policy). The PR-event sysext run 41803253878 is green. The repo's
own dev workflow also runs on this branch: its PR path builds the legacy
flavors and then runs the legacy integration through this PR's modified
test.yml (run 41803254637; it was still in progress at the time of writing),
which is the backward-compatibility check for the sysext input gating.

Draft until: the deferred items above have owners (the green-run condition is
met).

Node-local packages belong to the node owner, not to the shared boot
root: the engine builds every SCI extension as an EROFS-root GPT
image, standalone on the pinned published base, so a package change
rebuilds exactly its own extension and the base image stops churning
for every host. Seven extensions ship: base identities, compute,
network, storage, libvirt and the two hypervisor flavours.

Builds are byte-reproducible, so a produced raw traces to its inputs:
version-keyed timestamps, per-image seeds and three conscious-bump
pins (apt suite, toolchain manifest, resolved configuration). The
tools tree is bootstrapped from the pinned suite and accepted only
when its manifest digest matches the committed pin. The numeric name
prefixes state the merge order systemd-sysext applies, the virt
drivers exclude libvirt-owned paths through RemoveFiles= lists
because the overlay merge provides them at runtime, and every path
has exactly one owning extension. The qemu extension carries the VNC
port policy that the legacy image kept in /etc/libvirt/qemu.conf, as
a template with a tmpfiles C rule, because /etc is outside the merge.
`mkosi/build.sh --bootstrap` prepares sandbox, tools tree and base
without building images, for a caller that only needs the toolchain.

Signed-off-by: Fabian Wiesel <[email protected]>
A delivery is only safe to apply when its raws are intact and were
built for the host's base version. The step verifies SHA256SUMS before
anything lands, refuses a base-version mismatch through the manifest's
TARGET_BASE gate, reconciles idempotently, activates crash-safely and
runs as a privileged pod with the host root mounted.

Image mode takes the staging the image carries. Fetch mode materializes
the same staging from a published build: it verifies the detached
inventory signature, downloads only the files whose staged hash
differs, proves the complete set and prunes files the inventory no
longer lists. The manifest lists every deployable, and the step never
deploys a raw it does not list. The host tests pin the build, staging,
fetch and activation contracts.

Signed-off-by: Fabian Wiesel <[email protected]>
An unverified layer silently widens the boot root's trust boundary:
an unsigned or tampered extension could change what a host runs. Enable
the signed build: every raw gets a dm-verity hash tree and a signature
partition, the committed certificate is the anchor a merging host
checks, and the private key never enters the repository.

A three-scenario VM boot test closes the loop: the signed set must boot
the merged system, and tampered or unsigned layers must be refused. An
artifact is trusted only after the merged system has booted, so the
boot evidence is part of the delivery proof.

Signed-off-by: Fabian Wiesel <[email protected]>
@fwiesel
fwiesel force-pushed the sysext-virt branch 13 times, most recently from 2626df8 to 814415e Compare November 3, 2026 07:35
Local-only proof dies with the machine that ran it. The workflow runs
the quick suite on every change and, on push, builds all extensions,
runs the full suite with the VM boot test and publishes one test
artifact: the raws staging as plain files, including the delivery step
and the run certificate.

The integration job consumes that artifact the way a consumer would: it
restores the run certificate, bootstraps the pinned toolchain,
rebuilds the delivery image from the verified staging, boots two Garden
Linux hypervisors and proves the sysext refresh and the QEMU and
Cloud-Hypervisor live migrations. The consumer workflow stays callable
standalone, so any prior run's raws can be re-tested by run ID. The
adapter tests pin every workflow contract structurally, on the parsed
workflow.

The publish job derives the one product from the same artifact: one
release per build, tagged <Release>-g<short sha>, carrying the raws,
the shims, the manifest, the run certificate and the signed inventory.
It drops the delivery step and fails closed if it is still present, so
the signature covers exactly what is uploaded. It runs only when the
signing key is configured; without it the release steps are skipped.

The hosted runners need explicit accommodations: mkosi's unprivileged
sandbox needs the AppArmor user-namespace restriction lifted,
qemu-system-x86 provides the virtual qemu-kvm package, /dev/kvm must be
accessible for the VM boot test, the finished qemu scenario's VM is
stopped before the cloud-hypervisor stage because its nested I/O
starves the staging copy, and the debug collection includes the HV
serial consoles.

Signed-off-by: Fabian Wiesel <[email protected]>
Write down the single-owner layout, the delivery contract and the
operating procedures while they are current, so the design doc is the
single source of truth. The doc states the design as it is: every
invariant, gotcha and open item in Simplified Technical English, no
change log, no facts restated from tracked files.

Signed-off-by: Fabian Wiesel <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant