Harden the remote sysext delivery path (signing custody, base anchor, egress, retention) #402

Context

PR #401 (branch sysext-virt, commit 2cf3634) adds a fetch mode to
delivery/sci-delivery.sh. It downloads a signed SHA256SUMS inventory and
SHA256SUMS.sig, verifies the signature against FETCH_PUBKEY, fetches only
changed files, prunes stale staged files to the inventory, and then runs the
existing integrity, base-version and activation flow.

CI publishes one release per build (<Release>-g<short sha>) from the raws
artifact. The publish job (contents: write, push only) signs and uploads.
REMOVE=<name> tombstones are the only deletion path. The design rationale
(systemd-sysupdate must not own /var/lib/extensions: versioned names +
InstancesMax >= 2 + systemd-sysext merging every *.raw) is in
docs/sysext-virt.md §11/§13.

This issue records the production-hardening work deliberately left out of that
feature. Each item states the gap and its acceptance criteria.

1. Key custody

The signing key is a repository secret named SYSEXT_UNHARDENED_SIGNING_KEY
(cert CN sci-sysext-unhardened). Any workflow allowed to run on a permitted
ref can read it.

• Move it to an environment-restricted secret or an OIDC signing service.
• Document rotation: key set in the chart, two certificates during overlap.
• Drop unhardened from the secret name, the key file
  (sysext-unhardened-signing.pem), the cert CN and the chart value
  (sciSysext.unhardenedSigningKey).

Acceptance: a documented rotation drill, and no unhardened name left.

2. Base-image anchor

Fetched raws are signed by mkosi with the same key, so a host must already
carry the matching certificate at /usr/lib/verity.d/verity.crt. /usr is
read-only at runtime, so the agent cannot install it. Only the integration
overlay injects it today.

Acceptance: a stock base image that carries the certificate merges a
fetched raw; a base without it refuses clearly.

3. Egress from the network role

ovn-controller is not hostNetwork (unlike compute-kvm); fetching from the
release origin is unproven there.

Acceptance: a recorded one-command probe from that pod (or a documented
proxy) and a fallback when it fails.

4. Release provenance

Release assets are replaceable and the tag is created with
--target "$GITHUB_SHA". Consider pinning FETCH_SUMS_DIGEST per role in the
chart, or adding an attestation (gh attestation) and verifying it.

Acceptance: a host cannot be moved to a different build by a re-upload or a
DNS/TLS substitution.

5. Retention and rollback

Version retention is deliberately outside /var/lib/extensions. Add the
retained previous set (/var/lib/sci-sysext/previous/<build-id>) and a
rollback path.

Acceptance: a bad build can be reverted without a re-download.

6. Fetch robustness

• Resume/partial retry for large raws.
• Atomic staging swap (the current pass mutates staging in place and repairs on
  the next pass).
• Behaviour with two passes overlapping.
• A bounded total failure time.

Acceptance: a killed fetch never leaves a host without a working set and
never wedges a later pass.

7. Tombstone edge cases

Removal-side crash replay has no in-repo test (only an external reproducer).
Also untested: duplicate REMOVE=, malformed FETCH_SUMS_DIGEST, unwritable
FETCH_DIR, explicit STAGING_DIR in fetch mode, and a listed name ending in
.part.

Acceptance: tests for each.

8. Publish verification

The publish job regenerates SHA256SUMS and immediately runs sha256sum -c
against it (a tautology). Verify the downloaded artifact's own checksums
before regenerating.

Acceptance: a corrupted artifact fails the publish job.

9. Packaging hygiene

systemd-sysupdate currently travels inside the sci-51-virt-qemu delta. If
nothing uses it, drop it with RemoveFiles=.

Acceptance: no unused update tool in the payload.

10. Docs

After the above, update docs/sysext-virt.md §11/§13 and delivery/README.md
with the final trust model and the chart contract.

Current green evidence

• The branch's sysext push runs carry the full suite and both hypervisor
  migrations (QEMU and Cloud Hypervisor). The run for 2cf3634 was still
  executing when this issue was filed
  (run);
  earlier push runs on the branch were green.
• The PR's checks are green
  (PR #401).
• The publish job currently skips with a notice until the signing secret
  exists; that is expected, not a failure.