Repository navigation
on-prem: install, update and roll back the real engine offline in CI - #1918
Conversation
bin/build-release.sh builds a release archive from uv.lock (pinned wheels, hash-checked) plus the flow wheel, a checksum and an openssl signature. bin/test-offline-install.sh builds two signed releases, then as root inside run-egress-blocked.sh installs the first from its wheelhouse, refuses a tampered second, updates to the signed second, rolls back and runs the air-gap attestation. The new flow-onprem-offline job runs it, and runs bin/test-update.sh, which no job ran before.
…kout The version check ran python -c from packages/flow, so it imported the source checkout and reported 2.6.0 after the update to 2.6.0.post1. Run it isolated from /, require the module to live in the venv, and compare the module with the installed distribution and the CLI's --version.
A writer that found the audit lock owned by an exited process removed it without re-checking. If the owner released between the read and the removal and another writer took the lock, the breaker deleted the live lock and two writers could chain to the same parent. Break stale locks only under a second lock and only if the lock still names the same dead owner. Add a deterministic test that forces the handover, and print the verifier output when the concurrent-writer check fails.
…r(1) Ubuntu 26.04 ships uutils mkdir, which reports success to both callers when two processes race to create one directory. The audit-log writer lock and the release update lock both treated a zero exit from mkdir as ownership, so on that image concurrent audit writers chained to the same parent (the "concurrent audit writers preserve one hash chain" CI failure) and a second update could proceed beside a live one. Ownership is now the shell's own noclobber create of the pid file inside the lock directory (O_CREAT|O_EXCL), and the stale-lock breaker lock is an exclusive file too. Tests model the uutils behavior with a mkdir shim that never reports EEXIST; on the previous code they fail.
Independent adversarial review: APPROVEReviewed head
Syntax, release consistency, generality and source-boundary checks passed; the source-boundary run required an explicit repository name and disposable Git metadata because the transfer omitted Shellcheck and actionlint were unavailable; I did not run the full suite, sdist/container path, systemd service or an in-flight job. Synthetic private signing keys were not retained. This approval does not clear the inherited #1915 BLOCK or replace the separate #1922 lock review. |
Part of the on-prem deployment work. Stacked on #1915 (base
agent/142-c), which addsbin/run-egress-blocked.sh.What this does
The on-prem kit already had a signed offline update and rollback, but its only test (
bin/test-update.sh) fakes the venv build, and no CI job ran it. Nothing proved that a release archive actually holds everything the engine needs, or that the real engine installs and upgrades on a machine with no network.deploy/on-prem/bin/build-release.shbuilds a release archive on a connected host frompackages/flow/uv.lock: it downloads exactly the pinned wheels foropenadapt-flow[privacy], checks each against the lock's hashes (pip download --require-hashes), adds theopenadapt-flowwheel, and writes the archive, a.sha256, and an openssl.sigwhen given the key. UPDATE.md's hand recipe resolves the newest matching dependencies instead; it now points at the script first.deploy/on-prem/bin/test-offline-install.sh stage DIRbuilds two signed releases (this checkout as V1, the same source re-versioned as V1.post1 as V2) and a throwaway key, and deletes the private key afterwards.test-offline-install.sh verify DIRruns as root insiderun-egress-blocked.shand refuses to run outside it. It checks the machine has no route out, then: first install from V1's wheelhouse withinstall.sh --wheelhouse(pip has--no-indexand no network, so any wheel missing from the archive fails it); a tampered V2 archive is refused and leaves V1 current; a signed update to V2 (current and previous pointers, engine and privacy extra import at the new version, bundle and queued job unchanged); rollback to V1; thenverify-airgap.sh --probe --audit(the egress probe fails to reach the internet, and the audit hash chain is intact).flow-onprem-offlinein packages.yml runsbin/test-update.sh, then stage, then verify. It is wired into packages-gate for flow changes, and the workflows README lists it.Testing
bash -nare clean.bin/test-update.shpasses locally on macOS: 45 passed, 0 failed.bin/build-release.shlocally on macOS: builds an unsigned release with 89 wheels from the lock.openadaptlauncher distribution isn't installed in the venv I used (PackageNotFoundError). That is a gap in my environment; this branch doesn't touch those files.python -cfrom packages/flow, so it imported the source checkout instead of the installed engine. Fixed in f5eb3c3: the check now runs isolated from/, requires the module to live inside the release venv, and compares the module version, the installed distribution version andopenadapt-flow --version.Not tested