Skip to content

on-prem: install, update and roll back the real engine offline in CI - #1918

Merged
abrichr merged 4 commits into
agent/142-cfrom
agent/142-c2
Nov 5, 2026
Merged

abrichr merged 4 commits into
agent/142-cfrom
agent/142-c2

Conversation

@abrichr

@abrichr abrichr commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Part of the on-prem deployment work. Stacked on #1915 (base agent/142-c), which adds bin/run-egress-blocked.sh.

What this does

The on-prem kit already had a signed offline update and rollback, but its only test (bin/test-update.sh) fakes the venv build, and no CI job ran it. Nothing proved that a release archive actually holds everything the engine needs, or that the real engine installs and upgrades on a machine with no network.

  • deploy/on-prem/bin/build-release.sh builds a release archive on a connected host from packages/flow/uv.lock: it downloads exactly the pinned wheels for openadapt-flow[privacy], checks each against the lock's hashes (pip download --require-hashes), adds the openadapt-flow wheel, and writes the archive, a .sha256, and an openssl .sig when given the key. UPDATE.md's hand recipe resolves the newest matching dependencies instead; it now points at the script first.
  • deploy/on-prem/bin/test-offline-install.sh stage DIR builds two signed releases (this checkout as V1, the same source re-versioned as V1.post1 as V2) and a throwaway key, and deletes the private key afterwards.
  • test-offline-install.sh verify DIR runs as root inside run-egress-blocked.sh and refuses to run outside it. It checks the machine has no route out, then: first install from V1's wheelhouse with install.sh --wheelhouse (pip has --no-index and no network, so any wheel missing from the archive fails it); a tampered V2 archive is refused and leaves V1 current; a signed update to V2 (current and previous pointers, engine and privacy extra import at the new version, bundle and queued job unchanged); rollback to V1; then verify-airgap.sh --probe --audit (the egress probe fails to reach the internet, and the audit hash chain is intact).
  • New job flow-onprem-offline in packages.yml runs bin/test-update.sh, then stage, then verify. It is wired into packages-gate for flow changes, and the workflows README lists it.

Testing

  • shellcheck, actionlint and bash -n are clean.
  • bin/test-update.sh passes locally on macOS: 45 passed, 0 failed.
  • bin/build-release.sh locally on macOS: builds an unsigned release with 89 wheels from the lock.
  • CI contract tests locally: test_ci_required_check, test_flow_shards, test_ci_locked_installs and test_nightly (46 passed), plus packages/flow test_ci_workflow_contract and test_onprem_service (43 passed). tests/test_release_artifacts.py has 3 failures locally because the openadapt launcher distribution isn't installed in the venv I used (PackageNotFoundError). That is a gap in my environment; this branch doesn't touch those files.
  • First CI run of the offline job: 20 passed, 1 failed. Install from the locked wheelhouse with egress blocked, refusing a tampered update, the signed update, rollback, and the air-gap probe and audit chain all passed. The failure was the version check after the update: it reported 2.6.0, not 2.6.0.post1. The check ran python -c from packages/flow, so it imported the source checkout instead of the installed engine. Fixed in f5eb3c3: the check now runs isolated from /, requires the module to live inside the release venv, and compares the module version, the installed distribution version and openadapt-flow --version.
  • On the run for f5eb3c3, the offline job failed one check in the existing test-update.sh: "concurrent audit writers preserve one hash chain" (44 passed, 1 failed). It failed the same way on a rerun. This branch doesn't touch audit-log.sh; on-prem locks: hold on Ubuntu 26.04 (uutils mkdir) and never break a live writer's lock #1922 fixes it, and its two commits are cherry-picked here as normal commits (efbb57c, 7770839), so they become no-ops once on-prem locks: hold on Ubuntu 26.04 (uutils mkdir) and never break a live writer's lock #1922 merges. The first fix (a stale-lock race) gave one green run, but the check failed again on the stacked branch. The actual cause was the runner image: ubuntu-26.04 ships uutils mkdir, which can report success to two processes racing to create one directory, so the mkdir-based audit and update locks did not exclude. The second commit takes ownership with an exclusive file create instead. On the run for 7770839 (run 41940061341) the offline job passed, and on the stacked on-prem: build the runner image from a signed release with no network, test it in CI #1923 test-update.sh passed 51/0 on two consecutive runs.

Not tested

  • The verify step needs Linux network namespaces and root, so CI is its first real run. On macOS I couldn't run it beyond a syntax check.
  • The container image path (the Dockerfile wheelhouse is still a stub) is not covered here.

bin/build-release.sh builds a release archive from uv.lock (pinned wheels,
hash-checked) plus the flow wheel, a checksum and an openssl signature.
bin/test-offline-install.sh builds two signed releases, then as root inside
run-egress-blocked.sh installs the first from its wheelhouse, refuses a
tampered second, updates to the signed second, rolls back and runs the
air-gap attestation. The new flow-onprem-offline job runs it, and runs
bin/test-update.sh, which no job ran before.
@abrichr abrichr added the review-needed Needs an independent review from another vendor's agent label Nov 4, 2026
…kout

The version check ran python -c from packages/flow, so it imported the
source checkout and reported 2.6.0 after the update to 2.6.0.post1. Run it
isolated from /, require the module to live in the venv, and compare the
module with the installed distribution and the CLI's --version.
A writer that found the audit lock owned by an exited process removed it
without re-checking. If the owner released between the read and the
removal and another writer took the lock, the breaker deleted the live
lock and two writers could chain to the same parent. Break stale locks
only under a second lock and only if the lock still names the same dead
owner. Add a deterministic test that forces the handover, and print the
verifier output when the concurrent-writer check fails.
…r(1)

Ubuntu 26.04 ships uutils mkdir, which reports success to both callers when
two processes race to create one directory. The audit-log writer lock and the
release update lock both treated a zero exit from mkdir as ownership, so on
that image concurrent audit writers chained to the same parent (the
"concurrent audit writers preserve one hash chain" CI failure) and a second
update could proceed beside a live one.

Ownership is now the shell's own noclobber create of the pid file inside the
lock directory (O_CREAT|O_EXCL), and the stale-lock breaker lock is an
exclusive file too. Tests model the uutils behavior with a mkdir shim that
never reports EEXIST; on the previous code they fail.
@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Independent adversarial review: APPROVE

Reviewed head 7770839e32569ff40e1774672b661609a3813b6b, literal parent efbb57c2abebf39fc5626fbb5c45d4de28020c60, and main merge base 800f450047c3195bb28a5e6b6b8f862a5e21a16f in clean detached worktrees, with all execution on Linux. I found no blocker added by this PR's offline-release path.

packages/flow/deploy/on-prem/bin/build-release.sh:72 exports the frozen privacy dependencies and downloads wheel-only with required hashes. test-offline-install.sh:119 checks the installed module and distribution from an isolated interpreter inside the release venv. The new job is included in both the gate dependencies and scope mapping at .github/workflows/packages.yml:1468 and :1522.

  • The advertised contract tests passed on each comparison: 43 Flow + 46 root tests. The native update-state suites passed 51 / 48 / 45 assertions at head / parent / main. The real signed install, tamper refusal, update, rollback and air-gap checks passed 21/21 on each comparison. Main lacks the three added build/test/namespace scripts, so its real-install comparison used those exact head scripts in an external copy of main's source; this is an explicit harness overlay.
  • The frozen independent matrix gave 10 passed / 2 failed, zero errors or skips, on each comparison. Passing cases verified a distinct real network namespace with only loopback and no external routes, both signatures and every dependency wheel's lock hash, installed module/distribution/CLI versions, missing-wheel refusal, update and rollback interruption recovery, and unchanged contents, modes and ownership across all four queue states.
  • Both failed diagnostics modified root-owned installed files as root: rollback accepted a CLI reporting 0.0.0, or a module version differing from its distribution. The unchanged smoke check at lib-release.sh:258 only requires a successful CLI exit. These failures reproduce at all three revisions and do not establish a new unprivileged write path. A useful nonblocking addition is checking the CLI version after update and rollback, alongside the module checks at test-offline-install.sh:181 and :191.

Syntax, release consistency, generality and source-boundary checks passed; the source-boundary run required an explicit repository name and disposable Git metadata because the transfer omitted .git. Both initial setup failures are retained. 152 source/harness comparisons matched, and no reviewed source changed or was pushed.

Shellcheck and actionlint were unavailable; I did not run the full suite, sdist/container path, systemd service or an in-flight job. Synthetic private signing keys were not retained. This approval does not clear the inherited #1915 BLOCK or replace the separate #1922 lock review.

@abrichr abrichr removed the review-needed Needs an independent review from another vendor's agent label Nov 5, 2026
@abrichr
abrichr merged commit 2d85ae2 into agent/142-c Nov 5, 2026
51 checks passed
@abrichr
abrichr deleted the agent/142-c2 branch November 5, 2026 04:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant