[abrichr]

Part of the on-prem deployment work. Stacked on #1918 (base agent/142-c2), which adds bin/build-release.sh and the flow-onprem-offline CI job.

What this does

The container option in deploy/on-prem didn't work as shipped. The Dockerfile installed openadapt-flow[privacy] from a ./wheels directory that nothing produced and that wasn't pinned to the lock. Its command, deploy/on-prem/bin/run-queue.sh (and bin/run-queue.sh in docker-compose.yml), was a relative path resolved from the storage volume, so the runner could not start. Nothing built or ran the image.

• bin/build-image.sh builds the image from a signed release archive (the one build-release.sh makes). It checks the signature against the pinned vendor key with the same code as install.sh --update, refuses a bad archive before calling docker, extracts the archive with the bounded extractor, and runs docker build --network none. pip only sees the release's own lock-pinned wheels, so a missing wheel fails the build instead of reaching an index. It tags openadapt-flow-onprem:<version> and can docker save it.
• The Dockerfile installs exactly the declared version from those wheels, the same way install.sh does, then runs pip check. It uses an absolute command path. The base image defaults to python:3.12-slim pinned by digest and must already be loaded on the host.
• docker-compose.yml no longer builds anything. It runs the image named in OPENADAPT_FLOW_IMAGE (required, no default) with pull_policy: never and the absolute run-queue path. The internal network and the no-ports rule are unchanged.
• bin/test-container.sh runs in the flow-onprem-offline job, after the releases are staged and before the egress-blocked install. It checks that:
  • a tampered archive is refused and no image is built;
  • the same release with the presidio_analyzer wheel removed fails to build, on that package;
  • the signed release builds;
  • inside the image, with no network, the engine and privacy extra import from the image's venv at the release version, and the CLI reports it;
  • the default command resolves, and run-queue.sh files a malformed job under failed/ and records it in the audit log on a bind-mounted storage root;
  • run-queue.sh refuses to start with scrubbing off;
  • compose refuses to start without the image named, never pulls, publishes no ports, and uses an internal network;
  • a TCP connect to 1.1.1.1:443 works from the same image on Docker's default network (the control) and fails from the compose runner;
  • the compose runner stays up watching the queue.
• README.md and docs/ON_PREM.md describe the new build step.

Testing

• Locally (macOS, no Docker daemon): shellcheck and actionlint are clean.
• I ran build-image.sh against a 2.6.0 release built from this lock (89 wheels), signed with a throwaway key, with a stub docker that records its arguments and the build context:
  • The tampered copy was refused by the signature check, and docker was never called.
  • The signed archive passed checksum and signature checks. It produced a context with release/ (VERSION, requirements.lock.txt, 89 wheels) and on-prem/, and called docker build --network none --pull=false.
• Contract tests: packages/flow test_ci_workflow_contract and test_onprem_service 43 passed. Root test_ci_required_check, test_flow_shards, test_ci_locked_installs and test_nightly 46 passed. check_release_consistency is OK.
• CI, flow-onprem-offline job on ubuntu-26.04 (run 41941053864, head d6db42e): test-update.sh 51 passed, 0 failed; test-container.sh 18 passed, 0 failed; the egress-blocked install and verify 21 passed, 0 failed. That is one green Linux run of the container checks.
• The first CI run of test-container.sh (head 977dfd7) failed 2 checks, both test bugs, fixed in d6db42e. With docker build -q, pip's error never reached the log, so the missing-wheel build failed for the right reason but the test couldn't tell; it now uses --progress=plain. The runner is root in the image and audit-log.sh makes the audit directory 0700, so the host could not read the log; the test now reads it from inside the image.
• Earlier runs on this stack failed "concurrent audit writers preserve one hash chain". The cause was the uutils mkdir on Ubuntu 26.04, which can report success to two racing callers, and that broke the audit and update locks. The fix is on-prem locks: hold on Ubuntu 26.04 (uutils mkdir) and never break a live writer's lock #1922, merged into this branch from on-prem: install, update and roll back the real engine offline in CI #1918.

Not tested

• A real bundle run inside the container. The test only drives the queue with a malformed job.
• The VLM appliance profile. It still needs a GPU host and a local model, and it reuses the same image.
• Arm64 hosts. The release has to be built on the target's architecture and Python minor version, as build-release.sh says.

[abrichr]

Independent adversarial review: BLOCK

P2: Keep unrelated local files out of the image. build-image.sh:69–70 copies the deployment directory with a short exclusion list; Dockerfile:38 then embeds everything copied. With an unchanged signed archive, adding an owned synthetic runtime-local.txt beside the scripts makes that file readable inside the real built image. Local site configuration or other data could therefore travel with a saved image. Stage only the required deployment files. The named storage/wheels/release exclusions passed; no confidential data was used.

P2: Refuse a missing base before invoking the builder. build-image.sh:73–76 does not enforce the documented local-base requirement. With an absent --base pointing to an owned loopback registry listener, the real Docker daemon opened a TCP connection and attempted HTTPS metadata resolution despite --network none --pull=false. The build then failed. Those flags did not prevent base resolution over the network. Check the base locally and enforce the advertised offline boundary before building.

Validation on WSL / Docker 29.9.3: the exact shipped container suite passed 18/18, including real engine/privacy imports, malformed-job audit, and Compose egress controls. The 43 Flow + 46 root contract tests passed on head d6db42e0a, literal parent 977dfd702, and stacked base 7770839e3. Frozen recorder probes produced 1 failure/8 passes on head and parent; three native Docker probes produced 2 failures/1 pass on each. The cached-base/package-version control passed. The stacked base lacks the new build entry point; its nine probe passes only confirm that absence. Source hashes stayed stable. Bash syntax, release consistency, generality and source-boundary checks passed; shellcheck/actionlint were unavailable. Initial runner PATH and listener-accounting errors are retained separately. The native canary image used a minimal synthetic wheel; the separate shipped suite used the real locked release. No real workflow bundle, VLM/GPU or Arm64 run is claimed.

[abrichr]

Answering BLOCK 6648883346 at 62c4849.

P2, unrelated local files: build-image.sh no longer copies the deploy directory with an exclusion list. It copies a named list into the build context: Dockerfile, docker-compose.yml, onprem.example.yaml, README.md, COMPLIANCE.md, UPDATE.md, and the four scripts the image runs (run-queue.sh, audit-log.sh, verify-airgap.sh, run-egress-blocked.sh). The scripts keep their modes. A named file that is missing, or is a symlink, stops the build. Host-only installers, units and build or test scripts stay out too.

P2, network base resolution: before docker build, it now runs docker image inspect on the base (--base, or the Dockerfile's digest-pinned default) and refuses if the base isn't loaded locally. It always passes that same base as BASE_IMAGE, so the image it checks is the one it builds.

test-container.sh now:

• Builds the signed release from a copy of the deploy directory with bin/site-notes.txt, vendor.key, onprem.yaml and storage/left.job placed beside the scripts. It checks that the image's /opt/openadapt-flow/deploy/on-prem holds exactly the 10 named files.
• Checks that an absent --base is refused with "is not in the local image store" before "== building", and that no image is tagged.

Testing, and what wasn't tested: Docker isn't available to me (the Mac's daemon isn't running, and I couldn't install it on my Linux box), so I haven't run the new test-container.sh checks against real Docker. Since this PR targets agent/142-c, the flow-onprem-offline job doesn't run on it either. Instead I ran build-image.sh against a recording docker shim, with a signed placeholder release and a site copy holding those stray files:

• Here, the context is the 10 named files. A default or --base image that isn't loaded is refused after one image inspect and before any build, and a loaded one builds with BASE_IMAGE set.
• At d6db42e, the context also held bin/site-notes.txt, vendor.key, onprem.yaml, the installers, launchd/systemd and the test scripts, and both absent-base cases went straight to docker build.

shellcheck finds no warnings. test-update.sh passes 49, and test_onprem_reboot.py plus test_onprem_service.py pass 41. GitHub reports this PR as conflicting with agent/142-c in .github/workflows/packages.yml. That conflict was already there, and this commit doesn't touch that file. I left it alone.

[abrichr]

From: Grok, To: coordinator

Merged origin/main at bc5357a (conflict-free locally after deepening the shallow main fetch). onprem_service+reboot: 41 passed.