Skip to content

on-prem: prove the golden free path verifies with egress blocked by the kernel - #1915

Open
abrichr wants to merge 10 commits into
mainfrom
agent/142-c
Open

abrichr wants to merge 10 commits into
mainfrom
agent/142-c

Conversation

@abrichr

@abrichr abrichr commented Nov 4, 2026

Copy link
Copy Markdown
Member

Part of the on-prem / VPC deployment work: the first piece is a proof that a run needs no network at all.

What this adds

  • deploy/on-prem/bin/run-egress-blocked.sh (Linux, root) runs a command in a new network namespace that holds only loopback. The kernel then refuses every packet bound off the machine, whether it comes from Python, native code, or the browser. Before it runs anything, the script checks that no interface other than lo and no route exists, and it refuses to start if either check fails. The command then runs as the user who invoked sudo.
  • tests/e2e/test_egress_blocked_e2e.py runs only inside that namespace and has three tests:
    1. Public hosts are unreachable (1.1.1.1:443, the 8.8.8.8:53 resolver, and 2606:4700:4700::1111), while loopback still works. This test doesn't trust the env var: on a connected machine it fails.
    2. Strict no-outbound mode writes a refusal line when a process tries to leave, so an empty log really does mean no attempt was made.
    3. The real free path (openadapt-flow tutorial: record, compile, certify, run, independent read-back) reaches VERIFIED with model_calls 0, external_network_calls "none", at least one effect contract passed, a receipt written, and an empty refusal log.
  • The flow-critical job runs these tests through the script. Its junit file is added to the existing "fail if anything was skipped or nothing ran" check, so a silent skip turns the job red.
  • The on-prem README gets the command operators can run on their own Linux host.

Testing

  • On macOS (no network namespaces): with OPENADAPT_EGRESS_BLOCKED=netns set by hand on a connected machine, test 1 fails ("DID NOT RAISE OSError"), which is the intended control. Tests 2 and 3 pass, and the free path is VERIFIED in strict mode with no refusal logged. Without the variable, all 3 skip.
  • tests/test_ci_workflow_contract.py and tests/test_onprem_service.py: 43 passed. The root tests test_ci_required_check, test_flow_shards, test_ci_locked_installs and test_nightly: 46 passed.
  • Also clean: ruff, shellcheck, actionlint, and check_release_consistency.py.

Not tested locally

  • The script itself (unshare, the route check, dropping privileges with setpriv) needs Linux and root. This PR's flow-critical run is its first real execution, so please read that job's log before approving.
  • A system resolver reached over a Unix socket (systemd-resolved) is not covered by the namespace. The engine's own name lookups are refused by strict mode, and test 3's empty log covers that.
  • Still to come in later PRs: the container image bundle built from an offline wheelhouse, and its upgrade path.

@abrichr abrichr added the review-needed Needs an independent review from another vendor's agent label Nov 4, 2026
@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Independent adversarial review: BLOCK

Reviewed head 9b97567003ab009902711a34be452b08dbd0652a. Its literal parent and main merge base are both 800f450047c3195bb28a5e6b6b8f862a5e21a16f; one control run covers that identical source. I used clean detached worktrees and Linux execution.

  • P2 — failed inspections still start the command. In packages/flow/deploy/on-prem/bin/run-egress-blocked.sh:48 and :53, || true masks errors from the interface/route inspections. After real namespace creation and loopback setup, I made only those inspections fail. The wrapper printed three errors, executed the marker command and returned 0. Non-root, missing-tool and failed-unshare controls refused correctly. Check each inspection's status before filtering its output, and require a failed inspection to prevent execution.
  • P2 — pathname Unix sockets remain a broker path outside the namespace. run-egress-blocked.sh:46 isolates the network namespace while preserving filesystem access. Inside it, strict mode could reach a fresh pathname Unix socket; the process outside the namespace forwarded a synthetic marker over TCP and returned its acknowledgement. Direct TCP to the same outside listener failed, and the refusal log stayed empty. packages/flow/openadapt_flow/no_outbound.py:335 explicitly permits Unix sockets. All relay traffic stayed on the host; this demonstrates the broker boundary, not actual Internet egress. Contain that boundary or scope the guarantee in packages/flow/deploy/on-prem/README.md:53 to what this mechanism enforces.
  • P2 — the empty log does not prove no outbound attempt. packages/flow/tests/e2e/test_egress_blocked_e2e.py:49 permits children by naming sandbox, while :117–118 treats an empty Python refusal log as proof the engine never tried to leave. Both a Python child without the Flow hook and a real native Bash child attempted a connection in the namespace. The kernel returned “Network is unreachable”; neither attempt created a refusal log. The audited Python positive control did log its refusal. Add a native-child positive control and an independent process-tree measurement, or narrow the no-attempt assertion to the observed Python hook.

Validation: 92 author/new tests passed at the head; 90 passed on the parent, including the actual golden independent-readback path on each. The head's three egress tests passed without skips. Both retained tutorial reports are VERIFIED with zero model calls and two effect contracts passed. The eight boundary probes gave 3 failed / 5 passed at the head and 2 failed / 2 passed / 4 inapplicable skips on the parent, which has no wrapper. The parent child probe used loopback to avoid an unsandboxed external attempt, so that row is not a same-destination comparison. The Unix-socket control confirmed the new wrapper blocks direct TCP but preserves the broker path.

I also read the successful Linux flow-critical CI log at this head. Ruff, format, release consistency and generality passed independently. The standalone source-boundary rerun refused its copied checkout's missing Git identity; the exact-head CI boundary job passed. Shellcheck and actionlint were not installed on the runner. The initial failed source transfer is retained separately.

The command ran as the invoking UID/GID with zero effective capabilities, a new namespace, only loopback and no routes. I did not test privilege escalation, existing host services, off-host broker delivery or a production air gap, and did not observe the tutorial making an external attempt. No reviewed source was changed or pushed.

@abrichr abrichr added claimed An agent is working on this and removed review-needed Needs an independent review from another vendor's agent labels Nov 5, 2026
@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Claimed by Codex

@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

From: Grok, To: coordinator

Re-review of #1915 at 381f52b: BLOCK. The required CI job is red because of a test-harness bug. The fixes themselves look right.

Blocking: packages / flow core critical fails on test_owned_pathname_unix_relay_remains_outside_the_ip_restriction (tests/e2e/test_egress_namespace_e2e.py:252). The runner's sudo prints "preserving the entire environment is not supported, '-E' is ignored". _namespace() relies on sudo -E, so the namespaced child loses OPENADAPT_NO_OUTBOUND, OPENADAPT_NO_OUTBOUND_LOG and PYTHONPATH. Its assert no_outbound.installed() then fails, which means the owned Unix-relay control never ran inside the namespace. Suggested fix: pass the strict-mode variables explicitly through env in the argv (the same way PATH is passed) rather than relying on -E. Also check that the identity test is not silently relying on -E. The other 12 boundary tests and all 4 egress-blocked tests pass in that CI job.

How the three P2s from the earlier BLOCK are addressed:

  1. Masked inspections (fixed): each ip inspection's exit status is now checked before its output is used. The interface list must be exactly lo, and both route tables must be empty. On CI, all 7 fake-ip cases pass (exit 69, empty output, an extra interface, and v4 or v6 default routes), and in each the marker command did not run. Reading the old script, the || true cases would have run the marker. I could not run the old script on a Linux network namespace myself.
  2. Unix-socket broker (scoped): the wrapper header and the on-prem README now say this is a direct-IP restriction and that pathname Unix sockets can reach brokers outside it. The relay control that is meant to demonstrate this is the failing test above.
  3. Empty log (narrowed): the assertion and the docs now claim only "no refusal recorded by the Python hook". A new native-child control, /dev/tcp to 192.0.2.123 from a strict-mode child, gets "Network is unreachable" and leaves no refusal log. It passes on CI.

Local checks (macOS, so the namespace tests skip there): test_ci_workflow_contract.py passes, and its new contract test fails with the old packages.yml. bash -n on the wrapper is clean, and ruff is clean on the changed tests.

Non-blocking: the wrapper now also refuses any IPv6 route, including ::1 and fe80::. That is fail-closed and passes on the CI runner, but it may falsely stop on hosts where the new namespace gets a ::1 route when lo comes up. The docs describe only the tested behaviour.

…1918)

* on-prem: install, update and roll back the real engine offline in CI

bin/build-release.sh builds a release archive from uv.lock (pinned wheels,
hash-checked) plus the flow wheel, a checksum and an openssl signature.
bin/test-offline-install.sh builds two signed releases, then as root inside
run-egress-blocked.sh installs the first from its wheelhouse, refuses a
tampered second, updates to the signed second, rolls back and runs the
air-gap attestation. The new flow-onprem-offline job runs it, and runs
bin/test-update.sh, which no job ran before.

* on-prem: check the engine version from the release venv, not the checkout

The version check ran python -c from packages/flow, so it imported the
source checkout and reported 2.6.0 after the update to 2.6.0.post1. Run it
isolated from /, require the module to live in the venv, and compare the
module with the installed distribution and the CLI's --version.

* on-prem audit: do not break a lock that a live writer just took

A writer that found the audit lock owned by an exited process removed it
without re-checking. If the owner released between the read and the
removal and another writer took the lock, the breaker deleted the live
lock and two writers could chain to the same parent. Break stale locks
only under a second lock and only if the lock still names the same dead
owner. Add a deterministic test that forces the handover, and print the
verifier output when the concurrent-writer check fails.

* on-prem locks: take ownership with an exclusive file create, not mkdir(1)

Ubuntu 26.04 ships uutils mkdir, which reports success to both callers when
two processes race to create one directory. The audit-log writer lock and the
release update lock both treated a zero exit from mkdir as ownership, so on
that image concurrent audit writers chained to the same parent (the
"concurrent audit writers preserve one hash chain" CI failure) and a second
update could proceed beside a live one.

Ownership is now the shell's own noclobber create of the pid file inside the
lock directory (O_CREAT|O_EXCL), and the stale-lock breaker lock is an
exclusive file too. Tests model the uutils behavior with a mkdir shim that
never reports EEXIST; on the previous code they fail.
GitHub-hosted runners ignore sudo -E. Put OPENADAPT_* and
PYTHONPATH on the env argv so the netns child still installs
the no-outbound hook, and stop asking sudo to preserve the
whole environment.
@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

From: Grok

Fix for the red flow core critical harness bug at 1ef78c0:

  • _namespace no longer uses sudo -E. It puts every needed variable (OPENADAPT_*, PYTHONPATH, …) on the env KEY=VAL argv so GitHub-hosted runners that ignore -E still install the no-outbound hook in the netns child.
  • CI workflow and the wrapper comment match that rule.
  • New tests/test_egress_namespace_argv.py: 2 passed locally; same tests fail against the pre-fix helper (control).

Untested here: the real Linux OPENADAPT_TEST_NETNS=1 suite (needs the runner). Waiting on CI for that.

Dropping sudo -E shifted PATH= to argv[3], so the root case's
argv[4:4] -u insert landed after the assignment. GNU env then
treated -u as the command (env: '-u': No such file or directory)
and failed critical CI on test_real_namespace_and_invoking_identity[True].
@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

From: Grok

Pushed d59b6e2: critical fail was env: '-u': No such file or directory on test_real_namespace_and_invoking_identity[True]. After dropping sudo -E, PATH= moved to argv[3] so the root case's argv[4:4] -u insert sat after the assignment; GNU env treated -u as the command. Insert is now right after env. Unit argv tests (3) pass locally; netns e2e needs Linux CI.

@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

From: Grok: retrigger CI after d59b6e2 (prior push had no pull_request runs).

@abrichr abrichr closed this Nov 5, 2026
@abrichr abrichr reopened this Nov 5, 2026
No behavior change; retriggers PR CI for d59b6e2.
@abrichr
abrichr marked this pull request as draft November 5, 2026 09:42
@abrichr
abrichr marked this pull request as ready for review November 5, 2026 09:42
@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

From: Grok: #1915 pushes after 1ef78c0 created no check suites. Sibling #1958 carries the same tip for CI. Prefer merging via whichever goes green; do not merge both.

@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

From: Grok

Local Linux verification (Colima privileged python:3.12-bookworm): test_real_namespace_and_invoking_identity[False] and [True] both PASSED in 2.08s on tip 6d7c1f0. The root case no longer hits env: '-u'. PR check suites still not attaching to #1915/#1958; Packages workflow_dispatch 42005355205 is in_progress.

Clears CONFLICTING merge state so pull_request CI can run.
Took main's lock hardening (FIFO/hard-link create_exclusive) for
audit-log.sh, lib-release.sh, and test-update.sh; kept netns env -u fix.
@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

From: Grok

Merged origin/main @ebeaaf0c0 to clear CONFLICTING (GitHub was skipping pull_request workflows). Took main's FIFO/hard-link lock hardening for audit-log.sh, lib-release.sh, and test-update.sh. Netns env -u fix unchanged. Local Colima: identity[False/True] and argv units pass; nonroot setup-failure case expects non-root and fails when the suite runs as root.

@abrichr abrichr added review-needed Needs an independent review from another vendor's agent and removed claimed An agent is working on this labels Nov 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-needed Needs an independent review from another vendor's agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant