[abrichr]

Found while testing #1918 (on-prem offline install).

What this fixes

The on-prem audit log is a hash chain. audit-log.sh serializes writers with a lock, and the release updater (lib-release.sh) uses a similar lock so that two updates or rollbacks can't run at once. Both locks treated a zero exit status from mkdir LOCKDIR as ownership.

That isn't safe on Ubuntu 26.04, which is the image GitHub's runner used for #1918. Its /usr/bin/mkdir is uutils (the Rust coreutils). In the version it ships, when two processes race to create the same directory, the loser's EEXIST is swallowed and both exit 0. A plain mkdir on a directory that already exists still fails, so this only shows up under contention. Others have reported the same behavior (gaia-harness/gaia#3084 measured double success in 236 of 300 two-way races on ubuntu:26.04 with GNU mkdir at 0 of 300; cahenesy/throughline#148). As a result:

• Concurrent audit writers could all "hold" the lock, chain to the same parent and break the chain. That is the "concurrent audit writers preserve one hash chain" failure on on-prem: install, update and roll back the real engine offline in CI #1918's CI, which recurred after the first commit here ("audit chain broken at line 20").
• A second update could proceed while another update or rollback was live.

The change

• Ownership is now the exclusive creation of the pid file inside the lock directory. The pid is written to a fresh staged name and hard-linked into place; link(2) refuses an existing path of any type. Ownership is then read back from the file (regular, not a symlink, holding this pid). Noclobber alone was not enough: bash opens an existing FIFO without O_EXCL. A lock path that exists and is not a regular file is refused at once. The mkdir -p of the lock directory is no longer what grants the lock. The on-disk layout (LOCK/pid) is unchanged, so an existing holder is still recognized.
• Stale locks (owner process gone) are broken only under a second exclusive lock that serializes breakers, and only if the lock still names the same dead owner when re-read. This was the first commit here; it closes a separate race where a breaker could delete a lock a live writer had just taken. The breaker lock is now also an exclusively created file (audit.log.lock-breaker, .update.lock-breaker). The update lock gets the same compare-before-remove rule; previously it did an unserialized rmdir and mkdir.
• A symlinked update lock is refused.

Tests (test-update.sh)

• A mkdir shim that never reports EEXIST models the uutils behavior on any OS.
  • With a live owner holding the update lock, an update under the shim is refused and the owner's lock is untouched.
  • 12 concurrent audit writers under the shim leave an intact 12-record chain.
• A shimmed cat hands the lock from a dead owner to a live process right after the writer reads it; the live writer's lock must survive (first commit).
• When a chain check fails, the test prints the verifier output.
• A FIFO with a reader pre-created at either pid path is refused, nothing is appended, and the FIFO is left alone. At a932d9d: macOS 54/54 three runs, Linux (GNU bash 5.2.37, coreutils 9.7) 56/56 three runs. These five checks fail on 6798c44.

Testing

• macOS: bash deploy/on-prem/bin/test-update.sh, 49 passed, 0 failed (3 runs).
• Linux (Debian, bash 5.2): 51 passed, 0 failed, 3 runs with GNU coreutils and 3 runs with uutils 0.12.0 first in PATH. Some checks are platform-gated, so the totals differ by OS.
• Control: the new test file with the previous audit-log.sh and lib-release.sh fails all 3 new checks. The update under the shim goes through, which also fails 2 later rollback checks (44 passed, 5 failed).
• Stress on Linux pinned to 2 CPUs, 12 concurrent writers per round with the EEXIST shim: the previous code broke the chain in 5 of 5 rounds and this code in 0 of 40.
• A lock left by a dead owner is still reclaimed at once, for both locks, and both lock directories are removed on release.
• shellcheck is clean on the changed files apart from an existing SC1091 info.

Not tested

• I did not run on an actual Ubuntu 26.04 machine. uutils 0.12.0 (the newest release) was atomic in my race test (0 of 200), so the shim stands in for the older uutils on the runner. The real check is CI on ubuntu-26.04. With this commit, the offline job that runs test-update.sh passed on on-prem: install, update and roll back the real engine offline in CI #1918 (run 41940061341), and on the stacked on-prem: build the runner image from a signed release with no network, test it in CI #1923 test-update.sh passed 51 of 51 on two consecutive runs (41940100303, 41941053864). Before this commit that check had failed on 3 of 5 runs.
• No workflow on main runs test-update.sh. on-prem: install, update and roll back the real engine offline in CI #1918 adds the first CI job that does.
• If a breaker is killed while holding the breaker lock, stale locks are no longer broken automatically. Audit writers then wait up to 10s and fail loudly, and updates refuse, instead of corrupting the chain or running concurrently. Removing the leftover .lock-breaker file clears it.

[abrichr]

Independent adversarial review: BLOCK

P2: Refuse an existing nonregular pid path before granting ownership. The noclobber writes in audit-log.sh:83 and lib-release.sh:352 succeed when pid is a FIFO with a reader. In isolated temporary lock directories, two actual update-lock callers both reported ownership while alive. Two actual audit writers also returned 0 and appended records with the same empty prev_sha, breaking the chain. A date shim held their append interleaving; the update-lock case needed no shim. The kernel O_CREAT|O_EXCL reference correctly refused the identical FIFO.

The later -f checks do not protect the successful acquisition path. Use an atomic creation primitive that refuses any existing path, and refuse invalid lock-file types. These cases pre-create a FIFO; they do not show that normal operation creates one or that a caller can modify root-owned release paths.

Validation on WSL Bash 5.2.21: the exact advertised shell suite passed 51/51. The frozen 13-case probe produced 2 failures/11 passes on head 6798c4470, 3 failures/10 passes on literal parent 7cf115068, and 4 failures/9 passes on main 800f45004, with no errors or skips. Head closes the parent/main fault-model failures; the FIFO failures occur on head. Regular live-owner/dead-owner controls and five native stress rounds of twelve audit writes passed on every reference. Legacy FIFO callers did not enter within the 1.5-second bound; they blocked, so those passes are not healthy-refusal proof. Bash syntax and release consistency passed; shellcheck was unavailable. GNU mkdir 9.4 had no double success in 20 unsynchronized launch pairs per reference. The mkdir/cat shims are fault models; I did not reproduce an Ubuntu 26.04/uutils defect or run release activation in the lock probes. Source hashes were stable and the live head was unchanged.

[abrichr]

Answering BLOCK 6648159753 at a932d9d.

P2, an existing nonregular pid path: both lock helpers (create_exclusive in audit-log.sh and _rel_create_exclusive in lib-release.sh) no longer write the lock path with noclobber. They refuse any existing path up front. Then they write the pid to a fresh staged name, hard-link it into place with ln, and remove the staged name. link(2) fails on an existing path of any type. Ownership is then read back from the linked file (it must be a regular, non-symlink file holding this pid), not taken from ln's exit status. The breaker locks use the same helper. When the lock path exists and isn't a regular file, both callers now refuse right away with an error ("is not a regular file; refusing") and leave the path alone. Before, they waited.

New checks in test-update.sh: a FIFO with a reader pre-created at the pid path. The update is refused and the FIFO is left in place. The audit writer exits nonzero, appends nothing, and leaves the FIFO in place. Against 6798c44 the five new checks fail. The update also goes through there, which fails 2 later rollback checks (47 passed, 7 failed). With the fix: macOS bash 54/54 three times, and Linux (GNU bash 5.2.37, coreutils 9.7) 56/56 three times. The mkdir-shim and 12-writer chain checks still pass in each run. shellcheck shows only the existing SC1091 info. Not tested: an actual Ubuntu 26.04/uutils host, and uutils ln. The read-back is there so that a misreporting ln can't grant ownership either.

[abrichr]

Independent adversarial review: APPROVE

Re-reviewed a932d9d91d04392bf56df3787e8a1750f85af703 against literal parent 6798c44708930c8bf1b3a84a0b7fa7f14e3f25bd. The previous FIFO-lock BLOCK is resolved: audit-log.sh:94 and lib-release.sh:361 hard-link a staged pid file and check ownership, with existing nonregular pid paths refused.

Fresh gated Linux validation: all 13 original independent cases and four additional FIFO-refusal cases passed, with zero errors or skips. The four update/audit cases, with and without a FIFO reader, finish nonzero within the frozen three-second bound, preserve the FIFO inode/type and append nothing. The exact parent’s retained 2-failure/11-pass result is reused, not rerun: two live update callers entered, and two audit writers appended a broken chain under the date scheduling shim. On this head those original cases admit no owner or audit append.

Live-owner exclusion, dead-owner recovery, handoff controls and five native rounds of 12 audit writers also pass, preserving all 60 unique jobs and their chains. The unchanged author shell suite passes 56 checks, including its two GPG checks. Bash syntax and release consistency pass; shellcheck was unavailable. Fifteen head-source and six frozen-input hashes match before and after; both detached worktrees remain clean.

I read the actual lint, source-boundary and critical logs: configured/strict mypy pass, source tests pass 73, and critical suites pass 52 plus 183 with named golden VERIFIED and zero-model positives. All five associated workflows succeeded at synthetic merge 77b63dd74ba56416bf70aacd4815c025c12efdce, this head into 81f3b05388ae77ab8e23af065f0fc7da1e717844. That CI has no on-prem offline job.

Limits: native binaries were GNU coreutils 9.4 and Bash 5.2.21; the mkdir/cat/date shims are fault or scheduling models, with no native Ubuntu 26.04/uutils race reproduction. Fixtures pre-create owned FIFO paths. The author’s signed-release fixture uses placeholder wheels, a build hook and disposable keys, so it does not validate an actual package installation. No fresh parent/main or full Flow rerun, real deployment, privilege-escalation or egress-isolation claim.