[wchilders-nvidia]

Adds acknowledg-forge, a service that keeps an AcknowlEDG review alive for
each open GitHub or GitLab pull request:

• Test - Changes: rematerialized expectation diffs;
• Test - Failures: from the PR's latest CI run;
• Bench - C / Bench - C++: compared with the checked-in baselines.

It can also comment on the PR when results change. The PR ships the service
as a public container image and deploys it from CI.

Initial drafts for GitHub and GitLab support are both included, primarily to ensure that the core logic is sufficiently abstract that this can conceptually be used with different software forges to allow downstream repositories (of various former EDG customers -- that may be in a variety of different software forges) to also leverage the software for their internal processing.

The code primarily authored by Claude and is at this point very lightly reviewed (mostly at the system configuration level). I'll be making my way through it myself, but wanted to get this up so others can start getting eyes on it.

This will be an important service for us as we start to try and process some of the larger PRs that have broader testing impact (e.g., #201).

Stacked on #332, whose commit is the first one here. This PR's change is
the second commit, Initial implementation of AcknowlEDG Forge, which is all
this description covers.

The second commit adds ~15,250 lines:

Area	Lines	Weight
Existing code modified (edgtest, edg-rematerialize-test-run, edgdocker, AcknowlEDG deploy/packaging)	~110	Highest
Workflows	~220	High
Container image, compose.yaml, deploy helper, host runtime script	~710	High (deploy-remote.py runs as root)
Isolation: runners, publisher server and process pool, sandbox.py, local.py	~2,400	High
Core: dev_tools/pylibs/edgacknowledg/forge/ (the rest)	~3,600	High
Forge adapters (GitHub, GitLab) and dev scripts	~940	Medium
Tests	~4,300	Medium
Docs (README.md, SANDBOXING.md)	~1,070	Medium
uv.lock files	~1,900	Generated; run uv lock --check

Read in this order:

1. Changes to existing code (below).
2. dev_tools/services/acknowledg-forge/README.md, the design overview and
   deployment, then SANDBOXING.md, the security model (with diagrams).
3. compose.yaml: the four containers, what each mounts, and their networks.
4. forge/__init__.py, the protocols.
5. The processing path: ingest → db/store → processor →
   publisher/client → publisher/server → publisher/pool → announcer.
6. The isolation code: runner/ (roles, server, client), sandbox.py,
   and the deploy helper and host script (deploy-remote.py,
   prepare-runtime.sh).

Changes to existing code

• edgtest:
  • .run_test_env is no longer sourced. Sourcing it was arbitrary code
    execution for untrusted PRs. No such files are tracked here.
    Does any out-of-tree test home use one?
  • Reading a run archive (.rt.tar.gz) refuses link members instead of
    following them.
  • Copying a test tolerates a test without recordings.
• edg-rematerialize-test-run: handles new tests and their
  recordings, takes configurations from recording names, reports
  NO PREVIOUS OUTPUT as edgy does, and writes changes.elist (so Test -
  Changes is served).
• edgdocker registers the image.
• AcknowlEDG: the server wheel excludes edgacknowledg.forge;
  forge-only changes don't trigger acknowledg-deploy.yml; the README links
  the forge.

Security boundaries

A deployment is four Docker containers from one image (compose.yaml); see
SANDBOXING.md for the full model.

Container	Runs	Sees	Networks
service	the service: forge token, webhook secret, database, clones	all state	the forge; webhooks; build; publish
builder	a runner server: edg-rematerialize-test-run only	review trees read-only, writes only build-staging/	build (internal)
publisher	the publisher server, answering reviewers in a pool of processes	review trees read-only	publish, tools (internal); AcknowlEDG
tools	a runner server: ifc-dump, cg_annotate only	review trees read-only	tools (internal)

• Containers: non-root, no capabilities, no-new-privileges,
  read-only root file systems, a bounded tmpfs /tmp, memory and process
  limits, Docker's default seccomp profile (no user namespaces). Only the
  webhook port is published, and only on the host's 127.0.0.1, for a
  reverse proxy.

• Networks pair each client with its server and nothing else; internal
  networks have no route out. Connections carry no credentials of their own,
  so the networks are the boundary: test_containers.py checks every pairing.

• Runners (runner/): each role's allowed commands, roots, and
  variables are fixed in code (runner/roles.py), the same with or without
  containers. Absolute paths outside the roots are refused; the environment is
  cleared; a command's process group is killed when it ends, times out, is
  cancelled, or its connection goes; between commands, stray processes are
  killed and /tmp emptied.

• Builds write only to build-staging/; the service checks what was
  written (no links, no hard links) before renaming it into the review.

• Publisher: the server only manages sections. Every reviewer request
  is answered in a pool process (round-robin, 4 by default) that builds the
  section's handler, answers, and drops it. A process that misses the
  request's deadline (150 s) is killed with whatever it started and replaced,
  as is one that exits; the reviewer gets the client library's failure reply.
  The AcknowlEDG API key reaches it over its connection, never a file or the
  environment.

• workspace.py:

  • artifact extraction rejects escaping paths and links, and caps the
    decompressed size (including inner .rt.tar.gz files);
  • no writes through symlinks;
  • git runs with no hooks, and credentials never go in argv.

• Webhooks:

  • signatures checked in constant time;
  • payloads only identify what changed, and state is always re-fetched;
  • unconfigured repositories are ignored, and the queue is deduplicated.

• http.py: artifact downloads that redirect to storage don't send
  credentials. They're size-capped.

• Comments contain no PR-supplied text.

• Host: systemd decrypts the credentials; prepare-runtime.sh (root,
  ExecStartPre) copies them to /run for the service container only.

• deploy-remote.py runs as root, through sudo, and is installed
  by an admin:

  • it accepts only ghcr.io/edgcpp/acknowledg-forge@sha256:…, path-free
    ws(s)/http(s) URLs, and acknowledg-forge-* service names;
  • it writes only the root-owned <service>.deploy.env, pulls the image,
    restarts the unit, and removes only that repository's unused images.

  What can someone holding the deploy key do? Intended answer: deploy
  another image published from this repository.

• The publish job has no access to environment secrets, and the image
  contains no credentials or configuration.

Deliberate trade-offs (documented in SANDBOXING.md):

• Docker runs as root (the containers don't): a kernel or runtime escape
  reaches the host as root.
• The runner containers are long-lived, not one per job: a PR that
  compromises one could affect later commands in it (mitigated by the stray
  cleanup, read-only file systems, no network, and the service's checks).

Core logic

• Store:
  • a generation counter marks PRs dirty;
  • a base-branch push alone isn't material, so PRs aren't all reprocessed on
    every push to main;
  • work completed for an older generation doesn't clear a newer change;
  • retries back off, and the service gives up after 8 attempts until the PR
    changes.
• Processor:
  • Changes: keyed by (merge-base, head).
  • Failures: from the latest completed CI attempt, kept while a new run
    is in progress, dropped when CI passes.
  • Run directories are swapped in by rename.
  • Closing a PR removes its workspace; the clone is deleted only when no
    open PR needs it.
• Publisher: one long-lived connection from the service
  (PublisherClient) to the publisher server, which gives each connection its
  own AcknowledgPublisher (one batch connection to AcknowlEDG, read-only
  sections, dropped sections re-added). If the connection goes, or a request
  goes unanswered for 15 minutes, the client reconnects and serves every
  review again.
• Announcer:
  • comments only when a PR is settled and its content fingerprint changed;
  • nothing on restart, close, or reopen;
  • a fresh database adopts existing PRs silently;
  • posting is idempotent, using a marker in each comment.
• Database: portable types and an Alembic migration that matches the
  models.
• Adapters: GitHub maps pull request state, run_attempt, and
  artifacts. GitLab maps per-job artifacts and URL-encoded project paths, and
  is tested only against mocked APIs.
• Without containers (ACKNOWLEDG_FORGE_UNSANDBOXED=1, local
  development only): the service runs the runner and publisher servers itself
  on 127.0.0.1, so the control flow is the same but nothing is confined (for
  single-user machines).

Verification

Verified (locally):

• Tests (122):
  • the PR lifecycle against a fake forge, with builds going through a real
    builder runner server;
  • the publisher client, server, and process pool against a real AcknowlEDG
    server, including reconnecting and replaying;
  • the pool: deadlines, replacing killed or crashed processes, failed starts;
  • runner policies, cancellation, and losing the connection;
  • archive and symlink attacks, webhook signatures, idempotent comments, and
    the deploy helper's argument checks.
• The containers (13 tests, on the built image, Docker 29.9 with Compose
  5.6): each container's user, capabilities, mounts, and the networks it can
  reach; the webhook port on loopback only; builds and tools reached through
  their runners; the publisher running its server and 4 pool processes.
• The service itself, started both in the containers and without them,
  against an unreachable forge: it starts, and reaches its runners and the
  publisher.
• Tooling: strict mypy (40 files).

Not verified:

• real GitHub or GitLab APIs, a full end-to-end run with a real forge and
  real artifacts through the containers, and the workflows running on GitHub
  (including the new container job on ubuntu-24.04 and ubuntu-26.04);
• the host procedure on a real server: the systemd unit running Compose,
  prepare-runtime.sh as root, encrypted credentials, deploy-remote.py
  against a real Docker, and PostgreSQL (the suite passed on PostgreSQL 17
  before the container rework; not re-run since).

Known limitations

• Tooling changes need a new image, because the tooling is frozen into
  it. The deploy workflow rebuilds it on any dev_tools/bin or
  dev_tools/pylibs change.
• Crafted inputs can make some reviewer requests slow (e.g. a regex that
  backtracks on long .elog lines, uncapped command output). The process pool
  bounds each request to its deadline, so this can't stall the service; fixes
  for the inputs themselves will follow separately.

[wchilders-nvidia]

Here's a settled storage breakdown based on current PRs:

4.0K    ./build-staging
364M    ./repos/github
364M    ./repos
2.8M    ./reviews/ac3df272-142b-40de-9cce-0750d86e0ea6
1.1M    ./reviews/826b7b4b-6710-4e60-871d-046b31626a86
2.1M    ./reviews/96b2f0a2-0e79-4d2b-80ab-8409cff9e97a
5.7M    ./reviews/d00cf678-27f2-4f95-861b-0c55c5e5a23c
1.2M    ./reviews/6e73b169-d7da-4e6a-8c17-887f07b3f72d
6.8M    ./reviews/2494ce4a-09f7-42c4-8973-32be2a1b9a76
3.1M    ./reviews/7bb9030d-62c9-4cc5-bebb-947959441d68
48M     ./reviews/80533251-ea46-418c-8906-8a75d6a6d495
6.7M    ./reviews/a44f3a18-f7d9-4a3d-a85f-cdefe14d6106
3.4M    ./reviews/d93905b7-3559-4643-9908-6a234a813c0a
3.4M    ./reviews/d2faccca-29d5-4f07-accd-b012f6f7f6f4
6.8M    ./reviews/cf4e5dae-27d5-4854-b119-5e24fb04ccf4
5.7M    ./reviews/8fda0cdc-2241-44b3-8404-aedc1e7b6f00
32M     ./reviews/0e4ea431-a4cf-4db1-95cc-3b2a8b9736ad
5.7M    ./reviews/275a9c02-baad-4604-8c62-d4a508f0cf6b
15M     ./reviews/4c5515bc-1102-42d5-bd1e-1df657ff2dec
2.8M    ./reviews/7378e54f-0b2f-4dae-86ce-3b30b72b7ec8
150M    ./reviews
515M    .

and a high level overview of the directory structure (tree -L 4):

.
├── build-staging
├── repos
│   └── github
│       └── edgcpp%2Fcompiler.git
│           ├── branches
│           ├── config
│           ├── config.worktree
│           ├── description
│           ├── FETCH_HEAD
│           ├── HEAD
│           ├── hooks
│           ├── info
│           ├── objects
│           ├── packed-refs
│           ├── refs
│           └── worktrees
├── reviews
│   ├── 0e4ea431-a4cf-4db1-95cc-3b2a8b9736ad
│   │   └── base
│   │       └── tests
│   ├── 2494ce4a-09f7-42c4-8973-32be2a1b9a76
│   │   ├── base
│   │   │   └── tests
│   │   ├── head
│   │   │   ├── benchmarks
│   │   │   └── tests
│   │   └── staging
│   ├── 275a9c02-baad-4604-8c62-d4a508f0cf6b
│   │   ├── head
│   │   │   ├── benchmarks
│   │   │   └── tests
│   │   └── staging
│   ├── 4c5515bc-1102-42d5-bd1e-1df657ff2dec
│   │   └── base
│   │       └── tests
│   ├── 6e73b169-d7da-4e6a-8c17-887f07b3f72d
│   │   └── base
│   │       └── tests
│   ├── 7378e54f-0b2f-4dae-86ce-3b30b72b7ec8
│   │   └── base
│   │       └── tests
│   ├── 7bb9030d-62c9-4cc5-bebb-947959441d68
│   │   └── base
│   │       └── tests
│   ├── 80533251-ea46-418c-8906-8a75d6a6d495
│   │   ├── base
│   │   │   └── tests
│   │   ├── head
│   │   │   ├── benchmarks
│   │   │   └── tests
│   │   └── staging
│   ├── 826b7b4b-6710-4e60-871d-046b31626a86
│   │   └── base
│   │       └── tests
│   ├── 8fda0cdc-2241-44b3-8404-aedc1e7b6f00
│   │   ├── head
│   │   │   ├── benchmarks
│   │   │   └── tests
│   │   └── staging
│   ├── 96b2f0a2-0e79-4d2b-80ab-8409cff9e97a
│   │   └── base
│   │       └── tests
│   ├── a44f3a18-f7d9-4a3d-a85f-cdefe14d6106
│   │   ├── base
│   │   │   └── tests
│   │   ├── head
│   │   │   ├── benchmarks
│   │   │   └── tests
│   │   └── staging
│   ├── ac3df272-142b-40de-9cce-0750d86e0ea6
│   │   └── base
│   │       └── tests
│   ├── cf4e5dae-27d5-4854-b119-5e24fb04ccf4
│   │   ├── base
│   │   │   └── tests
│   │   ├── head
│   │   │   ├── benchmarks
│   │   │   └── tests
│   │   └── staging
│   ├── d00cf678-27f2-4f95-861b-0c55c5e5a23c
│   │   ├── head
│   │   │   ├── benchmarks
│   │   │   └── tests
│   │   └── staging
│   ├── d2faccca-29d5-4f07-accd-b012f6f7f6f4
│   │   └── base
│   │       └── tests
│   └── d93905b7-3559-4643-9908-6a234a813c0a
│       └── base
│           └── tests
├── state.db
├── state.db-shm
└── state.db-wal

85 directories, 9 files