Repository navigation
ci: gate releases on an approved environment and sign, attest and SBOM release images - #1985
Conversation
Verify that a release tag points at a commit reachable from develop or a release/* branch, and expose the version without the leading v. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Branch builds from the build-push workflow now go to ghcr.io/<owner>/<repo>-dev instead of the production package, so every tag and manifest list from this workflow lands in the development package. Production images are published by the release workflow only. Co-Authored-By: Claude Haiku 5.5 <[email protected]>
Checks the tag ruleset, release environment, branch rules, teams and GHCR package settings through read-only GitHub API calls. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Split the release workflow into preflight, approve, build and publish jobs: - preflight checks that the ref is a release tag whose commit is on develop or a release/* branch, and that the version is not already published - approve waits on the protected release environment before anything is pushed to the registry - build publishes only exact version tags (no moving X.Y or latest tags) and adds an SBOM and mode=max provenance to every image - publish creates the manifest lists, signs them with cosign keyless and pushes a GitHub build-provenance attestation for each The production manifest list now contains only the production platform images; previously it also picked up the -dev images. Runs for the same tag no longer cancel each other. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Document the published image tags, how to verify signatures and build provenance, and the repository settings that protect releases. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
|
Claude finished @arnaubennassar's task in 1m 25s —— View job Review of #1985: release gating, signing and attestationThe overall design is sound. The I read the diff and the scripts but did not run anything, so none of the findings below were run. I could not check that the new action SHAs match their tags ( 🔴 Required
🟡 Suggested
🟢 Optional
Template summary🚀 What's NewRelease publishing is now gated: 🐛 Bug FixesThe production manifest list no longer includes 📋 Config UpdatesNone. No aggkit configuration changes. Admins must configure the GitHub settings listed in the runbook (teams,
|
…ase tags Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…tors Check the release environment for administrator bypass, all release/* branches (paginated) and bypass actors on the branch rulesets. State that only rulesets are read and that organization admins may bypass the tag ruleset (audited). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ements Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…nto ci/release-gating-attestation
|
Thanks for the review. I checked the action SHAs against upstream ( Required
Suggested Optional
|
The preflight job pinned actions/checkout to v5.1.0 while the build job uses v7.0.1. Pin preflight to the same commit so both jobs run the same checkout action. Co-Authored-By: Claude Haiku 5.5 <[email protected]>
joanestebanr
left a comment
There was a problem hiding this comment.
Suggestion: move approve after build, staging the images in aggkit-dev and promoting by digest
Right now approve sits before build because build pushes the per-platform images by digest to ghcr.io/agglayer/aggkit. That works, but the reviewer approves without seeing what was built, and a release that fails to build still consumes an approval.
What about this flow instead?
preflight → build (push by digest to aggkit-dev) → approve → publish (promote to aggkit + sign + attest)
build pushes the platform images to ghcr.io/agglayer/aggkit-dev instead of aggkit. The production package is not touched before approval, so the guarantee “nothing reaches aggkit without a reviewer” still holds.
approve reviews something concrete: the image digests, the “Verify version in image” result, the SBOM and provenance.
publish copies the approved digests from aggkit-dev to aggkit (e.g. docker buildx imagetools create -t @sha256:...), then signs and attests as it does today. The digest is the same, so what was reviewed is exactly what gets published.
…er approval The release workflow now runs preflight, build, approve and publish. The build job pushes the per-platform images by digest to the staging package ghcr.io/agglayer/aggkit-dev, so the reviewer approves images that were actually built and the production package is untouched before approval. After approval, publish creates the tagged manifest lists in ghcr.io/agglayer/aggkit from the staged digests of the same run, then signs and attests them. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
|
Thanks @joanestebanr, adopted.
Things worth a look:
I also updated |
joanestebanr
left a comment
There was a problem hiding this comment.
Checked 89bc85b: the flow is now preflight → build → approve → publish. build pushes by digest to aggkit-dev only, approve runs after the build and lists the staged digests, and publish promotes those exact digests from the run's artifacts to aggkit, then signs and attests them. LGTM, thanks!
Non-blocking: staged release digests are untagged versions in aggkit-dev. If a cleanup policy for untagged versions is ever added to that package, it should leave a wide enough window to cover the approval wait.
The publish job is the only job that uses the release environment and it logs in to ghcr.io/agglayer/aggkit with a credential stored only as secrets of that environment. The workflow token no longer has write access to the production package, and the staging and signing steps keep working with the same login. The settings check reports the environment secrets, and the runbook describes how to create the credential and restrict the package.
|
Summary of what changed since @joanestebanr's review (09:31 UTC, 19 Nov): 1. Release images are staged, then approved, then promoted (89bc85b)
2. The production push credential is held only by the
3. Docs and merges Still to do outside the PR: set the two secrets in the |
🔄 Changes Summary
Addresses the release-path finding in agglayer/issues#112: today any
v*.*.*tag push publishes production images toghcr.io/agglayer/aggkitwith no review, a movingX.Ytag that a later tag silently overwrites, and no signature, SBOM or verifiable provenance. After this change no single account can publish a production image.release.ymlis nowpreflight->build->approve->publish.preflight(read-only) runs the newscripts/release/check_tag_ancestry.sh. The ref must bevX.Y.ZorvX.Y.Z-<pre>, and the tagged commit must be an ancestor oforigin/developor anorigin/release/*branch. It also refuses to continue if the version is already published inghcr.io/agglayer/aggkit, so versions are immutable.buildproduces each platform image withsbom: trueandprovenance: mode=max, checks the version inside the image, and pushes it by digest to the staging packageghcr.io/agglayer/aggkit-dev, not toaggkit. The job summaries list the staged digests. The digest names go to an artifact.approveuses thereleaseenvironment (required reviewers, prevent self-review). The reviewer approves images that were actually built; after the approval the job prints the tag, commit, version and the staged digests.publishruns after the approval. It creates the tagged manifest lists inghcr.io/agglayer/aggkitfrom the staged digests of the same run (taken from the build artifacts, not from a tag lookup; it checks the digest format and that exactly one image per platform is present), signs each manifest list with cosign (keyless) and adds a GitHub build-provenance attestation pushed to the registry. It is the only job withid-token: writeandattestations: write. Top-levelpermissions: {}, no job hascontents: write, and the new actions are pinned by commit SHA.ghcr.io/agglayer/aggkitbefore the approval.buildhaspackages: writebecause it pushes toaggkit-dev;GITHUB_TOKENpermissions are not scoped per package, so what keepsbuildout ofaggkitis what its steps do. The runbook documents this and an optional hardening (an environment secret for pushing toaggkit).imagetools createflattens the staged per-platform indexes into the production manifest list, so the SBOM and provenance attestation manifests of the platform images are carried over; blobs are copied (or mounted) between the two packages of the same registry.ghcr.io/agglayer/aggkit-dev(build-push-docker-image.yml). Production images are only published byrelease.yml, which also stages its release candidates inaggkit-devbefore the approval. Tag naming of branch builds is unchanged. No workflow in the repo consumes the oldghcr.io/agglayer/aggkit:<branch>_...tags.X.Y.Z[-pre],X.Y.Z[-pre]-no_shellandX.Y.Z[-pre]-dev. The movingX.Y,X.Y-no_shellandX.Y-devtags are no longer published, and there is nolatest. Branch and PR tag types are removed from the release metadata. Deployments should pin an exact version or, better, a digest.-devplatform images, so the production manifest list could include dev images.publishnow selects only the platform images of its own variant.scripts/release/check_release_settings.sh: a read-only (GET only) audit of the GitHub controls the release path relies on (tag ruleset,releaseenvironment, branch protection, teams, package ACL). It printsPASS/FAIL/MANUALper control and exits non-zero on anyFAIL(--jsonis supported).docs/release_verification.md(operators:cosign verify,gh attestation verify, SBOM inspection, pin by digest) anddocs/release_admin_runbook.md(admins), linked fromdocs/SUMMARY.mdanddocs/release_lifecycle.md. Images published before this change are unsigned.X.Y,X.Y-no_shellandX.Y-devimage tags are no longer published. Consumers following them must pin an exactX.Y.Zversion or a digest.📋 Config Updates
✅ Testing
bash scripts/release/check_tag_ancestry_test.shcovers a commit on develop, onrelease/0.11only, on an unrelated branch, a malformed tag and a non-tag ref.bash scripts/release/check_release_settings_test.shuses a stubghand covers all-pass, a missing tag ruleset, an environment without prevent-self-review, and the package ACL reported as manual.shellcheck scripts/release/*.shis clean.actionlinton both workflows: see the note below.approvewith the digests in the build summaries and nothing inaggkit; the author cannot approve; a tag on an unreviewed branch fails inpreflight; after approvalpublishpromotes the staged digests andcosign verifysucceeds). Not verified without a real run: the cross-package copy of the staged images withdocker buildx imagetools create(it needs the repository to have Actions write onaggkit-dev).actionlint note:
arm-runner-2204(andamd-runner-2204) are self-hosted runner labels that actionlint does not know by default. This is already the case ondevelop. With those labels declared in an actionlint config,release.ymlis clean. The remaining shellcheck info/style findings inbuild-push-docker-image.ymlare pre-existing and untouched by this PR.🐞 Issues
📝 Notes
Settings that must be applied by admins before the next release (step-by-step in
docs/release_admin_runbook.md; verify withscripts/release/check_release_settings.sh). Without them the gates in this PR are not enforced:release-approversandrelease-manager.release: required reviewersrelease-approvers, prevent self-review, deployment policy limited to tagsv*, no admin bypass.refs/tags/v*(creation, update, deletion and non-fast-forward restricted; bypass onlyrelease-manager; organization admins may also bypass, and the audit log records it).developandrelease/*require a PR with at least one approval, no direct pushes, force-pushes or deletion.aggkit: disable inherited repository access and grant Actions write only toagglayer/aggkit. GHCR packageaggkit-dev: must exist and grant Actions write toagglayer/aggkitbefore the first release, because release builds stage their images there; do not enable cleanup of untagged versions, and keep write access to it tight.Tag workflows run the workflow file from the tagged commit, so the ancestry check is only a guarantee together with items 3 and 4. Residual risks (colluding maintainers, other workflows with
packages: write) and an optional further hardening are in the runbook.