[wadewilliams]

Why

During a rolling deploy, old and new containers overlap. That's fine for web servers. Queue workers and schedulers are different: the old worker keeps running old task code through its drain, which can last up to an hour, after the new web container has already migrated the database. These services should drain first and start last.

Releases also deploy by version tag (v0.9.2), not only by commit SHA. Every consumer of the tag has to handle that: git fetch and checkout, image pull, restore, status. And a restore has to put back what was actually running, not what git HEAD or :latest says.

What

deploy.strategy: drain-first is a new per-service label. rolling remains the default. A deploy runs in three phases:

1. Drain. Drain-first services stop in reverse deploy.order. Each pulls its new image first, so a bad tag fails before anything stops. Then each runs docker stop --time <deploy.drain>: SIGTERM (Celery's warm shutdown), then SIGKILL when the drain runs out. Nothing rolls until every drain has finished.
2. Roll. Rolling services deploy in deploy.order, as before.
3. Start. Drain-first services start in deploy.order, at the replica count they had before, and every replica must pass its health check.

With web (100), api (200) and worker (300), the worker drains first and starts last, after the services it depends on.

This is stop-before-start for singletons such as Celery Beat, so old and new never overlap. Closes #26, as a strategy on deploy.role: app rather than the new role the issue proposed.

Image tags: versions, SHAs, and exact restores

• Git: git fetch --tags --force origin, so a new release tag is always present and a re-pushed tag moves. checkout --detach handles lightweight tags, annotated tags and SHAs.

• Images: compose config is rendered with the deploy's DEPLOY_TAG. A warning is printed when an app image doesn't end in :<tag> (e.g. DOCKER_IMAGE set without :${DEPLOY_TAG:-latest}).

• What's running is read from the containers, and named by git. Before changing anything, the deploy records each app service's running image ID (the most common one wins, with a warning if they differ) and the tag it was started from. A revert uses that tag when all of these hold:

  • it's a git ref (a release tag or a SHA)
  • it still names the running image locally
  • it isn't the tag being deployed (this deploy pulls that, which would move a re-pushed tag)

  Otherwise (:latest, a moved tag, a digest, no tag) it uses the full SHA of the commit checked out before the deploy, which CI must push every image under. No tags are created and nothing is written on the host.

• Known limit: redeploying the commit that's already running leaves no other git name for the running build. If its image was rebuilt under the same tag, a revert restores the rebuild. The deploy warns up front. This is not a regression: master didn't roll shipped services back at all.

• The deploy stops before changing anything when:

  • Docker can't say what's running
  • another compose project runs this deploy's services in this directory. A release that renamed the project would otherwise start a second stack beside the first. An unrelated project that merely shares the directory is fine.

• What's deployed: .git/deploy-tag (inside .git, so the tree stays clean) records the tag on every successful deploy of all app services. After a revert it's rewritten to the restore tag. A --service deploy that leaves some app services out clears it, since services then run different tags.

The manual-ops hazard

With image: repo:${DEPLOY_TAG:-latest}, a person running docker compose up -d on the host silently moves every service to :latest. The design: flow-deploy knows what's deployed and lends it to manual commands.

• flow-deploy tag prints the deployed tag. It comes from the running app containers when they agree, else from the record when nothing runs. Mixed tags, untagged images, or nothing to go on give an error ("set DEPLOY_TAG"), never a guess.
• flow-deploy compose <args> runs the compose wrapper with that DEPLOY_TAG. An explicit DEPLOY_TAG wins. It refuses rather than fall back to :latest.
• A wrapper such as script/prod can default it, which makes script/prod up -d safe too: if [ -z "${DEPLOY_TAG:-}" ] && command -v flow-deploy >/dev/null; then DEPLOY_TAG=$(flow-deploy tag 2>/dev/null) && export DEPLOY_TAG; fi. flow-deploy always sets DEPLOY_TAG when it calls the wrapper, to a placeholder while looking up the tag, so this never recurses.
• flow-deploy status shows the deployed tag and where it came from.

Why this design: the containers are the only source that can't drift from reality. The record covers the "everything is stopped" case. Refusing on ambiguity is the safe failure, because a wrong guess recreates production on the wrong image.

A failed deploy leaves no trace

Any failure puts every app service back on the exact image it ran before the deploy. That covers a failed health check, a pull or compose error, an unexpected exception, or a signal: SIGTERM, Ctrl-C, or SIGHUP.

• The rolling service that failed keeps its old container, as before.
• Rolling services that already shipped are rolled back (new), in reverse order, by a rolling deploy onto their restore tag with no pull (compose pulls it only if it's missing). If a previous-image container survived its cutover (a cleanup failure), the new containers are removed instead. Stopped leftovers are removed first, since compose up --scale would restart them.
• Every drain-first service comes back on its previous image at its replica count, in deploy.order. That includes one already healthy on the new tag. New-image containers are drained first. A service that wasn't running before stays stopped.
• The repo goes back to the previous commit.
• The lock is always released. Keeping it was never a guard: acquire() breaks any lock whose PID has exited.

Further signals are ignored while the revert runs, and logging survives a closed stdout or stderr. A signal that arrives while a drain-first container is stopping is held until that container has stopped and been removed, and then the deploy reverts. The drain's docker stop runs in its own session, so a terminal's Ctrl-C can't cut it short.

A cancelled CI job does not revert a remote deploy. deploy-action runs ssh with no pty. tests/integration/test_ci_cancel.py shows against a real sshd that the remote process receives no signal and runs to completion (or reverts on its own failure). With ssh -tt it would receive SIGHUP.

Exit codes: 0 success, 1 failed and fully reverted, 2 locked, and new 3 failed and not fully reverted. Exit 3 means a git restore, rollback, restore, or a failed service's own cleanup failed. Someone needs to look.

Rolling deploys: master's behavior, plus listed changes

tests/test_rolling_parity.py replays 15 rolling scenarios on a simulated Docker host (tests/fake_host.py) against tests/fixtures/rolling_master.json, recorded by running the same scenarios on master 0.3.0. Every docker, compose and git command, its env, the exit code, and the lock state must match once the listed departures are applied. Deliberate changes, each pinned by that test:

• Pre-deploy checks: the snapshot (running image, plus restore tag via git rev-parse --verify and docker image inspect), and the project-name check.
• git fetch --tags --force, and compose config rendered with DEPLOY_TAG.
• A later service's failure rolls back services that already shipped. Previously they stayed on the new tag beside services on the old one.
• Fixes a master bug: a failed health check no longer replaces the old container. Scale-back passes --no-recreate. On master it recreated the surviving old container on the new, failing image (found by the integration suite).
• Scale-back is skipped when the container it should leave alone couldn't be stopped and removed. With two left, it removes one and might pick the wrong one. The cutover still succeeds with a warning, as on master.
• A failed git restore exits 3 and releases the lock.
• A signal restores the repo before exiting.

Operator-facing additions

• Killed containers are reported. After a drain or a cutover stop, a container that exited 137 or used the whole drain timeout gets a warning: its PID 1 isn't handling SIGTERM. The deploy still succeeds.
• Host-side run log. Every deploy is recorded in .git/flow-deploy.log (header, every line, exit code; last 20 runs; inside .git, so the tree stays clean). It's written only while the deploy lock is held. status shows how the last run ended, including "no exit recorded" for one that was killed.
• The record holds the deployed commit. .git/deploy-tag now records the SHA as well as the tag.
• A revert waits for what it restored. Restored services must reach their previous replica count and pass their health check. If they don't, the deploy exits 3.
• Discovery doesn't need the app's env. If compose config fails in CI (no .env), flow-deploy config retries with --no-interpolate and warns on stderr, naming the unset variables. Stdout stays pure JSON. It always sets DEPLOY_TAG to a placeholder, so a wrapper that defaults it never asks.
• Atomic deploy lock. Acquirers take an flock on .git/deploy-lock.guard while they check and write the lock, closing a check-then-write race (pre-existing) that let two deploys starting together both proceed.

Upgrading

Projects already on flow-deploy keep working without changes. Rolling-only behavior differs only as listed above. To use version-tag deploys and get exact reverts:

1. Push every image under its full commit SHA, plus the version tag for releases. A revert falls back to the previous commit's full SHA whenever the running tag can't be trusted (:latest, a moved tag, a digest, no tag). :latest is optional; flow-deploy never relies on it.

   tags: |
     ${{ env.IMAGE }}:${{ github.sha }}
     ${{ steps.release.outputs.tag && format('{0}:{1}', env.IMAGE, steps.release.outputs.tag) || '' }}

2. Deploy by the version tag, with the SHA as the fallback. The git tag must be pushed before the deploy runs:

   - uses: flowcanon/deploy-action@master
     with:
       tag: ${{ steps.release.outputs.tag || github.sha }}

3. Name every app image with DEPLOY_TAG, directly or through .env: DOCKER_IMAGE=ghcr.io/org/app:${DEPLOY_TAG:-latest}. A bare image name always runs :latest, and the deploy warns about it.
4. Make manual compose commands keep the deployed version. Either use flow-deploy compose up -d, or have the compose wrapper (script/prod) default DEPLOY_TAG and fail closed:

   if [ -z "${DEPLOY_TAG:-}" ]; then
     DEPLOY_TAG=$(flow-deploy tag) || {
       echo "script/prod: set DEPLOY_TAG (flow-deploy tag couldn't tell)" >&2
       exit 1
     }
     export DEPLOY_TAG
   fi

   On a fresh host with nothing running, pass DEPLOY_TAG=vX.Y.Z explicitly.
5. Label queue workers and schedulers deploy.strategy: drain-first, with deploy.drain set to the longest job plus a margin. Any deploy.order workaround that kept workers late becomes unnecessary.
6. Keep accessories off the app's .env. Drop env_file from postgres, redis and the like, and list the variables they need under environment: (interpolated from .env). Then app-env edits don't recreate them. Scope manual changes: flow-deploy compose up -d --no-deps <service>.
7. Make PID 1 handle SIGTERM. Use an exec-form CMD, or exec the server in the entrypoint. Otherwise every drain waits out its timeout and kills in-flight work (flow-deploy now warns).
8. Upgrade flow-deploy on each host before relying on the new behavior. deploy-action only updates the copy in CI, which runs config. Pass upgrade: true, or run flow-deploy upgrade on the host.

deploy-action: no changes required. It passes tag through, and flow-deploy fetches tags itself.

Docs

README (Image Tags, manual commands, revert model), SPEC (lifecycle with snapshot and revert steps, deploy order and drain-first, §2.8 image tags, §4.7 tag, §4.8 compose, exit codes, the wrapper example, the workflow example) and docs/github-actions.md (the image tag contract, version-tag deploys with a SHA fallback, release-builder created-tag, troubleshooting). site/docs.html renders the markdown, so it needs no change. CHANGELOG is generated by release-builder.

Review

Codex reviewed every round. The image-tag work took five passes on an alias-tag design. That design was then replaced, at the owner's direction, with restores named by git tags and SHAs. The replacement took four more passes, and the operator-facing additions took four more. A final adversarial review for cruft cut the accessory drift check (warn-only, sensitive to Compose versions) before merge. Fixed along the way:

• Shipped-service rollback after a cutover cleanup failure. Rollback now goes by image ID and keeps a surviving previous-image container.
• Scale-back is skipped while the container it should leave alone might still exist.
• Renamed compose projects are refused. The check is narrowed to projects running this deploy's services, so a shared directory is fine.
• A tag-plus-digest ref is checked as repo:tag, not as the digest.
• A redeploy of the running tag restores by SHA, since the deploy's pull moves that tag.
• Stopped leftovers are removed before a revert's compose up --scale. That includes those of the failed service itself, so the next deploy can't restart them.
• flow-deploy tag after a revert from :latest answers with the SHA when every container runs that one image.
• The run log is opened only under the lock, so a locked-out deploy can't clobber it.
• A restored service with zero containers up counts as not restored.
• The lock race (pre-existing).
• A signal mid rolling service: the service being rolled is now reverted too, not left with old and new side by side.
• Doc mismatches.

Declined, with reasons:

• docker ps --filter status=exited needs --all: it doesn't. I checked on a real daemon, and the code comments it.
• Keep recognizing old alias containers: the alias design was never released.
• The same-SHA redeploy limit: accepted, warned and documented, since it follows from using no host-side tags.

Follow-ups

• Simplify cleanup-failure recovery during revert #32 Simplify cleanup-failure recovery during revert
• Create missing accessories on deploy, never change existing ones #33 Create missing accessories on deploy, never change existing ones
• Make the flock the deploy lock itself #34 Make the flock the deploy lock itself
• Switch script/prod to the fail-closed DEPLOY_TAG default once hosts are past 0.3.0 #35 Switch this repo's script/prod to the fail-closed DEPLOY_TAG default once hosts are past 0.3.0
• Push the docs image by full commit SHA too #36 Push the docs image by full commit SHA too

Testing

• script/lint clean.
• Unit: poetry run pytest gives 338 passed at 100% coverage. The tests run on FakeHost, which simulates docker, compose and git from in-memory state, so a test declares what runs and what fails rather than a call sequence.
• Integration (real Docker): poetry run pytest -m integration --no-cov -p no:randomly gives 32 passed. New scenarios:
  • an annotated version-tag deploy
  • a SHA deploy
  • restoring a previously running version (v1 → v2 ok, then v3 fails, so everything returns to v2, shipped web included)
  • a first deploy from :latest, restored by the previous commit's SHA after :latest is re-pushed and pulled
  • a digest-pinned worker, restored by the previous commit's SHA
  • another project with different services in the directory doesn't block a deploy
  • SIGKILL warnings (drain and cutover)
  • the host-side log across a success and a failure, and status
  • a revert whose restored worker stays unhealthy exits 3
  • discovery with the image variable unset
  • flow-deploy compose up -d changes nothing after a deploy
  • flow-deploy compose refuses with nothing to go on
  • a moved and re-pushed tag
  • another compose project in the directory is refused

🤖 Generated with Claude Code

https://claude.ai/code/session_01Be6ghMqRxW6R4WS3vJY1Yv