From 75f40d7b8c19d68137a32387466d54c61b23efa3 Mon Sep 17 00:00:00 2001 From: Adambomb210 Date: Tue, 3 Nov 2026 14:27:38 -0500 Subject: [PATCH 1/3] docs(production): the steps that turn production on, and what's stale Adds section 0.2: the nine steps from today's state to production on weft (Weave, Slack, secrets, tailnet, fallback rev, switch, DNS, edge, checks), how anyone with infra push can write production's secrets before it has ever run, and the decisions to make before members apply (large requests, Weave's one-grant cap, members exempt from the code of conduct, production CI deploys). Fixes the passages that still said alastor, arm64, a trusted proxy count of 2, and that the module had never run. Co-Authored-By: Claude Opus 5.5 --- docs/dev/production.md | 70 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 58 insertions(+), 12 deletions(-) diff --git a/docs/dev/production.md b/docs/dev/production.md index ed69b73..c64dade 100644 --- a/docs/dev/production.md +++ b/docs/dev/production.md @@ -11,8 +11,8 @@ staging run first ([staging.md](staging.md)); this page assumes everything there - **On any other Docker host: this Compose file**, with the opt-in `proxy` (Caddy, automatic TLS) and `backup` profiles. Sections 2 to 5 describe that path. -Section 1 (DNS, Weave, Slack, Vast) applies to both. The image builds for arm64, which alastor is, as well as amd64 -(checked under emulation; not yet run on arm64 hardware). +Section 1 (DNS, Weave, Slack, Vast) applies to both. The image builds for amd64 (weft, where staging runs) and arm64 +(checked under emulation only). ### 0.1 On weft (NixOS) @@ -30,11 +30,12 @@ sudo systemctl start krater-build # the running site keeps serving meanwhile sudo systemctl restart krater-web # migrates, then starts; krater-worker follows ``` -Settings live in `hosts/weft/krater.nix` (hostnames, Weave client id, the fallback commit). The proposed hostnames -are `krater` and `krater-storage` under `patchworklabs.org`, added by a PR to -[patchworklabsorg/dns](https://github.com/patchworklabsorg/dns). They're proxied through Cloudflare like Weave's, -whose edge certificate covers only one subdomain level. Behind Cloudflare and Traefik, `KRATER_TRUSTED_PROXY_COUNT` -probably needs to be 2 (see the host file). Secrets are the agenix file +Settings live in `hosts/weft/krater.nix` (hostnames, Weave client id, the fallback commit); production's one switch +is `hosts/weft/krater-production.nix`. Production's hostnames are `krater` and `krater-storage` under +`patchworklabs.org`, added by a PR to [patchworklabsorg/dns](https://github.com/patchworklabsorg/dns), next to +staging's `krater-staging` and `krater-staging-storage`. They're proxied through Cloudflare like Weave's, whose edge +certificate covers only one subdomain level. `KRATER_TRUSTED_PROXY_COUNT` is 3: Cloudflare, Traefik on alastor and +nginx on weft each add one address (see the host file). Secrets are the agenix file `secrets/krater-env.age`, with the keys listed at the top of `modules/krater/default.nix`, plus `secrets/central-prod-krater.age` (the database password alone, for skein); infra's `bin/new-krater-secrets` writes both. The module splits the env file so each container gets only its own. Getting a module change @@ -44,7 +45,8 @@ that, only the `Deploy` workflow. The database is Krater's schema `krater` in `patchwork_central` on skein (infra's `modules/central-db`), not a Postgres of its own: the role `krater` has `search_path = krater`, so Krater's tables, its procrastinate queue and `alembic_version` all live in that schema, and nothing changes in Krater's code. `KRATER_DATABASE_URL` points at skein -(its tailnet name from alastor). skein's nightly dump is the backup; a restore is of the schema `krater` and goes +(its tailnet name, pinned in the host file's `extraHosts` because weft has no MagicDNS). Staging does the same with its +own schema on hem. skein's nightly dump is the backup; a restore is of the schema `krater` and goes through whoever runs skein. **Compute off (the first deploy).** Krater ships to production with `KRATER_COMPUTE_MODE=off`: proposals, review, @@ -66,6 +68,51 @@ the file, restart, run `python -m krater.rotate_project_secrets` in the portal c `KRATER_PROJECT_SECRETS_KEY_FILE`: a Fernet key file (an agenix secret) mounted into the portal only. Without it the feature is off. Losing the key loses every stored secret, so back it up with the other agenix secrets. +### 0.2 Turning production on + +The state on 2026-11-03. Staging on weft has run the whole flow live: a member's proposal, an amendment and a +completion, each approved by another account, the budget granted and reclaimed, the gallery and the OSPool PDF. +Production is the same module behind `hosts/weft/krater-production.nix`, which is still `enable = false`. The host +file's "Turn on production" comment is the short form of these steps, in this order: + +1. **Weave production.** Krater's app exists (its client id is in the host file). Check on its app page: redirect + URI `https://krater.patchworklabs.org/auth/callback`, scopes `openid profile email groups roles slack directory`, + access "everyone", and the roles `reviewer` and `admin` given to the Ganymede Reviewer and Ganymede Admin groups + ([weave-integration.md](../weave-integration.md)). +2. **Slack.** Krater's app in the real Patchwork workspace ([slack-setup.md](slack-setup.md)), with interactivity URL + `https://krater.patchworklabs.org/slack/interactions`. Note the feed and alerts channel ids: they aren't secret and + go in the host file (`slackFeedChannelId`, and `KRATER_SLACK_ALERTS_CHANNEL_ID` in `extraEnv`). +3. **Secrets.** `secrets/krater-env.age` exists, but its `KRATER_WEAVE_CLIENT_SECRET`, `KRATER_SLACK_BOT_TOKEN` and + `KRATER_SLACK_SIGNING_SECRET` are empty, and Krater refuses to start until they're set. Only the file's recipients + (Jasper, weft) can edit it. But production has never run, so nothing in it or in `central-prod-krater.age` is in + use yet: anyone who can push to infra can replace both instead, without reading them. On a branch, delete the two + files, then run infra's `bin/new-krater-secrets production` in a shell with agenix (WSL with Nix works), with the + three values in its environment, typed into that shell and never into a file in the repo. It writes both files + with a new database password. Once production has started, only the recipients can change them. +4. **Tailnet.** In `jaspermayone/infra` `tailscale/policy.hujson`, give `tag:patchwork-krater` `pw-skein:5432` again + (Jasper; the grant and test lines are in infra#32's body). +5. **The first build.** Production has no CI deploy, so its first build is the host file's fallback `rev`. Point it + at a current `main` commit first; the old one predates the launch hardening, the funder packet and every fix + since. +6. **Switch on.** Set `enable = true` in `krater-production.nix`, deploy skein (the role and its password), then weft, + then run `systemctl start krater-prod-build` on weft by hand. +7. **DNS.** The `krater` and `krater-storage` records in patchworklabsorg/dns, proxied to alastor like staging's. +8. **The edge.** Bump `patchwork-infra` in `jaspermayone/infra` and deploy alastor, which adds the public routes. +9. **Check it** (section 3, "Check it"): `/healthz` and `/healthz/worker` answer 200, an admin signs in and sees + Review and Admin, a TEST proposal opens its Slack channel, and an outside uptime check watches `/healthz/worker`. + +Decide before members apply (none of these blocks the switch): + +- **Large requests.** Krater has no ceiling on one grant. The default policy is one approval from any reviewer, and + the hourly cap limits only how fast a budget is spent. Add an approval policy on `/admin` with a `min_budget_cents` + floor (more approvals, or the admin group) for large requests. +- **Weave's one-grant cap.** Weave records Ganymede grants by hand on a member's identity record, one per person for + life (`config/kyc/knot.yml`). Krater doesn't read or write it, and a member can hold several approved projects. +- **Members exempt from the code of conduct** stay Slack guests, so Krater's Slack membership gate blocks their + submissions until they're full members of the Slack. +- **Production CI deploys.** Give production its own deploy key and user (`krater-prod-deploy`) when releases should + follow `main`, as staging's do. + ## 1. Before the first deploy - **DNS:** point these A/AAAA records at the host. The names are examples. @@ -201,7 +248,6 @@ docker compose --env-file "$KRATER_ENV_FILE" exec -T db pg_dump --format=custo ## 6. Not covered yet -- **Backup alerts.** The alerts above don't watch `db-backup`: check its logs (on alastor, skein's nightly dump). -- **Off-machine backup copies** and a screenshot volume backup (section 4). -- **The NixOS module has never run on NixOS.** It evaluates and its units build (in a `nixos/nix` container), and - the image builds from a GitHub commit for arm64, but nothing has started on alastor yet. +- **Backup alerts.** The alerts above don't watch `db-backup`: check its logs (on weft, skein's nightly dump). +- **Off-machine backup copies** and a screenshot volume backup (section 4). On weft, screenshots sit in SeaweedFS on + weft's own disk, and neither infra's `modules/krater` nor weft's host config backs it up yet. From 27639d80ccf0a5cd1c5bf1ea5f743343693b23d4 Mon Sep 17 00:00:00 2001 From: Adambomb210 Date: Tue, 3 Nov 2026 14:30:05 -0500 Subject: [PATCH 2/3] docs(slack): Krater's app needs groups:read The live client finds a project's existing private channel with conversations.list (types=private_channel) when conversations.create answers name_taken. Slack needs groups:read for that, and the manifest, the setup guide and the spec left it out, so a retried channel creation would fail instead of reusing the channel. Co-Authored-By: Claude Opus 5.5 --- docs/SPEC.md | 3 ++- docs/dev/slack-app-manifest.yaml | 1 + docs/dev/slack-setup.md | 2 ++ 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/SPEC.md b/docs/SPEC.md index e7eba7d..c62bcb1 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -283,7 +283,8 @@ answers 503 once its heartbeat is 5 minutes old, for an uptime check outside Kra mirrored into Krater in v1; the project page links to the channel instead. **App requirements** -- Bot scopes: `groups:write` (creating, archiving, and removing people from project channels), `groups:write.invites`, +- Bot scopes: `groups:write` (creating, archiving, and removing people from project channels), `groups:read` + (finding a project's channel by name when creating it answers `name_taken`), `groups:write.invites`, `chat:write`, `users:read` (the membership gate, when Weave has no `slack_member` for someone). A workspace that restricts who may remove members from private channels refuses the removal; Krater logs it, and an admin removes the person by hand. diff --git a/docs/dev/slack-app-manifest.yaml b/docs/dev/slack-app-manifest.yaml index 0ec9d81..fc8cc5d 100644 --- a/docs/dev/slack-app-manifest.yaml +++ b/docs/dev/slack-app-manifest.yaml @@ -12,6 +12,7 @@ oauth_config: scopes: bot: - chat:write + - groups:read - groups:write - groups:write.invites - users:read diff --git a/docs/dev/slack-setup.md b/docs/dev/slack-setup.md index 10b1d14..5ae966b 100644 --- a/docs/dev/slack-setup.md +++ b/docs/dev/slack-setup.md @@ -17,6 +17,8 @@ name, the bot scopes and interactivity (steps 1, 2 and 5 below). Then do steps 3 code-of-conduct acceptance, so Krater needs its own app entirely). 2. **OAuth & Permissions** -> **Scopes** -> **Bot Token Scopes**, add exactly what `docs/SPEC.md` lists: - `groups:write` -- create/archive private channels + - `groups:read` -- find a project's channel by name (`conversations.list`) when creating it answers + `name_taken`, e.g. after a half-finished earlier attempt - `groups:write.invites` -- invite the submitter, credited builders and reviewers - `chat:write` -- post/update the review message and admin-override notices - `users:read` -- look up a user's guest/restricted/deleted status (`users.info`) for the Slack membership From ce58f18b8e8c45f874e6a6a590c4304d827dbeeb Mon Sep 17 00:00:00 2001 From: Adambomb210 Date: Wed, 4 Nov 2026 20:27:55 -0500 Subject: [PATCH 3/3] docs(production): progress on 2026-11-04, approval pipeline first Slack app, secrets and first-build rev are on infra#73; DNS is dns#18. Jasper's steps are left. Co-Authored-By: Claude Opus 5.5 --- docs/dev/production.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/dev/production.md b/docs/dev/production.md index c64dade..6e9d85a 100644 --- a/docs/dev/production.md +++ b/docs/dev/production.md @@ -75,6 +75,12 @@ completion, each approved by another account, the budget granted and reclaimed, Production is the same module behind `hosts/weft/krater-production.nix`, which is still `enable = false`. The host file's "Turn on production" comment is the short form of these steps, in this order: +**Progress on 2026-11-04.** The first launch is the approval pipeline only; compute stays off. Steps 2, 3 (except the +Weave client secret) and 5 are done on patchworklabsorg/infra#73. That PR has the Ganymede Review app's feed and +alerts channel ids, the new secrets, and the rev. The DNS records are patchworklabsorg/dns#18, which replaces dns#12. +Jasper's steps are left: checking the Weave app and setting `KRATER_WEAVE_CLIENT_SECRET` (`agenix -e krater-env.age`), +then steps 4 and 6–9. + 1. **Weave production.** Krater's app exists (its client id is in the host file). Check on its app page: redirect URI `https://krater.patchworklabs.org/auth/callback`, scopes `openid profile email groups roles slack directory`, access "everyone", and the roles `reviewer` and `admin` given to the Ganymede Reviewer and Ganymede Admin groups