diff --git a/.gitattributes b/.gitattributes index 3a59662..d454cfd 100644 --- a/.gitattributes +++ b/.gitattributes @@ -4,3 +4,6 @@ # Self-hosted fonts are binary; never let autocrlf touch them. *.woff2 binary + +# The self-hosted box copies these onto Linux as they are (deploy/selfhost/README.md). +deploy/selfhost/** text eol=lf diff --git a/deploy/selfhost/README.md b/deploy/selfhost/README.md new file mode 100644 index 0000000..be1f345 --- /dev/null +++ b/deploy/selfhost/README.md @@ -0,0 +1,230 @@ +# Krater on a self-hosted Docker box + +Moves Krater off weft (Jasper's lab, behind the alastor edge and the tailnet) onto one Ubuntu machine running Docker, +reached through a **Cloudflare Tunnel**. Why: weft's internet and DNS drop out (2026-11-19 18:06Z: every call to Vast, +Weave and Quilt failed with `Temporary failure in name resolution`). Every drop stalls the scheduler and the reports +of running boxes. + +What moves: + +| On weft today | On the box | +| --- | --- | +| `krater-prod-web`, `-worker` (NixOS units, image `ghcr.io/patchworklabsorg/krater:`) | `portal`, `worker`, same image | +| `krater-prod-storage` (SeaweedFS 4.47, `/var/lib/krater-prod/storage`) | `storage`, same version, `/srv/krater/prod/storage` | +| the `krater` schema in `patchwork_central` on skein (Postgres 18) | `db` (Postgres 18), database `krater`, role `krater` | +| `krater-prod-follow` timer (deploys the `production` branch) | `krater-follow@prod.timer` (same rules) | +| `krater-prod-backup` before each deploy, skein's nightly dump | the same pre-deploy dump, plus `db-backup` daily | +| Cloudflare -> alastor (Traefik) -> tailnet -> nginx on weft | Cloudflare -> tunnel -> `portal` / `storage` | + +The hostnames stay the same: `krater.patchworklabs.org` and `krater-storage.patchworklabs.org`. Running boxes +have the portal's URL built in, and Weave's redirect URI and Slack's interactivity URL use it too. Nothing outside +needs to change except DNS. + +Staging follows the same steps later, in its own directory (section 6). Do production first. + +Weave (on alastor) is not in the lab. Quilt (batting) and Stash (hank) are, but Krater only sends to them through +outboxes that retry, so a lab outage delays those events and nothing else. + +## Rules that can break production + +1. **Never run weft and the box at the same time with the same label prefix.** Each scheduler destroys every + `krater-prod-` instance on the Vast account that isn't in *its* database. The box would kill boxes weft started + after the dump, and weft would kill boxes the box starts. Weft's web and worker stop **before** the final dump and + never start again. Don't test the box with `KRATER_COMPUTE_MODE=vast` beforehand; use `off` (section 3). +2. **Keep weft's secrets.** That means the same `KRATER_SECRET_KEY`, the same S3 keys, the same Vast key, and above + all the same Fernet file (`project-secrets-key`). A new Fernet key loses every project secret members stored. + Only Jasper (and weft) can decrypt them. +3. **Keep the gap short.** Running boxes keep training while Krater is away, but a runner stops its machine after + `runner_self_stop_minutes` with no answer (180 by default; a project can set less). Plan on 30 minutes from + stopping weft to DNS pointing at the tunnel. + +## 1. The box + +- Ubuntu with Docker Engine and the Compose plugin (`docker compose version` 2.24 or newer), on amd64: CI builds + only amd64 images. +- Disk: at least 80 GB free under `/srv`, since the store's cap is 50 GB, plus the database, dumps and images. RAM: + at least 4 GB free, which is what weft had for staging and production together. +- It stays on: no sleep or suspend, and Docker starts at boot (`systemctl enable docker`). Ethernet is better than + Wi-Fi, and a UPS is better still. +- Give Docker resolvers that don't depend on the router, so one bad DNS server can't do to the box what it did to + weft. In `/etc/docker/daemon.json`, then `systemctl restart docker`: + ```json + { "dns": ["1.1.1.1", "8.8.8.8", "9.9.9.9"] } + ``` +- Nothing needs to be open inbound. The tunnel dials out. + +```bash +sudo mkdir -p /srv/krater/prod/{db,storage,backups} +sudo chmod 700 /srv/krater/prod +sudo git clone https://github.com/patchworklabsorg/krater.git /srv/krater/prod/src +sudo cp /srv/krater/prod/src/deploy/selfhost/krater.env.example /srv/krater/prod/krater.env +sudo chmod 600 /srv/krater/prod/krater.env +``` + +## 2. What Jasper hands over + +These come from `patchworklabsorg/infra`, decrypted with `agenix -d` (only `jsp` and weft are recipients). Send them +through a channel that isn't Slack, such as 1Password: + +- `secrets/krater-env.age`: every `KRATER_` key. Copy them into `krater.env` except `KRATER_DATABASE_URL`, which is + new here. +- `secrets/krater-vast-key.age` -> `/srv/krater/prod/vast-key` +- `secrets/krater-project-secrets-key.age` -> `/srv/krater/prod/project-secrets-key`, as is + +On the day (section 4), Jasper also does three things: +- stops weft's units; +- makes the final dump; +- keeps weft's storage up for the last sync. + +`chmod 600` each file. Fill in the rest of `krater.env` from the comments in it: make a new `POSTGRES_PASSWORD`, and +use the same password in `KRATER_DATABASE_URL`. + +## 3. The tunnel, and a dry run + +The tunnel needs someone with access to the patchworklabs.org zone in Cloudflare. + +1. In Zero Trust, go to Networks > Tunnels and create a `cloudflared` tunnel named `krater-prod`. Copy its token into + `/srv/krater/prod/tunnel.env` as `TUNNEL_TOKEN=`, mode 600. +2. **Don't add the public hostnames yet.** `krater` and `krater-storage` already exist as records pointing at alastor, + and adding them here would take traffic from weft. Section 4 does that step. + +Dry run, with compute off so nothing touches Vast. It checks the settings, the image, the database and the store: + +```bash +cd /srv/krater/prod +sudo sed -i 's/^KRATER_COMPUTE_MODE=.*/KRATER_COMPUTE_MODE=off/' krater.env +sudo KRATER_IMAGE_TAG=$(curl -s https://krater.patchworklabs.org/healthz | python3 -c 'import json,sys; print(json.load(sys.stdin)["revision"])') \ + docker compose -f src/deploy/selfhost/compose.yml --env-file krater.env up -d db storage migrate portal +curl -s http://127.0.0.1:3012/healthz # {"status": "ok", "revision": ""} +``` + +That runs migrations on an empty database, which only proves the plumbing. Throw it away afterwards. Stop the stack +and delete `db/` and `storage/` (the real data arrives in section 4). Then set `KRATER_COMPUTE_MODE=vast` again: + +```bash +sudo docker compose -f src/deploy/selfhost/compose.yml --env-file krater.env down +sudo rm -rf db storage && sudo mkdir db storage +sudo sed -i 's/^KRATER_COMPUTE_MODE=.*/KRATER_COMPUTE_MODE=vast/' krater.env +``` + +**Copy the store ahead of time.** It's up to 50 GB, so most of it should move before the day. Bring up `storage` on +the box alone, then copy weft's buckets in through the public storage hostname with the S3 keys from `krater.env`. +Downloads through Cloudflare have no size cap, and the copy only reads from weft. + +```bash +sudo KRATER_IMAGE_TAG=unset docker compose -f src/deploy/selfhost/compose.yml --env-file krater.env up -d storage +set -a; . ./krater.env; set +a +export RCLONE_CONFIG_WEFT_TYPE=s3 RCLONE_CONFIG_WEFT_PROVIDER=SeaweedFS RCLONE_CONFIG_WEFT_ENDPOINT=https://krater-storage.patchworklabs.org \ + RCLONE_CONFIG_WEFT_ACCESS_KEY_ID=$KRATER_S3_ACCESS_KEY_ID RCLONE_CONFIG_WEFT_SECRET_ACCESS_KEY=$KRATER_S3_SECRET_ACCESS_KEY +export RCLONE_CONFIG_BOX_TYPE=s3 RCLONE_CONFIG_BOX_PROVIDER=SeaweedFS RCLONE_CONFIG_BOX_ENDPOINT=http://127.0.0.1:3013 \ + RCLONE_CONFIG_BOX_ACCESS_KEY_ID=$KRATER_S3_ACCESS_KEY_ID RCLONE_CONFIG_BOX_SECRET_ACCESS_KEY=$KRATER_S3_SECRET_ACCESS_KEY +rclone lsd weft: # krater-compute, krater-screenshots +for b in krater-compute krater-screenshots; do rclone copy --checksum --transfers 8 weft:$b box:$b; done +``` + +Run the copy again the evening before to shrink the last pass. + +## 4. The cutover + +Do it at a quiet hour, with Jasper. Post in the members' channel first, since sign-in and the CLI will be away +for about half an hour. Running boxes keep running. + +**Jasper, on weft:** +```bash +touch /var/lib/krater-prod/follow-hold # no deploy in the middle +systemctl stop krater-prod-follow.timer +systemctl stop krater-prod-worker krater-prod-web # nothing starts or kills boxes from here on +systemctl start krater-prod-backup # the final dump, after both are stopped +ls -t /var/lib/krater-prod/backups | head -1 # send that file to the box +``` +Leave `krater-prod-storage` running for the last copy. Don't run `nixos-rebuild` on weft until section 5. A rebuild +restarts the units. + +**On the box:** +```bash +cd /srv/krater/prod +# 1. The last copy of the store (only what changed since the earlier pass). +for b in krater-compute krater-screenshots; do rclone copy --checksum --transfers 8 weft:$b box:$b; done +# 2. The database: db alone, then the dump. The schema arrives as `krater`, owned by the role `krater`. +# --wait: on an empty data directory Postgres initializes first, and a restore sent before then fails. +sudo KRATER_IMAGE_TAG=unset docker compose -f src/deploy/selfhost/compose.yml --env-file krater.env up -d --wait db +sudo KRATER_IMAGE_TAG=unset docker compose -f src/deploy/selfhost/compose.yml --env-file krater.env exec -T db \ + pg_restore --no-owner --no-privileges --exit-on-error -U krater -d krater < /path/to/krater-prod-