Skip to content

feat(krater): krater staging on weft, production ready behind one switch - #32

Merged
jaspermayone merged 29 commits into
mainfrom
krater-compute
Oct 23, 2026
Merged

jaspermayone merged 29 commits into
mainfrom
krater-compute

Conversation

@Adambomb210

@Adambomb210 Adambomb210 commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Krater on weft: staging now, production ready and off

Krater runs on weft, its own lab VM (pve01 VMID 361, 10.100.90.61, 4 GB, 40 GB disk). One app per VM: sudo and the docker group are root, so the people who run Krater cannot read the secrets of Quilt or Weave. Adam's original description is kept below for context.

  • Staging runs now: krater-staging.patchworklabs.org, Weave staging, the krater schema in patchwork_central on hem.
  • Production is in the config and its secrets exist, but it is off. hosts/weft/krater-production.nix (enable = false) is its one switch. weft's units and nginx, alastor's routes and skein's role all read that file, so they change together.

What changed on top of Adam's commits:

  • Merged main twice (hosts per app, Quilt on batting, weft bring-up feat(weft): add the weft host key, give it ad-join, and size it for krater #60). Adam's commits are unchanged.

  • hosts/alastor/krater.nix is now hosts/weft/krater.nix. The krater-alastor export is gone.

  • Two instances from one module. modules/krater/instance.nix is the old module, with every name from a prefix. modules/krater/default.nix imports it twice:

    Staging Production
    option patchwork.services.krater patchwork.services.krater-production
    units, network, image repo krater-*, krater, krater:<sha> krater-prod-*, krater-prod, krater-prod:<sha>
    state, run dirs /var/lib/krater, /run/krater /var/lib/krater-prod, /run/krater-prod
    ports (loopback) 3010, 3011 3012, 3013
    hostnames krater-staging, krater-staging-storage krater, krater-storage
    Weave staging, client id set production, client id set (GwmIPZen…)
    database hem (extraHosts 100.67.124.123) skein (extraHosts 100.109.223.51)
    env file krater-staging-env.age krater-env.age
    Vast label prefix krater-staging- krater-prod-

    Each build removes old tags of its own image repository only, so staging never deletes a production image. Each store keeps 8 GB of objects at most (storage.maxVolumes = 8): the 40 GB disk holds the system, two stores, two image repositories and the BuildKit cache.

  • Client ids (not secret): staging vTt8nTKFw7Qau8S2aBfhXrBQuCi1PbduyF0x6sH5QJo, production GwmIPZenHuqRjUjawfJyDYceEcdXDJibTbmEEkuRJDU. They are stagingWeaveClientId and productionWeaveClientId, each next to its issuer. The production id does not turn production on: only the switch does.

  • Staging is on in this config now: the id is set and krater-staging-env.age exists (Jasper set its Weave client secret, 2675e4e).

  • Production guard: with the switch on, the weft build fails if productionWeaveClientId is empty. An empty key in krater-env.age fails the unit at once (Krater's settings check).

  • Database: hosts/central-db-schema.nix is schemaFor <switch>, and also exports schemaFor. krater is in stagingOnlyApps while the switch is off, and in apps (skein) when it is on. skein's and hem's builds do not change today.

  • Catch-all vhost on weft: an unknown Host, or a name whose instance is off, gets 444. It never falls through to the staging portal.

  • nginx on weft (tailscale0:80) has the vhosts of each instance only while it is on. It passes alastor's X-Forwarded-Proto, appends to X-Forwarded-For, and does no real_ip rewrite. KRATER_TRUSTED_PROXY_COUNT is 3 (Cloudflare, Traefik, nginx).

  • alastor routes (nixosModules.edge-alastor): krater-staging-weft and krater-staging-storage-weft always; krater-weft and krater-storage-weft only with the switch on (patchwork.edge.kraterProduction, default from the switch file).

  • Other module changes: extraHosts (--add-host), a settings check before the database wait, docker builder prune after builds, storage.maxVolumes, and compute.labelPrefix required when compute is on. Fallback rev b081a13 on Krater's main. zramSwap on weft.

  • KRATER_QUILT_URL is empty in both. Never point staging at production Quilt.

  • Staging's Slack app is a separate app in a test workspace. Production uses the real Patchwork Slack app.

  • KRATER_ENV is production in both: Krater has no staging mode. The construction border and STAGING label need a change in the Krater app.

Secrets

File Recipients State
krater-staging-env.age jsp, weft done: database URL (hem), secret key, S3 keys, Weave client secret. Slack keys still empty
central-staging-krater.age jsp, hem done
krater-env.age jsp, weft made by bin/new-krater-secrets production: database URL (skein), secret key, S3 keys. Jasper fills in later: KRATER_WEAVE_CLIENT_SECRET, KRATER_SLACK_BOT_TOKEN, KRATER_SLACK_SIGNING_SECRET
central-prod-krater.age jsp, skein done. skein reads it only once the switch is on
krater-staging-vast-key.age, krater-vast-key.age, krater-staging-project-secrets-key.age, krater-project-secrets-key.age jsp, weft rules only, no files: compute and project secrets are off

No value was printed. The checks read only key names and sizes.

flowchart LR
  user((browser)) --> cf[Cloudflare] --> alastor["alastor: Traefik"]
  alastor -- "tailnet :80" --> nginx["weft: nginx"]
  nginx --> sweb["krater-web :3010"]
  nginx --> ss3["krater-storage :3011"]
  sweb -- "tailnet :5432" --> hem["hem: patchwork_central staging"]
  nginx -. "switch on" .-> pweb["krater-prod-web :3012"]
  nginx -. "switch on" .-> ps3["krater-prod-storage :3013"]
  pweb -. "tailnet :5432, switch on" .-> skein["skein: patchwork_central"]
Loading

Staging rollout

  1. Merge this PR. Deploy hem (deploy .#hem), so the krater role gets its password.
  2. The first weft deploy blocks on the image build. krater-web has requires = [ "krater-build.service" ] and is wanted by multi-user.target, so activation starts the build and waits for it (TimeoutStartSec = 45min). The first build takes several minutes. deploy-rs's activation timeout is shorter than that, so pick one:
    • Build first: deploy weft with stagingWeaveClientId = "" (Krater off), run sudo systemctl start krater-build, then deploy again with the id. Or
    • Deploy with --magic-rollback false and a long --activation-timeout, and watch journalctl -u krater-build.
  3. Slack for staging (test workspace) in krater-staging-env.age, if the worker must talk to Slack.
  4. DNS in patchworklabsorg/dns: krater-staging and krater-staging-storage, proxied, to alastor.
  5. alastor: bump patchwork-infra in jaspermayone/infra and deploy alastor (--magic-rollback false).
  6. CI deploys: set deployAuthorizedKeys, and point the Krater repo deploy secrets at weft.

Turn on production

  1. Weave production: done. Krater's app exists, and its client id is in productionWeaveClientId.
  2. Secrets (Jasper): cd secrets && agenix -e krater-env.age, and fill in KRATER_WEAVE_CLIENT_SECRET, KRATER_SLACK_BOT_TOKEN, KRATER_SLACK_SIGNING_SECRET (the real Patchwork Slack app). The build does not check them. An empty one fails krater-prod-web at once.
  3. Tailnet (jaspermayone/infra tailscale/policy.hujson), then echo y | tailscale/bin/apply. Add the grant:
    {
    	// weft (Krater production) -> patchwork_central on skein.
    	"src": ["tag:patchwork-krater"],
    	"dst": ["pw-skein", "pw-skein-6"],
    	"ip":  ["tcp:5432"],
    },
    and in the tag:patchwork-krater test, move pw-skein:5432 from deny to accept: "accept": ["pw-skein:5432", "pw-hem:5432"].
  4. The switch: enable = true in hosts/weft/krater-production.nix. This also puts krater in apps for skein and adds the two alastor routes.
  5. Deploy skein (the krater role and schema, with the password in central-prod-krater.age), then weft (the same build wait as staging: krater-prod-build).
  6. DNS: krater and krater-storage, proxied, to alastor.
  7. alastor: bump patchwork-infra in jaspermayone/infra and deploy alastor.

Checks

  • nixosConfigurations.weft, skein, hem and seam evaluate. On weft, staging is on and production is off. skein's apps are selvage and weave only. skein's and hem's system derivations are the same as before the schema change.
  • checks.x86_64-linux.krater-weft evaluates, so all its eval-time checks pass: staging on by default, staging off when forced, production on when forced, the catch-all vhost, the edge default equal to the switch file, and the schema for both switch values.
  • jaspermayone/infra's alastor evaluates with this branch as patchwork-infra.
  • nix flake check --no-build passes.
  • Build checks.x86_64-linux.krater-weft (its grep steps), central-db and quilt-batting on warp, for 24d4375. Not run: Tailscale is stopped on the machine that ran the checks.
  • The rollouts above.

Original description (alastor, production)

Adds Krater, the Project Ganymede portal, as a NixOS module (nixosModules.krater) and its alastor settings (nixosModules.krater-alastor). alastor doesn't import it yet: that's a change in jaspermayone/infra once this merges.

What it runs

  • krater-web: the portal, and the API rented machines report to. It waits for the database, runs migrations, then serves on loopback behind Traefik at krater.patchworklabs.org.
  • krater-worker: background jobs (Slack, pricing, events to Quilt, and the compute scheduler when compute is on).
  • krater-storage: SeaweedFS's S3 gateway at krater-storage.patchworklabs.org. It holds gallery screenshots, and machines' checkpoints and outputs when compute is on.
  • krater-build: builds the image on alastor from one commit of patchworklabsorg/krater. There's no registry. Krater's Deploy workflow writes the commit to /var/lib/krater/source-rev as the krater-deploy user, which may run only those steps.

Database: patchwork_central

Krater has no Postgres of its own. Its data is the schema krater in patchwork_central on skein, like the other central apps:

  • krater = { } in hosts/central-db-schema.nix. Krater stores no personal data (only Patchwork IDs) and reads people through Weave, so it gets no shared schemas.
  • The role's search_path is krater, so Krater's tables, its job queue and alembic_version all live there. Krater needs no code change: its migrations and 1075 tests pass on Postgres 18 set up this way, with nothing in public.
  • skein's nightly dump is the backup.

Krater Compute: in the module, off on alastor

With compute.enable, the worker rents Vast.ai machines for members' jobs. The Vast key (compute.vastKeyFile) goes into the worker's env alone. alastor keeps it off (compute.enable = false) until the Vast key is rotated: the current key was used on staging. Turning it on:

  1. Put the new key in secrets/krater-vast-key.age.
  2. Uncomment the krater-vast-key secret in hosts/alastor/krater.nix.
  3. Set compute.enable and compute.vastKeyFile.

Project secrets (members' tokens for their jobs) work the same way with krater-project-secrets-key.age.

Secrets

bin/new-krater-secrets writes both new files and prints no value:

  • krater-env.age (jsp, alastor): the env file. KRATER_DATABASE_URL holds the role's password, plus KRATER_SECRET_KEY and the storage keys. It takes the Weave client secret and the Slack values from the environment when they're set.
  • central-prod-krater.age (jsp, skein): the role's password alone.

Both files are in this PR (ce73de8; the recipients were checked against secrets.nix). KRATER_WEAVE_CLIENT_SECRET, KRATER_SLACK_BOT_TOKEN and KRATER_SLACK_SIGNING_SECRET are empty because the Weave and Slack apps don't exist yet. Krater refuses to start in production until they're filled in with cd secrets && agenix -e krater-env.age (only Jasper can).

Before alastor imports it

  • bin/new-krater-secrets: both secrets, committed
  • The Weave client secret and the Slack bot token and signing secret, added to krater-env.age (Jasper)
  • Krater's Weave app: the client id goes in hosts/alastor/krater.nix, and the module refuses to build while it's empty
  • skein loads central-prod-krater.age into the central DB's password directory. skein's config doesn't load any app's password file yet, so Krater's role can't log in until it does.
  • The tailnet ACL lets alastor reach skein's port 5432, and a container on alastor resolves skein's tailnet name (as Quilt reaches seam)
  • DNS: krater and krater-storage (patchworklabsorg/dns, linked below)
  • deployAuthorizedKeys, and the ALASTOR_HOST and KRATER_DEPLOY_SSH_KEY secrets in the Krater repo
  • KRATER_TRUSTED_PROXY_COUNT: 2 if alastor's Traefik trusts Cloudflare's ranges in forwardedHeaders.trustedIPs

Checked

  • alastor's config with this module evaluates with compute off and with compute on (a placeholder env file, a test client id). The units are krater-build, krater-network, krater-storage, krater-web and krater-worker, and there are no timers.
  • The database wait works: it connects when the database is up and keeps waiting when it isn't. Migrations reach head against a central-db-style schema.
  • bin/new-krater-secrets ran in nixos/nix with throwaway keys. Both files decrypt, and it refuses to overwrite either one.
  • The module hasn't run on a NixOS host yet. Expect a round of fixes on the first deploy.

The fallback rev is Krater's claude/application-form (16de89a). After patchworklabsorg/krater#9 merges, set it to the merge commit.

This replaces the branches krater-module and krater-compute-off.

Adambomb210 and others added 15 commits October 9, 2026 21:16
Krater (the Project Ganymede portal) as Docker containers on alastor, in
the style of the weave module: Postgres, SeaweedFS screenshot storage, the
portal, its procrastinate worker, and the SkyPilot API server with its
oauth2-proxy sign-in.

There's no registry image. krater-build fetches one commit of
patchworklabsorg/krater and builds it with the repo's own Dockerfile,
tagged krater:<sha>, skipping commits already built. A deploy writes
/var/lib/krater/source-rev, runs `systemctl start krater-build` while the
old containers keep serving, then `systemctl restart krater-web`.

One agenix env file is split per container at start, so the SkyPilot
server never sees Krater's session key or database password. SkyPilot's
bootstrap values live in host state files, so they need no infra round
trip. krater-backup dumps the database nightly.

Exported as nixosModules.krater and nixosModules.krater-alastor. alastor
doesn't import it yet: it needs DNS, the Weave client ids and the two
secrets first.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- /var/lib/krater is 0755 like the weave module's state dir. At 0750
  krater-deploy couldn't reach source-rev, so every CI deploy failed.
  backups/ and secrets/ stay 0700 through tmpfiles, outside
  StateDirectory (whose mode is reapplied on every start), and dumps are
  written with umask 077.
- Weave's issuer is https://weave.patchworklabs.org, per its discovery
  document.
- One-level hostnames (krater-storage, krater-sky): the records are
  proxied through Cloudflare, whose edge certificate covers one level.
- Note on KRATER_TRUSTED_PROXY_COUNT behind Cloudflare plus Traefik.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
`latest` has already moved past the image Krater's staging was tested on.
Screenshot storage stays on self-hosted SeaweedFS for now.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
SkyPilot asks oauth2-proxy's /oauth2/auth who the user is and reads
X-Auth-Request-Email from the answer. oauth2-proxy sends those headers
only with set-xauthrequest on. Without it, every `sky api login` ended in
a 500 ("oauth2-proxy is enabled but did not return user info"). Found on
the local staging run; with the flag the login completes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…kyPilot

SkyPilot's Vast support can't rent a machine and copies the Vast key onto
members' hosts, so Krater ships with KRATER_COMPUTE_MODE=off until Krater
Compute replaces it. With skypilot.enable = false (the default) the module
leaves out krater-skypilot, krater-auth-proxy, their env and secrets, the
SkyPilot service token, the krater/skypilot state directory and the
krater-sky Traefik route, and no longer requires skypilot.domain or
weave.skypilotClientId. KRATER_SKYPILOT_MODE stays live: the budget
estimator still reads SkyPilot's public GPU catalog.

alastor sets skypilot.enable = false and drops skypilot.domain,
skypilotClientId and the unused krater-vast-key secret.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Krater dropped SkyPilot for Krater Compute, its own Vast layer. The module loses the SkyPilot server, its sign-in
proxy, their secrets, state and Traefik route, and gains compute.enable (off by default; KRATER_COMPUTE_MODE=vast
when on), compute.vastKeyFile (read into the worker's env only), compute.labelPrefix (unique per deployment on a
Vast account), compute.runnerUrl, projectSecretsKeyFile (Fernet keys, the portal's env only) and
weave.cliClientId (the krater CLI's public Weave app). KRATER_SKYPILOT_MODE becomes KRATER_PRICING_MODE.

The portal and worker now get their own env files (web.env, worker.env) on top of app.env, since the image runs
as a non-root user and can't read a root-only agenix file mounted in. alastor keeps compute off; the new secret
krater-project-secrets-key.age is declared but not created.

Checked by nix eval (compute off, on, and on without a key) and nixfmt.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The price-history run takes over a minute, and with one task at a time Krater's minute-by-minute compute
scheduler waited behind it (seen on staging). The scheduler and reconciliation share a lock, so they never overlap.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Krater's v1 (PR #1) no longer contains SkyPilot: it runs with compute off and
Krater Compute adds machines in its own release. The module drops the
skypilot.* options, the SkyPilot API server, its sign-in proxy, their
secrets, state and Traefik route, and skypilot-start.sh. KRATER_COMPUTE_MODE
is always off; KRATER_SKYPILOT_MODE becomes KRATER_PRICING_MODE (the
budget estimator's public GPU catalog, no server). alastor drops
skypilot.enable.

Adapted from the krater-compute branch's 021a7f9, without its compute
options (compute.*, projectSecretsKeyFile, weave.cliClientId) or the
per-container env files they need.

Checked by nix eval of krater-alastor (agenix stubbed): the Krater units
evaluate with no failed assertions; nixfmt is clean.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…e too

This branch already had the same removal with Krater Compute's options; its
version stays.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The default -volume.max=8 is used up by the first collections; a new bucket then
can't get a volume and every upload to it fails with a 500 (found on Krater
staging: screenshots after compute's checkpoints).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The default -volume.max=8 is used up by the first collections; a new bucket then
can't get a volume and every upload to it fails with a 500 (found on Krater
staging: screenshots after compute's checkpoints).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Conflicts were additive: README (repo table, the Krater section next to the Quilt ones), flake.nix
(the krater modules next to edge-alastor and quilt) and secrets/secrets.nix (krater-env and
krater-vast-key next to the Quilt and AD secrets). The Krater README section now says two hostnames
and, with compute off, only krater-env.age.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Krater's data moves to the schema `krater` in patchwork_central on skein
(modules/central-db), like the other apps. The module no longer runs a
krater-db container or its own backup timer; skein's nightly dump covers it.

- hosts/central-db-schema.nix: add krater (no shared schemas).
- secrets: central-prod-krater.age (jsp, skein) holds the role password;
  krater-env.age now carries KRATER_DATABASE_URL with the same password.
- bin/new-krater-secrets writes both files with new random values and
  prints none of them.
- krater-web waits for the database with the app's own engine, and needs
  network-online.

Checked: the alastor config evaluates; Krater's migrations reach head in a
central-db-style schema on Postgres 18 with nothing in public.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…m the environment

Only Jasper and alastor can decrypt krater-env.age once it exists, so
whoever runs the script can pass KRATER_WEAVE_CLIENT_SECRET,
KRATER_SLACK_BOT_TOKEN and KRATER_SLACK_SIGNING_SECRET in the environment
instead of leaving them for Jasper. The script names the ones it left empty.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…a main

Brings infra main (Quilt beside Krater, central-db, lab hosts) and the move of
Krater's database to the schema `krater` in patchwork_central onto the branch
with Krater Compute. Conflicts: the module keeps the compute options and the
per-container web/worker env files, and drops krater-db, postgresImage and the
backup timer; secrets.nix keeps both sides.

The fallback rev now points at claude/application-form (16de89a) instead of
PR #1's 4f130ca, so a first start without a deploy doesn't build old code.

Checked: the alastor config evaluates with compute off and with compute on
(the worker env alone gets the Vast key).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
bin/new-krater-secrets: a new database password (in both files), secret key
and storage keys. krater-env.age (jsp, alastor) leaves
KRATER_WEAVE_CLIENT_SECRET, KRATER_SLACK_BOT_TOKEN and
KRATER_SLACK_SIGNING_SECRET empty until the Weave and Slack apps exist;
Jasper fills them in with agenix -e. central-prod-krater.age is for skein.

.gitattributes keeps the script LF, so it runs from a Windows checkout too.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Comment thread hosts/weft/krater.nix
Comment on lines +87 to +89
# The `krater` CLI's public Weave app (Krater's docs/compute/cli.md).
# Empty keeps the CLI's API off.
cliClientId = "";

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Adambomb210 what is this lol

Comment thread hosts/alastor/krater.nix Outdated
Comment on lines +84 to +86
# Not secret, but only known once the app exists in Weave. The module
# refuses to build while it is empty.
clientId = "";

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is like... prob fine to be public

@jaspermayone jaspermayone changed the title feat(krater): Krater on alastor, database in patchwork_central feat(krater): krater staging on weft, database in patchwork_central on hem Oct 22, 2026
@jaspermayone jaspermayone changed the title feat(krater): krater staging on weft, database in patchwork_central on hem feat(krater): krater staging on weft, production ready behind one switch Oct 23, 2026
@jaspermayone
jaspermayone merged commit 8b8bcc3 into main Oct 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants