Repository navigation
feat(krater): krater staging on weft, production ready behind one switch - #32
Merged
Merged
Conversation
Krater (the Project Ganymede portal) as Docker containers on alastor, in the style of the weave module: Postgres, SeaweedFS screenshot storage, the portal, its procrastinate worker, and the SkyPilot API server with its oauth2-proxy sign-in. There's no registry image. krater-build fetches one commit of patchworklabsorg/krater and builds it with the repo's own Dockerfile, tagged krater:<sha>, skipping commits already built. A deploy writes /var/lib/krater/source-rev, runs `systemctl start krater-build` while the old containers keep serving, then `systemctl restart krater-web`. One agenix env file is split per container at start, so the SkyPilot server never sees Krater's session key or database password. SkyPilot's bootstrap values live in host state files, so they need no infra round trip. krater-backup dumps the database nightly. Exported as nixosModules.krater and nixosModules.krater-alastor. alastor doesn't import it yet: it needs DNS, the Weave client ids and the two secrets first. Co-Authored-By: Claude Opus 5.5 <[email protected]>
- /var/lib/krater is 0755 like the weave module's state dir. At 0750 krater-deploy couldn't reach source-rev, so every CI deploy failed. backups/ and secrets/ stay 0700 through tmpfiles, outside StateDirectory (whose mode is reapplied on every start), and dumps are written with umask 077. - Weave's issuer is https://weave.patchworklabs.org, per its discovery document. - One-level hostnames (krater-storage, krater-sky): the records are proxied through Cloudflare, whose edge certificate covers one level. - Note on KRATER_TRUSTED_PROXY_COUNT behind Cloudflare plus Traefik. Co-Authored-By: Claude Opus 5.5 <[email protected]>
`latest` has already moved past the image Krater's staging was tested on. Screenshot storage stays on self-hosted SeaweedFS for now. Co-Authored-By: Claude Opus 5.5 <[email protected]>
SkyPilot asks oauth2-proxy's /oauth2/auth who the user is and reads
X-Auth-Request-Email from the answer. oauth2-proxy sends those headers
only with set-xauthrequest on. Without it, every `sky api login` ended in
a 500 ("oauth2-proxy is enabled but did not return user info"). Found on
the local staging run; with the flag the login completes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…kyPilot SkyPilot's Vast support can't rent a machine and copies the Vast key onto members' hosts, so Krater ships with KRATER_COMPUTE_MODE=off until Krater Compute replaces it. With skypilot.enable = false (the default) the module leaves out krater-skypilot, krater-auth-proxy, their env and secrets, the SkyPilot service token, the krater/skypilot state directory and the krater-sky Traefik route, and no longer requires skypilot.domain or weave.skypilotClientId. KRATER_SKYPILOT_MODE stays live: the budget estimator still reads SkyPilot's public GPU catalog. alastor sets skypilot.enable = false and drops skypilot.domain, skypilotClientId and the unused krater-vast-key secret. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Krater dropped SkyPilot for Krater Compute, its own Vast layer. The module loses the SkyPilot server, its sign-in proxy, their secrets, state and Traefik route, and gains compute.enable (off by default; KRATER_COMPUTE_MODE=vast when on), compute.vastKeyFile (read into the worker's env only), compute.labelPrefix (unique per deployment on a Vast account), compute.runnerUrl, projectSecretsKeyFile (Fernet keys, the portal's env only) and weave.cliClientId (the krater CLI's public Weave app). KRATER_SKYPILOT_MODE becomes KRATER_PRICING_MODE. The portal and worker now get their own env files (web.env, worker.env) on top of app.env, since the image runs as a non-root user and can't read a root-only agenix file mounted in. alastor keeps compute off; the new secret krater-project-secrets-key.age is declared but not created. Checked by nix eval (compute off, on, and on without a key) and nixfmt. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The price-history run takes over a minute, and with one task at a time Krater's minute-by-minute compute scheduler waited behind it (seen on staging). The scheduler and reconciliation share a lock, so they never overlap. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Krater's v1 (PR #1) no longer contains SkyPilot: it runs with compute off and Krater Compute adds machines in its own release. The module drops the skypilot.* options, the SkyPilot API server, its sign-in proxy, their secrets, state and Traefik route, and skypilot-start.sh. KRATER_COMPUTE_MODE is always off; KRATER_SKYPILOT_MODE becomes KRATER_PRICING_MODE (the budget estimator's public GPU catalog, no server). alastor drops skypilot.enable. Adapted from the krater-compute branch's 021a7f9, without its compute options (compute.*, projectSecretsKeyFile, weave.cliClientId) or the per-container env files they need. Checked by nix eval of krater-alastor (agenix stubbed): the Krater units evaluate with no failed assertions; nixfmt is clean. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…e too This branch already had the same removal with Krater Compute's options; its version stays. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The default -volume.max=8 is used up by the first collections; a new bucket then can't get a volume and every upload to it fails with a 500 (found on Krater staging: screenshots after compute's checkpoints). Co-Authored-By: Claude Opus 5.5 <[email protected]>
The default -volume.max=8 is used up by the first collections; a new bucket then can't get a volume and every upload to it fails with a 500 (found on Krater staging: screenshots after compute's checkpoints). Co-Authored-By: Claude Opus 5.5 <[email protected]>
Conflicts were additive: README (repo table, the Krater section next to the Quilt ones), flake.nix (the krater modules next to edge-alastor and quilt) and secrets/secrets.nix (krater-env and krater-vast-key next to the Quilt and AD secrets). The Krater README section now says two hostnames and, with compute off, only krater-env.age. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Krater's data moves to the schema `krater` in patchwork_central on skein (modules/central-db), like the other apps. The module no longer runs a krater-db container or its own backup timer; skein's nightly dump covers it. - hosts/central-db-schema.nix: add krater (no shared schemas). - secrets: central-prod-krater.age (jsp, skein) holds the role password; krater-env.age now carries KRATER_DATABASE_URL with the same password. - bin/new-krater-secrets writes both files with new random values and prints none of them. - krater-web waits for the database with the app's own engine, and needs network-online. Checked: the alastor config evaluates; Krater's migrations reach head in a central-db-style schema on Postgres 18 with nothing in public. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…m the environment Only Jasper and alastor can decrypt krater-env.age once it exists, so whoever runs the script can pass KRATER_WEAVE_CLIENT_SECRET, KRATER_SLACK_BOT_TOKEN and KRATER_SLACK_SIGNING_SECRET in the environment instead of leaving them for Jasper. The script names the ones it left empty. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…a main Brings infra main (Quilt beside Krater, central-db, lab hosts) and the move of Krater's database to the schema `krater` in patchwork_central onto the branch with Krater Compute. Conflicts: the module keeps the compute options and the per-container web/worker env files, and drops krater-db, postgresImage and the backup timer; secrets.nix keeps both sides. The fallback rev now points at claude/application-form (16de89a) instead of PR #1's 4f130ca, so a first start without a deploy doesn't build old code. Checked: the alastor config evaluates with compute off and with compute on (the worker env alone gets the Vast key). Co-Authored-By: Claude Opus 5.5 <[email protected]>
This was referenced Oct 17, 2026
bin/new-krater-secrets: a new database password (in both files), secret key and storage keys. krater-env.age (jsp, alastor) leaves KRATER_WEAVE_CLIENT_SECRET, KRATER_SLACK_BOT_TOKEN and KRATER_SLACK_SIGNING_SECRET empty until the Weave and Slack apps exist; Jasper fills them in with agenix -e. central-prod-krater.age is for skein. .gitattributes keeps the script LF, so it runs from a Windows checkout too. Co-Authored-By: Claude Opus 5.5 <[email protected]>
2 of 3 tasks
Comment on lines
+87
to
+89
| # The `krater` CLI's public Weave app (Krater's docs/compute/cli.md). | ||
| # Empty keeps the CLI's API off. | ||
| cliClientId = ""; |
Comment on lines
+84
to
+86
| # Not secret, but only known once the app exists in Weave. The module | ||
| # refuses to build while it is empty. | ||
| clientId = ""; |
Member
There was a problem hiding this comment.
this is like... prob fine to be public
3 tasks done
… weave app exists
…ttings check, storage and cache limits
jaspermayone
approved these changes
Oct 23, 2026
1 of 2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Krater on weft: staging now, production ready and off
Krater runs on weft, its own lab VM (pve01 VMID 361, 10.100.90.61, 4 GB, 40 GB disk). One app per VM: sudo and the docker group are root, so the people who run Krater cannot read the secrets of Quilt or Weave. Adam's original description is kept below for context.
krater-staging.patchworklabs.org, Weave staging, thekraterschema inpatchwork_centralon hem.hosts/weft/krater-production.nix(enable = false) is its one switch. weft's units and nginx, alastor's routes and skein's role all read that file, so they change together.What changed on top of Adam's commits:
Merged
maintwice (hosts per app, Quilt on batting, weft bring-up feat(weft): add the weft host key, give it ad-join, and size it for krater #60). Adam's commits are unchanged.hosts/alastor/krater.nixis nowhosts/weft/krater.nix. Thekrater-alastorexport is gone.Two instances from one module.
modules/krater/instance.nixis the old module, with every name from a prefix.modules/krater/default.niximports it twice:patchwork.services.kraterpatchwork.services.krater-productionkrater-*,krater,krater:<sha>krater-prod-*,krater-prod,krater-prod:<sha>/var/lib/krater,/run/krater/var/lib/krater-prod,/run/krater-prodkrater-staging,krater-staging-storagekrater,krater-storageGwmIPZen…)extraHosts100.67.124.123)extraHosts100.109.223.51)krater-staging-env.agekrater-env.agekrater-staging-krater-prod-Each build removes old tags of its own image repository only, so staging never deletes a production image. Each store keeps 8 GB of objects at most (
storage.maxVolumes = 8): the 40 GB disk holds the system, two stores, two image repositories and the BuildKit cache.Client ids (not secret): staging
vTt8nTKFw7Qau8S2aBfhXrBQuCi1PbduyF0x6sH5QJo, productionGwmIPZenHuqRjUjawfJyDYceEcdXDJibTbmEEkuRJDU. They arestagingWeaveClientIdandproductionWeaveClientId, each next to its issuer. The production id does not turn production on: only the switch does.Staging is on in this config now: the id is set and
krater-staging-env.ageexists (Jasper set its Weave client secret, 2675e4e).Production guard: with the switch on, the weft build fails if
productionWeaveClientIdis empty. An empty key inkrater-env.agefails the unit at once (Krater's settings check).Database:
hosts/central-db-schema.nixisschemaFor <switch>, and also exportsschemaFor. krater is instagingOnlyAppswhile the switch is off, and inapps(skein) when it is on. skein's and hem's builds do not change today.Catch-all vhost on weft: an unknown Host, or a name whose instance is off, gets 444. It never falls through to the staging portal.
nginx on weft (
tailscale0:80) has the vhosts of each instance only while it is on. It passes alastor'sX-Forwarded-Proto, appends toX-Forwarded-For, and does noreal_iprewrite.KRATER_TRUSTED_PROXY_COUNTis 3 (Cloudflare, Traefik, nginx).alastor routes (
nixosModules.edge-alastor):krater-staging-weftandkrater-staging-storage-weftalways;krater-weftandkrater-storage-weftonly with the switch on (patchwork.edge.kraterProduction, default from the switch file).Other module changes:
extraHosts(--add-host), a settings check before the database wait,docker builder pruneafter builds,storage.maxVolumes, andcompute.labelPrefixrequired when compute is on. Fallback revb081a13on Krater's main.zramSwapon weft.KRATER_QUILT_URLis empty in both. Never point staging at production Quilt.Staging's Slack app is a separate app in a test workspace. Production uses the real Patchwork Slack app.
KRATER_ENVisproductionin both: Krater has no staging mode. The construction border and STAGING label need a change in the Krater app.Secrets
krater-staging-env.agecentral-staging-krater.agekrater-env.agebin/new-krater-secrets production: database URL (skein), secret key, S3 keys. Jasper fills in later:KRATER_WEAVE_CLIENT_SECRET,KRATER_SLACK_BOT_TOKEN,KRATER_SLACK_SIGNING_SECRETcentral-prod-krater.agekrater-staging-vast-key.age,krater-vast-key.age,krater-staging-project-secrets-key.age,krater-project-secrets-key.ageNo value was printed. The checks read only key names and sizes.
Staging rollout
deploy .#hem), so thekraterrole gets its password.krater-webhasrequires = [ "krater-build.service" ]and is wanted bymulti-user.target, so activation starts the build and waits for it (TimeoutStartSec = 45min). The first build takes several minutes. deploy-rs's activation timeout is shorter than that, so pick one:stagingWeaveClientId = ""(Krater off), runsudo systemctl start krater-build, then deploy again with the id. Or--magic-rollback falseand a long--activation-timeout, and watchjournalctl -u krater-build.krater-staging-env.age, if the worker must talk to Slack.krater-stagingandkrater-staging-storage, proxied, to alastor.patchwork-infrain jaspermayone/infra and deploy alastor (--magic-rollback false).deployAuthorizedKeys, and point the Krater repo deploy secrets at weft.Turn on production
productionWeaveClientId.cd secrets && agenix -e krater-env.age, and fill inKRATER_WEAVE_CLIENT_SECRET,KRATER_SLACK_BOT_TOKEN,KRATER_SLACK_SIGNING_SECRET(the real Patchwork Slack app). The build does not check them. An empty one failskrater-prod-webat once.tailscale/policy.hujson), thenecho y | tailscale/bin/apply. Add the grant:{ // weft (Krater production) -> patchwork_central on skein. "src": ["tag:patchwork-krater"], "dst": ["pw-skein", "pw-skein-6"], "ip": ["tcp:5432"], },tag:patchwork-kratertest, movepw-skein:5432fromdenytoaccept:"accept": ["pw-skein:5432", "pw-hem:5432"].enable = trueinhosts/weft/krater-production.nix. This also puts krater inappsfor skein and adds the two alastor routes.kraterrole and schema, with the password incentral-prod-krater.age), then weft (the same build wait as staging:krater-prod-build).kraterandkrater-storage, proxied, to alastor.patchwork-infrain jaspermayone/infra and deploy alastor.Checks
nixosConfigurations.weft,skein,hemandseamevaluate. On weft, staging is on and production is off. skein's apps areselvageandweaveonly. skein's and hem's system derivations are the same as before the schema change.checks.x86_64-linux.krater-weftevaluates, so all its eval-time checks pass: staging on by default, staging off when forced, production on when forced, the catch-all vhost, the edge default equal to the switch file, and the schema for both switch values.patchwork-infra.nix flake check --no-buildpasses.checks.x86_64-linux.krater-weft(its grep steps),central-dbandquilt-battingon warp, for 24d4375. Not run: Tailscale is stopped on the machine that ran the checks.Original description (alastor, production)
Adds Krater, the Project Ganymede portal, as a NixOS module (
nixosModules.krater) and its alastor settings (nixosModules.krater-alastor). alastor doesn't import it yet: that's a change injaspermayone/infraonce this merges.What it runs
krater-web: the portal, and the API rented machines report to. It waits for the database, runs migrations, then serves on loopback behind Traefik atkrater.patchworklabs.org.krater-worker: background jobs (Slack, pricing, events to Quilt, and the compute scheduler when compute is on).krater-storage: SeaweedFS's S3 gateway atkrater-storage.patchworklabs.org. It holds gallery screenshots, and machines' checkpoints and outputs when compute is on.krater-build: builds the image on alastor from one commit of patchworklabsorg/krater. There's no registry. Krater's Deploy workflow writes the commit to/var/lib/krater/source-revas thekrater-deployuser, which may run only those steps.Database: patchwork_central
Krater has no Postgres of its own. Its data is the schema
kraterinpatchwork_centralon skein, like the other central apps:krater = { }inhosts/central-db-schema.nix. Krater stores no personal data (only Patchwork IDs) and reads people through Weave, so it gets no shared schemas.search_pathiskrater, so Krater's tables, its job queue andalembic_versionall live there. Krater needs no code change: its migrations and 1075 tests pass on Postgres 18 set up this way, with nothing inpublic.Krater Compute: in the module, off on alastor
With
compute.enable, the worker rents Vast.ai machines for members' jobs. The Vast key (compute.vastKeyFile) goes into the worker's env alone. alastor keeps it off (compute.enable = false) until the Vast key is rotated: the current key was used on staging. Turning it on:secrets/krater-vast-key.age.krater-vast-keysecret inhosts/alastor/krater.nix.compute.enableandcompute.vastKeyFile.Project secrets (members' tokens for their jobs) work the same way with
krater-project-secrets-key.age.Secrets
bin/new-krater-secretswrites both new files and prints no value:krater-env.age(jsp, alastor): the env file.KRATER_DATABASE_URLholds the role's password, plusKRATER_SECRET_KEYand the storage keys. It takes the Weave client secret and the Slack values from the environment when they're set.central-prod-krater.age(jsp, skein): the role's password alone.Both files are in this PR (ce73de8; the recipients were checked against
secrets.nix).KRATER_WEAVE_CLIENT_SECRET,KRATER_SLACK_BOT_TOKENandKRATER_SLACK_SIGNING_SECRETare empty because the Weave and Slack apps don't exist yet. Krater refuses to start in production until they're filled in withcd secrets && agenix -e krater-env.age(only Jasper can).Before alastor imports it
bin/new-krater-secrets: both secrets, committedkrater-env.age(Jasper)hosts/alastor/krater.nix, and the module refuses to build while it's emptycentral-prod-krater.ageinto the central DB's password directory. skein's config doesn't load any app's password file yet, so Krater's role can't log in until it does.kraterandkrater-storage(patchworklabsorg/dns, linked below)deployAuthorizedKeys, and theALASTOR_HOSTandKRATER_DEPLOY_SSH_KEYsecrets in the Krater repoKRATER_TRUSTED_PROXY_COUNT: 2 if alastor's Traefik trusts Cloudflare's ranges inforwardedHeaders.trustedIPsChecked
krater-build,krater-network,krater-storage,krater-webandkrater-worker, and there are no timers.bin/new-krater-secretsran innixos/nixwith throwaway keys. Both files decrypt, and it refuses to overwrite either one.The fallback
revis Krater'sclaude/application-form(16de89a). After patchworklabsorg/krater#9 merges, set it to the merge commit.This replaces the branches
krater-moduleandkrater-compute-off.