Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
# Run on Linux even from a Windows checkout (core.autocrlf would turn them CRLF).
bin/new-krater-secrets text eol=lf
bin/new-stash-secrets text eol=lf
8 changes: 7 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ Declarative infrastructure for Patchwork Labs: every Patchwork host, the NixOS m
| `modules/weave/` | The Weave service module. |
| `modules/quilt/` | The Quilt service module. |
| `modules/krater/` | The Krater service module (portal, worker, screenshot storage; Krater Compute off by default). Exported as `nixosModules.krater`. |
| `modules/stash/` | The Stash service module (web, worker, archived files on a host directory). |
| `hosts/hank/stash.nix` | Stash production and staging on hank: instance settings and secrets. |
| `bin/new-stash-secrets` | Writes the two secrets of Stash staging or production with a new random database password and `SECRET_KEY_BASE`. Prints no value. |
| `hosts/weft/krater.nix` | Krater staging and production on weft: instance settings and secrets. Production is off until `hosts/weft/krater-production.nix` turns it on. |
| `tests/` | NixOS VM tests. |
| `secrets/` | agenix secrets. `secrets.nix` lists the recipients of each file. |
Expand All @@ -46,6 +49,7 @@ The lab hosts below are NixOS VMs on pve01, VLAN 90 (`patchwork.internal`). alas
| `hem` | 358 | 10.100.90.58 | 2 GB | Both staging clusters: central on 5432, restricted on 5433. Quilt staging uses the restricted one. |
| `batting` | 360 | 10.100.90.60 | 4 GB | Quilt prod + staging. nginx on `tailscale0:80` for alastor. |
| `weft` | 361 | 10.100.90.61 | 4 GB | Krater staging, and Krater production (off until its switch). nginx on `tailscale0:80` for alastor. |
| `hank` | 362 | 10.100.90.62 | 6 GB | Stash prod + staging (declared, not built yet). 80 GB disk for the archived files. nginx on `tailscale0:80` for alastor. |

The Windows hosts are managed by Ansible (`ansible/`):

Expand All @@ -64,10 +68,12 @@ flowchart LR
alastor -- "tailnet, HTTP :80" --> warp["warp: misc"]
alastor -- "tailnet, HTTP :80" --> batting["batting: Quilt"]
alastor -- "tailnet, HTTP :80" --> weft["weft: Krater staging"]
alastor -- "tailnet, HTTP :80" --> hank["hank: Stash (not built yet)"]
batting -- "tailnet :5432" --> seam["seam: patchwork_central_restricted"]
batting -. "staging :5433" .-> hem["hem: :5432 / :5433"]
weft -. "staging :5432" .-> hem
skein["skein: patchwork_central (no app connects yet)"]
hank -- "tailnet :5432" --> skein["skein: patchwork_central"]
hank -. "staging :5432" .-> hem
```

## Login
Expand Down
4 changes: 2 additions & 2 deletions ad/TIERING.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ The rule: an account of tier N logs on only to assets of tier N. A higher tier c
| Tier | Assets | Accounts |
|---|---|---|
| **0** | loom, spindle (DCs). bobbin (issuing CA). pattern (offline root CA). shuttle (NPS). The Proxmox hosts that run these VMs. The proposed Tier 0 jump host. remus, as long as it holds the `svc-ansible` password. | `adm_mayonej`, `svc-ansible`, `Administrator` (disabled), DSRM (LAPS-DSRM) |
| **1** | warp, batting, weft, skein, seam, hem (NixOS). thimble (VyOS edge). App servers. Weave and the host that runs it (alastor). | `lnx-<host>-admins` members. `svc-lnx-join`, `svc-pve-ldap`, `svc-dhcp-dns`, `svc-weave-ldap`. No dedicated Tier 1 person account yet. |
| **1** | warp, batting, weft, hank, skein, seam, hem (NixOS). thimble (VyOS edge). App servers. Weave and the host that runs it (alastor). | `lnx-<host>-admins` members. `svc-lnx-join`, `svc-pve-ldap`, `svc-dhcp-dns`, `svc-weave-ldap`. No dedicated Tier 1 person account yet. |
| **2** | Staff workstations (`Computers` OU). VPN sessions. | Daily accounts: `mayonej`, staff, members |

```mermaid
Expand All @@ -113,7 +113,7 @@ flowchart TB
bg["Administrator (disabled, break-glass)"]
end
subgraph T1["Tier 1: servers"]
nix["warp, batting, weft, skein, seam, hem"]
nix["warp, batting, weft, hank, skein, seam, hem"]
thimble["thimble VPN edge"]
weave["Weave on alastor, svc-weave-ldap"]
end
Expand Down
6 changes: 4 additions & 2 deletions ad/org.json
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@
{ "name": "secretary", "formerNames": ["GG-Secretary"], "description": "Secretary", "members": ["klukask"] },
{ "name": "ganymede-admins", "formerNames": ["GG-Ganymede-Admins"], "description": "Project Ganymede admins", "members": ["mayonej", "adam.sidwa"] },
{ "name": "weft-admins", "description": "Root on weft (Krater): logon and sudo on weft only, through hostGroupMembers. Sudo and the docker group on weft give every Krater secret.", "members": ["mayonej", "adam.sidwa"] },
{ "name": "hank-admins", "description": "Root on hank (Stash): logon and sudo on hank only, through hostGroupMembers. Sudo and the docker group on hank give every Stash secret and every archived file.", "members": ["mayonej"] },
{ "name": "ganymede-users", "formerNames": ["GG-Ganymede-Users"], "description": "Project Ganymede users", "members": [], "manageMembers": false },
{ "name": "hackboston-organizers", "formerNames": ["GG-HackBoston-Organizers"], "description": "HackBoston organizers", "members": ["cj", "daniel", "logan", "mayonej"] },
{ "name": "hackathonhelp-organizers", "formerNames": ["GG-HackathonHelp-Organizers"], "description": "hackathon.help organizers", "members": ["wei"] },
Expand All @@ -93,14 +94,15 @@
"members": [] }
],

"linuxHosts": ["batting", "hem", "seam", "skein", "warp", "weft"],
"linuxHosts": ["batting", "hank", "hem", "seam", "skein", "warp", "weft"],
"retiredHosts": ["sampler", "swatch"],
"hostGroups": [
{ "name": "lnx-{host}-users", "description": "Linux logon on {host}", "members": ["infra"] },
{ "name": "lnx-{host}-admins", "description": "Linux sudo on {host}", "members": ["infra"] }
],
"hostGroupMembers": {
"weft": { "users": ["weft-admins"], "admins": ["weft-admins"] }
"weft": { "users": ["weft-admins"], "admins": ["weft-admins"] },
"hank": { "users": ["hank-admins"], "admins": ["hank-admins"] }
},
"resourceGroups": [
{ "name": "password-reset", "formerNames": ["DL-Users-PasswordReset"], "description": "Reset password, unlock, change at next logon: Users/Staff, Board-Only, Sponsored-Projects",
Expand Down
4 changes: 2 additions & 2 deletions ansible/inventory/hosts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# ============================================================================
# Patchwork hosts that Ansible manages: the Windows servers only.
# ---------------------------------------------------------------------------
# The NixOS hosts (warp, batting, weft, skein, seam, hem) are declared in ../flake.nix. They
# The NixOS hosts (warp, batting, weft, hank, skein, seam, hem) are declared in ../flake.nix. They
# join AD through modules/ad-join, not through Ansible. Do not add them here.
# ad.yml -t host-groups reads their names from the flake.
#
Expand Down Expand Up @@ -49,6 +49,6 @@ all:
#
# warp, batting, NixOS, declared in ../flake.nix. Joined to AD by
# weft, skein, modules/ad-join.
# seam, hem
# seam, hem, hank
# pattern VMID 399. Offline root CA. Stopped. Never a target.
# ========================================================================
109 changes: 109 additions & 0 deletions bin/new-stash-secrets
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
#!/usr/bin/env bash
# Write one Stash instance's two secrets with new random values:
# staging secrets/central-staging-stash.age the password of the role
# stash in patchwork_central
# on hem (password only)
# secrets/stash-staging-env.age Stash staging's env file
# production secrets/central-prod-stash.age the same on skein
# secrets/stash-env.age Stash production's env file
# Both files get the same password. The env file also gets a new
# SECRET_KEY_BASE. WEAVE_CLIENT_ID, WEAVE_CLIENT_SECRET and GITHUB_TOKEN come
# from the environment when they are set there, e.g.
# set -a; . ./stash-third-party.env; set +a; bin/new-stash-secrets staging
# and are left empty otherwise. Only the file's recipients (Jasper, hank) can
# fill them in later, with
# cd secrets && agenix -e stash-staging-env.age
# The issuer and the hostnames are not secret: hosts/hank/stash.nix sets
# staging's, and production's are in the env file as plain values.
#
# Usage: bin/new-stash-secrets staging|production [database host]
# The database host defaults to hem's (staging) or skein's (production)
# tailnet name, for a Stash host that reaches it over Tailscale (hank, with
# the name pinned in networking.hosts).
# Needs age, nix and openssl. The script never prints a value, and it stops if
# one of the files exists. The recipients come from secrets/secrets.nix: the
# script encrypts with `age` itself, because `agenix -e` from a pipe fails on
# macOS ("cp: skipping file '/dev/stdin'") and reports nothing.
set -euo pipefail

case "${1:-}" in
staging)
db_file=central-staging-stash.age
env_file=stash-staging-env.age
db_host="${2:-hem.wildebeest-stargazer.ts.net}"
hosts=stash-staging.patchworklabs.org
issuer=https://weave-staging.patchworklabs.org
;;
production)
db_file=central-prod-stash.age
env_file=stash-env.age
db_host="${2:-skein.wildebeest-stargazer.ts.net}"
hosts=stash.patchworklabs.org
issuer=https://weave.patchworklabs.org
;;
*)
echo "Usage: $0 staging|production [database host]" >&2
exit 1
;;
esac

root="$(cd "$(dirname "$0")/.." && pwd)"
cd "$root/secrets"

for tool in age nix openssl; do
if ! command -v "$tool" >/dev/null; then
echo "$tool is not on PATH. Run this script in a shell that has it." >&2
exit 1
fi
done

# Encrypts stdin to the recipients that secrets.nix lists for the file.
encrypt() {
local file="$1" recipients
recipients="$(nix eval --raw -f secrets.nix "\"$file\".publicKeys" --apply 'keys: builtins.concatStringsSep "\n" keys')"
if [ -z "$recipients" ]; then
echo "secrets.nix lists no recipients for $file. Stop." >&2
exit 1
fi
age --encrypt --recipients-file <(printf '%s\n' "$recipients") --output "$file"
chmod 644 "$file"
}

for file in "$db_file" "$env_file"; do
if [ -e "$file" ]; then
echo "secrets/$file exists. Stop. To change a value, use: agenix -e $file" >&2
exit 1
fi
done

umask 077
# Hex only, so the password needs no escaping in the URL.
password="$(openssl rand -hex 32)"
secret_key_base="$(openssl rand -hex 64)"

# printf is a shell builtin, so the values are not in any process's arguments.
printf '%s' "$password" | encrypt "$db_file"

{
printf '%s\n' "CENTRAL_DATABASE_URL=postgres://stash:${password}@${db_host}:5432/patchwork_central?sslmode=disable"
printf '%s\n' "SECRET_KEY_BASE=${secret_key_base}"
printf '%s\n' "STASH_HOSTS=${hosts}"
printf '%s\n' "WEAVE_ISSUER=${issuer}"
printf '%s\n' "WEAVE_CLIENT_ID=${WEAVE_CLIENT_ID:-}"
printf '%s\n' "WEAVE_CLIENT_SECRET=${WEAVE_CLIENT_SECRET:-}"
printf '%s\n' "GITHUB_TOKEN=${GITHUB_TOKEN:-}"
} | encrypt "$env_file"

unset password secret_key_base

echo "Wrote secrets/$db_file and secrets/$env_file."
missing=""
for key in WEAVE_CLIENT_ID WEAVE_CLIENT_SECRET; do
if [ -z "${!key:-}" ]; then
missing="$missing $key"
fi
done
if [ -n "$missing" ]; then
echo "Left empty:$missing"
echo "Jasper fills them in with: cd secrets && agenix -e $env_file"
fi
13 changes: 12 additions & 1 deletion flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
"warp" # misc: redirects and other low-risk shared things
"batting" # Quilt
"weft" # Krater
"hank" # Stash
"skein" # patchwork_central, prod
"seam" # patchwork_central_restricted, prod
"hem" # both staging clusters
Expand Down Expand Up @@ -79,6 +80,9 @@
# The Krater service module alone, with no instance settings. weft
# imports it through hosts/weft/krater.nix.
krater = ./modules/krater;
# The Stash service module alone, with no instance settings. hank
# imports it through hosts/hank/stash.nix.
stash = ./modules/stash;
};

nixosConfigurations = nixpkgs.lib.genAttrs hosts mkHost;
Expand Down Expand Up @@ -136,7 +140,12 @@
ansible-filters =
pkgs.runCommand "ansible-filters"
{
nativeBuildInputs = [ (pkgs.python3.withPackages (p: [ p.jinja2 p.pyyaml ])) ];
nativeBuildInputs = [
(pkgs.python3.withPackages (p: [
p.jinja2
p.pyyaml
]))
];
src = ./ansible;
}
''
Expand Down Expand Up @@ -193,6 +202,8 @@
# agenix -e / -r, and bin/new-quilt-staging-secrets.
agenix.packages.${system}.default
pkgs.openssl
# bin/new-stash-secrets encrypts with age itself.
pkgs.age
];
};
}
Expand Down
9 changes: 9 additions & 0 deletions hosts/alastor/edge.nix
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,15 @@ let
backend = "weft";
hosts = [ "krater-staging-storage.patchworklabs.org" ];
};
# Stash on hank (../hank/stash.nix): production and staging.
stash-hank = {
backend = "hank";
hosts = [ "stash.patchworklabs.org" ];
};
stash-staging-hank = {
backend = "hank";
hosts = [ "stash-staging.patchworklabs.org" ];
};
# Every host in ../warp/redirects.nix. A new path on an existing host
# needs no change here.
redirects = {
Expand Down
3 changes: 3 additions & 0 deletions hosts/central-db-schema.nix
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ let
# Weave is the IDP, so it owns people.* and runs its migrations.
weave = { };
selvage.read = [ "people" ];
# Stash (the archive): archives, captures, artifacts, audit events, GoodJob, Active Storage
# rows. It stores requesters' Patchwork IDs only, so no shared schemas.
stash = { };
}
// (if kraterProduction then { inherit krater; } else { });
# Apps on staging (hem) only, until they have a production instance. skein
Expand Down
114 changes: 114 additions & 0 deletions hosts/hank/configuration.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# hank — Stash host (pve01 VMID 362, 10.100.90.62, 6 GB, 80 GB disk)
#
# One app per VM. Sudo and the docker group are root here, so only the
# break-glass admins (modules/common) and lnx-hank-admins get them. Only this
# host's key decrypts the Stash secrets.
# Public traffic arrives from alastor over the tailnet
# (nixosModules.edge-alastor) as plain HTTP to nginx on tailscale0:80.
#
# Tailnet tags: tag:patchwork (ssh from admins, port 80 from alastor) and
# tag:patchwork-stash. The policy in jaspermayone/infra must let
# tag:patchwork-stash reach skein:5432 (prod) and hem:5432 (staging) only.
#
# The disk holds the archive (/data/stash and /data/stash-staging): 80 GB,
# more than the other app VMs. Memory: Chromium and LibreOffice run in the
# worker container, with a 3 GB limit.
#
# Built on 2026-11-08 with nixos-anywhere from the installer ISO. Its host key
# is in secrets/secrets.nix.
{ config, ... }:
{
imports = [
../proxmox-vm.nix
../../modules/common
./stash.nix
];

networking.hostName = "hank";
patchwork.lanAddress = "10.100.90.62";
patchwork.banner.role = "stash host";

virtualisation.docker.enable = true;

# The containers run as uid 1000 (`rails` in the image) and own /data/stash.
# Reserve that uid for a system account here, so NixOS gives the local login
# accounts (modules/common) 1001 and up, and no person owns the archive.
users.users.stash = {
uid = 1000;
group = "stash";
isSystemUser = true;
description = "Stash containers (uid of the image's rails user)";
};
users.groups.stash.gid = 1000;

# The lab hosts do not use MagicDNS (modules/common, --accept-dns=false),
# so *.ts.net names do not resolve here. The Stash env files connect to
# skein and hem by tailnet name. The containers use the host network, and
# Docker copies this /etc/hosts into each one at start. If skein or hem
# joins the tailnet again, its address changes: update it here (and the
# aliases in jaspermayone/infra tailscale/policy.hujson). Addresses from
# `tailscale status` on 2026-11-08.
networking.hosts = {
"100.109.223.51" = [ "skein.wildebeest-stargazer.ts.net" ];
"100.67.124.123" = [ "hem.wildebeest-stargazer.ts.net" ];
};

zramSwap.enable = true;

services.nginx = {
enable = true;
# Only alastor reaches this, over the tailnet. alastor terminated TLS and
# set X-Forwarded-For and X-Forwarded-Proto (https); pass them on as they
# are. recommendedProxySettings would set X-Forwarded-Proto to http.
commonHttpConfig = ''
set_real_ip_from 100.64.0.0/10;
real_ip_header X-Forwarded-For;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header X-Forwarded-Host $host;
'';
virtualHosts = {
stash = {
serverName = "stash.patchworklabs.org";
listen = [
{
addr = "0.0.0.0";
port = 80;
}
];
locations."/" = {
proxyPass = "http://127.0.0.1:${toString config.patchwork.services.stash.port}";
# Artifact downloads redirect to a signed URL served by the same app:
# a git bundle or a video can be large, and slow.
extraConfig = ''
proxy_read_timeout 300s;
proxy_buffering off;
'';
};
};
stash-staging = {
serverName = "stash-staging.patchworklabs.org";
listen = [
{
addr = "0.0.0.0";
port = 80;
}
];
locations."/" = {
proxyPass = "http://127.0.0.1:${toString config.patchwork.services.stash.staging.port}";
extraConfig = ''
proxy_read_timeout 300s;
proxy_buffering off;
'';
};
};
};
};
# nginx is a host service, so the NixOS firewall applies to it (Docker's
# published ports would bypass it). Open only on the tailnet.
networking.firewall.interfaces.tailscale0.allowedTCPPorts = [ 80 ];

system.stateVersion = "26.05";
}
Loading
Loading