Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# Run on Linux even from a Windows checkout (core.autocrlf would turn them CRLF).
bin/new-krater-secrets text eol=lf
bin/new-stash-secrets text eol=lf
bin/new-selvage-secrets text eol=lf
65 changes: 65 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ Declarative infrastructure for Patchwork Labs: every Patchwork host, the NixOS m
| `modules/stash/` | The Stash service module (web, worker, archived files on a host directory). |
| `hosts/hank/stash.nix` | Stash production and staging on hank: instance settings and secrets. |
| `bin/new-stash-secrets` | Writes the two secrets of Stash staging or production with a new random database password and `SECRET_KEY_BASE`. Prints no value. |
| `modules/selvage/` | The Selvage service module (web and worker, production only). |
| `hosts/twill/selvage.nix` | Selvage production on twill: instance settings and secrets. |
| `bin/new-selvage-secrets` | Writes Selvage's restricted database password and env file. Reads the existing central password with your age identity. Prints no value. |
| `bin/set-env-secret` | Sets keys in an agenix env file, for example a Weave client id and secret. Asks for each value with the echo off. Prints no value. |
| `hosts/weft/krater.nix` | Krater staging and production on weft: instance settings and secrets. Production is off until `hosts/weft/krater-production.nix` turns it on. |
| `tests/` | NixOS VM tests. |
Expand All @@ -51,6 +54,7 @@ The lab hosts below are NixOS VMs on pve01, VLAN 90 (`patchwork.internal`). alas
| `batting` | 360 | 10.100.90.60 | 4 GB | Quilt prod + staging. nginx on `tailscale0:80` for alastor. |
| `weft` | 361 | 10.100.90.61 | 4 GB | Krater staging, and Krater production (off until its switch). nginx on `tailscale0:80` for alastor. |
| `hank` | 362 | 10.100.90.62 | 6 GB | Stash prod + staging (declared, not built yet). 80 GB disk for the archived files. nginx on `tailscale0:80` for alastor. |
| `twill` | 363 | 10.100.90.63 | 3 GB | Selvage prod (declared, not built yet). No staging yet. nginx on `tailscale0:80` for alastor, `/slack/` and `/up` only. |

The Windows hosts are managed by Ansible (`ansible/`):

Expand All @@ -75,6 +79,9 @@ flowchart LR
weft -. "staging :5432" .-> hem
hank -- "tailnet :5432" --> skein["skein: patchwork_central"]
hank -. "staging :5432" .-> hem
alastor -- "tailnet, HTTP :80" --> twill["twill: Selvage (not built yet)"]
twill -- "tailnet :5432" --> skein
twill -- "tailnet :5432" --> seam
```

## Login
Expand Down Expand Up @@ -304,6 +311,64 @@ Do the steps in this order.

To test top-ups on staging, add a sandbox `MERCURY_WRITE_TOKEN` with batting's public IP on its allowlist, and set `QUILT_TOPUPS` to `approval_only`.

## Selvage on twill

[Selvage](https://github.com/patchworklabsorg/selvage), the Patchwork Slack bot, runs on twill as two Docker containers, like Stash on hank. `hosts/twill/selvage.nix` is the instance. `modules/selvage` is the service module. Production only: there is no Selvage staging instance yet.

| Item | Value |
|---|---|
| URL | `https://selvage.patchworklabs.org` (router `selvage-twill` in `edge-alastor`). nginx on twill passes only `POST /slack/*` and `GET /up`. Every other path is 404. |
| Units | `selvage-web` (`./bin/rails db:migrate` in a one-off container, then `./bin/rails server` on `127.0.0.1:3009`) and `selvage-worker` (`bundle exec good_job start`, on the web container's image) |
| Image | `ghcr.io/patchworklabsorg/selvage:main` (option `image`). Pin it to `:<sha>` for a known build. |
| Databases | `patchwork_central` on skein and `patchwork_central_restricted` on seam. Role and schema `selvage` in each. |
| Secrets | `selvage-env.age`, `selvage-ghcr-token.age` (twill), `central-prod-selvage.age` (skein), `restricted-prod-selvage.age` (seam) |
| Tailnet | `tag:patchwork` and `tag:patchwork-selvage`. The second tag reaches skein:5432 and seam:5432 only. |

```mermaid
flowchart LR
slack((Slack)) --> traefik["alastor: Traefik"]
traefik -- "tailnet :80" --> nginx["twill: nginx, /slack/ and /up only"]
nginx -- "127.0.0.1:3009" --> web[selvage-web]
worker[selvage-worker] -- "waits for /up" --> web
web -- "tailnet :5432" --> skein["skein: patchwork_central, schema selvage"]
web -- "tailnet :5432" --> seam["seam: patchwork_central_restricted, schema selvage"]
worker --> skein
worker --> seam
```

The env file has `CENTRAL_DATABASE_URL` and `RESTRICTED_DATABASE_URL`. Their passwords must be the values in `central-prod-selvage.age` and `restricted-prod-selvage.age`. skein and seam set those values on the role `selvage`. The role has no `CREATE` on either database. `selvage-web` runs `db:migrate`, never `db:prepare`, and the migrations must not run `CREATE SCHEMA` or `enable_extension`.

The units do not start while `CENTRAL_DATABASE_URL`, `RESTRICTED_DATABASE_URL`, `SECRET_KEY_BASE`, `SLACK_SIGNING_SECRET` or `SLACK_BOT_TOKEN` is missing or empty in the env file (option `requiredEnv`). An empty signing secret would let anyone sign a Slack request.

To deploy a new `main` image, run `sudo systemctl restart selvage-web` on twill. That restarts the worker too.

### Go live

Do the steps in this order. Steps 5 to 11 are on this repo's branch, before the merge.

1. Merge the tailnet policy change in `jaspermayone/infra` (`tag:patchwork-selvage`, with grants to skein:5432 and seam:5432).
2. Merge the DNS record `selvage` (proxied to alastor) in `patchworklabsorg/dns`.
3. In `patchworklabsorg/selvage`, merge the image workflow and make sure that `image.yml` pushed `ghcr.io/patchworklabsorg/selvage:main`. The migrations must not run `CREATE SCHEMA` or `enable_extension`.
4. Make sure that VMID 363 and `10.100.90.63` are free, and that pve01 has 3 GiB of memory to spare (`terraform/proxmox/README.md`, "Memory warning").
5. Prepare AD for twill: run `.\converge.ps1` (it creates `twill-admins`), then `cd ansible && ansible-playbook ad.yml -t host-groups`. See "Login".
6. Create and install the VM: `bin/install-host twill`.
7. On twill, join the tailnet with both tags: `sudo tailscale up --advertise-tags=tag:patchwork,tag:patchwork-selvage`.
8. Get the host key: `ssh-keyscan -t ed25519 10.100.90.63`. Add it as `twill` in `secrets/secrets.nix`. Add `twill` to `selvage-env.age`, `selvage-ghcr-token.age` and `ad-join.age`. Remove the "not built yet" notes.
9. Write the secrets: `nix develop -c bin/new-selvage-secrets`. It writes `restricted-prod-selvage.age`, `selvage-env.age` and `selvage-ghcr-token.age`. It decrypts `central-prod-selvage.age` and `quilt-ghcr-token.age` with your age identity (`~/.ssh/id_ed25519`, or `AGE_IDENTITY`).
10. Re-encrypt only `ad-join.age`, so that twill can decrypt it. Do not run `agenix -r`: it re-encrypts every file, and the hash-based restart triggers then restart Stash and Krater with no change.

```bash
cd secrets
age -d -i ~/.ssh/id_ed25519 ad-join.age | age -e -o ad-join.age.new \
-R <(nix eval --raw -f secrets.nix '"ad-join.age".publicKeys' --apply 'k: builtins.concatStringsSep "\n" k')
mv ad-join.age.new ad-join.age
```
11. Fill in the Slack and Weave values. Selvage does not start without `SLACK_SIGNING_SECRET` and `SLACK_BOT_TOKEN`: `bin/set-env-secret selvage-env.age SLACK_SIGNING_SECRET SLACK_BOT_TOKEN SLACK_ADMIN_TOKEN WEAVE_CLIENT_ID WEAVE_CLIENT_SECRET SELVAGE_AUDIT_CHANNEL`. Add `SLACK_BROWSER_TOKEN` and `SLACK_BROWSER_COOKIE` the same way when you need them.
12. Commit the `.age` files and `secrets.nix`, and merge after CI passes. The lab hosts follow `main`, so seam creates the role `selvage` with its password, and twill starts Selvage. To do it at once: `deploy .#seam`, then `deploy .#twill`.
13. On seam, make sure that `central-db-setup` succeeded: `journalctl -u central-db-setup`. On twill, check `systemctl status selvage-web selvage-worker` and `curl -i http://127.0.0.1:3009/up`.
14. Route the public name: in `jaspermayone/infra`, run `nix flake update patchwork-infra`, then `deploy .#alastor`.
15. Point the Slack app's request URLs at `https://selvage.patchworklabs.org/slack/events`, `/slack/commands` and `/slack/interactions`.

## Commands

```bash
Expand Down
4 changes: 2 additions & 2 deletions ad/TIERING.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ The rule: an account of tier N logs on only to assets of tier N. A higher tier c
| Tier | Assets | Accounts |
|---|---|---|
| **0** | loom, spindle (DCs). bobbin (issuing CA). pattern (offline root CA). shuttle (NPS). The Proxmox hosts that run these VMs. The proposed Tier 0 jump host. remus, as long as it holds the `svc-ansible` password. | `adm_mayonej`, `svc-ansible`, `Administrator` (disabled), DSRM (LAPS-DSRM) |
| **1** | warp, batting, weft, hank, skein, seam, hem (NixOS). thimble (VyOS edge). App servers. Weave and the host that runs it (alastor). | `lnx-<host>-admins` members. `svc-lnx-join`, `svc-pve-ldap`, `svc-dhcp-dns`, `svc-weave-ldap`. No dedicated Tier 1 person account yet. |
| **1** | warp, batting, weft, hank, twill, skein, seam, hem (NixOS). thimble (VyOS edge). App servers. Weave and the host that runs it (alastor). | `lnx-<host>-admins` members. `svc-lnx-join`, `svc-pve-ldap`, `svc-dhcp-dns`, `svc-weave-ldap`. No dedicated Tier 1 person account yet. |
| **2** | Staff workstations (`Computers` OU). VPN sessions. | Daily accounts: `mayonej`, staff, members |

```mermaid
Expand All @@ -113,7 +113,7 @@ flowchart TB
bg["Administrator (disabled, break-glass)"]
end
subgraph T1["Tier 1: servers"]
nix["warp, batting, weft, hank, skein, seam, hem"]
nix["warp, batting, weft, hank, twill, skein, seam, hem"]
thimble["thimble VPN edge"]
weave["Weave on alastor, svc-weave-ldap"]
end
Expand Down
6 changes: 4 additions & 2 deletions ad/org.json
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@
{ "name": "ganymede-admins", "formerNames": ["GG-Ganymede-Admins"], "description": "Project Ganymede admins", "members": ["mayonej", "adam.sidwa"] },
{ "name": "weft-admins", "description": "Root on weft (Krater): logon and sudo on weft only, through hostGroupMembers. Sudo and the docker group on weft give every Krater secret.", "members": ["mayonej", "adam.sidwa"] },
{ "name": "hank-admins", "description": "Root on hank (Stash): logon and sudo on hank only, through hostGroupMembers. Sudo and the docker group on hank give every Stash secret and every archived file.", "members": ["mayonej"] },
{ "name": "twill-admins", "description": "Root on twill (Selvage): logon and sudo on twill only, through hostGroupMembers. Sudo and the docker group on twill give every Selvage secret, including its Slack admin token.", "members": ["mayonej"] },
{ "name": "ganymede-users", "formerNames": ["GG-Ganymede-Users"], "description": "Project Ganymede users", "members": [], "manageMembers": false },
{ "name": "hackboston-organizers", "formerNames": ["GG-HackBoston-Organizers"], "description": "HackBoston organizers", "members": ["cj", "daniel", "logan", "mayonej"] },
{ "name": "hackathonhelp-organizers", "formerNames": ["GG-HackathonHelp-Organizers"], "description": "hackathon.help organizers", "members": ["wei"] },
Expand All @@ -94,15 +95,16 @@
"members": [] }
],

"linuxHosts": ["batting", "hank", "hem", "seam", "skein", "warp", "weft"],
"linuxHosts": ["batting", "hank", "hem", "seam", "skein", "twill", "warp", "weft"],
"retiredHosts": ["sampler", "swatch"],
"hostGroups": [
{ "name": "lnx-{host}-users", "description": "Linux logon on {host}", "members": ["infra"] },
{ "name": "lnx-{host}-admins", "description": "Linux sudo on {host}", "members": ["infra"] }
],
"hostGroupMembers": {
"weft": { "users": ["weft-admins"], "admins": ["weft-admins"] },
"hank": { "users": ["hank-admins"], "admins": ["hank-admins"] }
"hank": { "users": ["hank-admins"], "admins": ["hank-admins"] },
"twill": { "users": ["twill-admins"], "admins": ["twill-admins"] }
},
"resourceGroups": [
{ "name": "password-reset", "formerNames": ["DL-Users-PasswordReset"], "description": "Reset password, unlock, change at next logon: Users/Staff, Board-Only, Sponsored-Projects",
Expand Down
5 changes: 3 additions & 2 deletions ansible/inventory/hosts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# ============================================================================
# Patchwork hosts that Ansible manages: the Windows servers only.
# ---------------------------------------------------------------------------
# The NixOS hosts (warp, batting, weft, hank, skein, seam, hem) are declared in ../flake.nix. They
# The NixOS hosts (warp, batting, weft, hank, twill, skein, seam, hem) are declared in ../flake.nix. They
# join AD through modules/ad-join, not through Ansible. Do not add them here.
# ad.yml -t host-groups reads their names from the flake.
#
Expand Down Expand Up @@ -49,6 +49,7 @@ all:
#
# warp, batting, NixOS, declared in ../flake.nix. Joined to AD by
# weft, skein, modules/ad-join.
# seam, hem, hank
# seam, hem, hank,
# twill
# pattern VMID 399. Offline root CA. Stopped. Never a target.
# ========================================================================
Loading
Loading