Skip to content

feat(alastor): post to Slack when Krater stops answering the edge ping - #112

Open
Adambomb210 wants to merge 1 commit into
mainfrom
claude/edge-ping-alert
Open

Adambomb210 wants to merge 1 commit into
mainfrom
claude/edge-ping-alert

Conversation

@Adambomb210

Copy link
Copy Markdown
Collaborator

Krater posts its own alerts, so when weft or Krater is down nothing says so. alastor already pings /healthz/edge every minute; this makes that ping alert.

  • krater-edge-ping keeps each URL's failed minutes in its StateDirectory.
  • It posts to a Slack incoming webhook after 3 failed minutes in a row, then every 60 minutes while the failure lasts, and once when the URL answers again. A failed Slack post doesn't fail the unit.
  • It is off until secrets/krater-edge-alert.age exists. This PR adds that file's recipients (jsp, alastor) to secrets.nix. To turn it on, make the webhook (Slack app → Incoming Webhooks → the ops channel), then:
    cd secrets && agenix -e krater-edge-alert.age   # paste the URL alone on one line
    
    Commit the file. The option defaults to its agenix path once it's there.
  • The age.secrets entry is declared only where agenix is (the edge test's bare system has none). It sits inside config, so reading options can't recurse.

Tested:

  • nix build .#checks.x86_64-linux.krater-weft passes from a clean clone, with two new assertions (off without a webhook; reads the credential when set).
  • The rendered script ran 64 failing minutes, then recovery, against a fake curl. It posted one alert per host at 3 minutes, one reminder at 63, and one recovery post.

Not covered: if alastor itself is down, nothing alerts.

🤖 Generated with Claude Code

krater-edge-ping counts each URL's failed minutes in its StateDirectory and,
with a Slack incoming-webhook URL, posts after 3 in a row, every 60 while it
lasts, and once when it answers again. Krater posts its own alerts, so it
can't say it's down; this can.

On once secrets/krater-edge-alert.age exists (recipients added to
secrets.nix); until then nothing changes. The secret is declared only where
agenix is, inside `config` so reading `options` can't recurse.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant