Skip to content

Polish batch 2: image checks, change limits, checkpoint warnings, compare extras, update notice - #99

Merged
Adambomb210 merged 1 commit into
mainfrom
claude/polish-batch-2
Nov 19, 2026
Merged

Adambomb210 merged 1 commit into
mainfrom
claude/polish-batch-2

Conversation

@Adambomb210

Copy link
Copy Markdown
Collaborator

Stacked on #98 (base claude/polish-batch). Once #98 merges, GitHub retargets this to main.

Image checks (krater.services.image_checks; docs in docs/compute/images.md)

  • Refused at submit and estimate:
    • images with no linux/amd64 variant (an arm64 build from a Mac), with the buildx --platform fix
    • images whose compressed size is bigger than the disk
  • Warned, on the form, in the API's warnings and in the CLI, and kept as image_warning events:
    • RTX 50-series GPUs with CUDA older than 12.8
    • images that may not fit unpacked
    • images over 10 GB, so slow starts; the provision-timeout message names the size too
    • pip refused by PEP 668
    • building CUDA code on an image with no toolkit
    • no Python
    • apt as a non-root user
  • The image's CUDA major version is stored on the job, and offers need cuda_max_good at least that.
  • Inspections are cached by digest in image_facts (new migration, additive). Docker Hub doesn't count the HEAD.
  • The runner sets PYTHONUNBUFFERED=1 unless the image or a secret sets it.
  • --code checks that python exists, installs with PIP_BREAK_SYSTEM_PACKAGES=1, and warns when the folder builds CUDA on a non-devel image.

Change limits

  • Change a queued or running job's max runtime and spend cap: the job page's Change limits, POST /api/v1/jobs/{id}/limits, or krater limits.
  • A higher cap raises the budget hold as far as the project's free budget allows.
  • Resuming or moving a box already at its cap is refused until its cap is raised. Before, it stopped again at once.

Checkpoint watch

  • A periodic task every 15 minutes posts to the project's Slack channel when a running interruptible job hasn't stored a checkpoint for two upload intervals. It posts once per attempt and mentions the submitter.
  • The job page uses the same rule, which now counts from the current machine's start after a move.

CLI (0.8.0)

  • compare adds state, cost, runtime and GPU rows, and --sweep NAME --project P.
  • Every API answer carries X-Krater-CLI-Latest. When it's newer than the CLI, the CLI prints how to update, to stderr and at most once a day. KRATER_NO_UPDATE_NOTICE=1 turns this off.

Ops alerts

  • Grouped network-outage alerts are now one network_outage alert. It opens once, is reminded like any other alert, and clears with a single post. The alerts it covers stay quiet.

The alastor uptime alert is patchworklabsorg/infra#112.

Tests: server 2114 passed, CLI 248 passed, Go vet and tests pass, ruff clean. Not tested on a paid run. The cuda_max_good filter hasn't been tried against live Vast offers.

🤖 Generated with Claude Code

…pare extras, CLI update notice, one outage clear

- Image checks at submit and estimate: arm64-only images and images bigger
  than the disk are refused. Warnings cover Blackwell GPUs with CUDA < 12.8,
  images that may not fit unpacked or are slow to start, PEP 668 pip,
  building CUDA code without a toolkit, no Python, and apt as non-root.
  Inspections are cached by digest (image_facts). Offers are filtered on the
  image's CUDA major version (cuda_max_good).
- Change a queued or running job's max runtime and spend cap (job page,
  POST /api/v1/jobs/{id}/limits, `krater limits`). Resuming or moving a box
  already at its cap is refused.
- Every 15 minutes the worker warns the project's Slack channel, once per
  attempt, about overdue checkpoints.
- `krater compare` adds state, cost, runtime and GPU rows, plus --sweep.
- CLI update notice from X-Krater-CLI-Latest (CLI 0.8.0).
- Grouped network-outage alerts open and clear once.
- Runner sets PYTHONUNBUFFERED. --code checks for Python and sets
  PIP_BREAK_SYSTEM_PACKAGES, and warns when the code builds CUDA on a runtime
  image.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@Adambomb210
Adambomb210 force-pushed the claude/polish-batch-2 branch from 87552f0 to d671fee Compare November 19, 2026 23:24
@Adambomb210
Adambomb210 changed the base branch from claude/polish-batch to main November 19, 2026 23:24
@Adambomb210
Adambomb210 merged commit 44c346c into main Nov 19, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant