Skip to content

feat(pause): pause everything, PR 2: children fail closed, gateway turns cancellable, cron skipped, side calls and new work refused - #919

Merged
OAraLabs merged 50 commits into
mainfrom
feat/pause-everything-2
Nov 15, 2026
Merged

OAraLabs merged 50 commits into
mainfrom
feat/pause-everything-2

Conversation

@OAraLabs

@OAraLabs OAraLabs commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Beacon iOS Piece 17, PR 2 of 3: pause everything reaches the rest of the daemon. Stacked on #918; retarget as it merges.

A child that cannot read the pause state treats it as paused (a must-fix from Will).

  • Workers and coding runs are separate processes, and they read the pause file. The daemon now tells every child it builds the exact file (PROMETHEUS_PAUSE_FILE, set in tasks/child_env.task_child_env), so a child never guesses a data folder. Model-written shells started as tasks, watchers or cron jobs (security/shell_floor.model_shell_env) are told too, and a child passes on the file it was told, so a grandchild reads the same file. The bash tool's shell isn't told; it reads the default place, which is the same file.
  • A child that was told and cannot reach the file's folder (missing or unreadable) is paused. A readable folder with no file means never paused. A folder hidden behind an empty mount looks readable and empty, so the file check can't tell; the daemon's own state, and its cancelling of the turns it runs, is the authority.
  • A process that wasn't told, such as a CLI run, reads the default place as before.
  • Tested with real child processes (sys.executable) spawned with the environment the daemon builds, covering four cases: paused; folder missing; folder unreadable; folder readable but no file.

Turns from Telegram, Slack and Discord run as tasks a pause can cancel.

  • AgentLoop.run_async re-enters itself once inside pause.run_as_turn. A context variable holds a weak reference to the turn's own task. The turn gets a task of its own, and the long-lived poller that called it is never cancelled.
  • A task the turn spawns copies the variable but is a different task. So a turn run from one of those (a worker's re-engage from its watcher task) gets a turn task of its own too, and a pause cancels it.
  • The body stays run_async's own, so the parity tests that read it are unchanged.
  • A paused turn surfaces to the gateway as Paused. Its existing error reply then reads "Prometheus is paused (since …)".

While paused:

  • A cron job that falls due is skipped: its last status is skipped (mark_job_run takes a status), its next run moves on so nothing is saved up or re-fired every tick, it gets a history row, and nothing is notified, because nothing failed. POST /api/cron/{name}/run answers 423 rather than 200 followed by a skipped run.
  • Side model calls reach no model.
    • LLMCallEnvelope.call (memory extraction, the curator, synthesis) fails according to its caller's own on_failure policy: raise Paused, return None, or a result carrying it. No silent-failures row.
    • LLMCallEnvelope.stream (session titles, conversation summaries) raises Paused before the provider is asked. Its contract is to always re-raise, so on_failure doesn't apply.
    • A pause in a round's stream is not recorded as a provider error and is not considered for the fallback model.
    • A summary pass stops, leaves its batches uncompacted for the next pass, and counts no failure toward the circuit breaker.
    • The curator's tick is skipped, not failed: no failure count, no back-off.
    • A prompt or agent hook that can't be asked blocks its event, even with block_on_failure: false (fail closed). It isn't counted as a hook failure.
    • A pause that meets the compaction cascade skips it, rather than failing the pass.
  • No new coding run or desktop task. POST /api/code and POST /api/computer/tasks answer 423. ComputerTaskRunner.start itself refuses (AgentPaused, 423 paused), so Telegram's /computer yes starts nothing either. It checks once before its awaits and again after the last one, just before the task is listed. A pause landing while it looks for the app would otherwise find nothing to stop.

Choices noted:

  • When in doubt, fail closed: an unreachable folder means paused.
  • The 423 from /api/code and cron run-now gives who paused and why to the operator only, like the other doors.
  • The desktop runner checks the pause after the person and held-session checks, so someone who isn't a person learns nothing about the pause.
  • Smallest change: wrap once in run_async rather than at six call sites; check once in each envelope method rather than at each caller.

Not in this PR:

  • New workers via create_agent_task. These come only from tools inside a turn, and a pause stops the turn.
  • Nicer per-gateway "paused" wording.
  • PR 3: the push alert and a doctor row.

Reviews:

  • Round 1: two reviews. The review session found lows only. An independent agent found three mediums: the in-turn flag leaked into spawned tasks; stream() wasn't checked; Telegram /computer yes was not covered.
  • All three mediums and the lows from both reviews are fixed here, red first (tests/test_pause_side_doors.py, plus additions to tests/test_pause_children_fail_closed.py and tests/test_computer_door.py).
  • Round 2: two reviews.
    • The review session found nothing medium or higher (two lows, backlogged).
    • The agent found one medium: a pause landing while the desktop runner looked for the app. Fixed red-first, along with its lows: the cascade, fail-open hooks, and the CHANGELOG claim about shells.
    • Backlogged:
      • a manual /curator run while paused still runs its non-model pass;
      • the web chat handler treats a pause met within a round, before the sweep cancels the turn, as a turn failure. PR 3 can tidy that wording.

Tests: red first in each part (tests/test_pause_children_fail_closed.py, tests/test_pause_cron_skip.py, tests/test_pause_side_doors.py). The child-environment exact-match test gains the one key on purpose. Local Mac suite at the head, with the venv on PATH (the coding-acceptance tests run python3 -m pytest in their sandbox): 16,070 passed, 0 failed. Ruff and the mypy gate are clean.

🤖 Generated with Claude Code

OAraLabs and others added 30 commits November 12, 2026 14:54
…front doors and the loop

engine/pause.py holds the state in <data>/pause.json (atomic, fsync'd, 0600;
unreadable = paused) and counts the turns running in this process.
PUT /api/pause (operator) writes it first, withdraws every pending approval,
interrupts bridge turns and cancels every other running turn, stops desktop
tasks, tasks and workers, and pauses coding runs through their control file
(now written atomically). DELETE /api/pause resumes for an owner device only,
never the API token. The chat routes, story dispatch, the Paperclip wake and
/v1 answer 423; the WebSocket send gets an error frame; run_loop refuses to
start and stops between rounds. The daemon reads the state first at boot.
/api/status gains paused; server_capabilities lists pause.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…' string for the drift scanner

test_run_loop_copies_before_it_reads_anything pins the copy as run_loop's
first statement; the config-drift scanner reads a bare 'device' constant as a
reader of whisper.device. The run_loop test passes a context with for_run.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…without the disk, odd files, turn tasks only, no name in the frame, slash commands refused

From the security review of #918.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… disk, and only turn tasks are cancelled

Resume uses PersonCheck, the bar hold lifts use: a device a person marked, not
held. An owner device alone is not a person, since the API token mints one
from this machine. The pause is in force before it is written; a failed write
answers persisted: false and the sweep still runs. Non-UTF-8 and deeply
nested pause files read as paused instead of raising. Only tasks created for a
turn (run_as_turn, a /v1 request task) are cancelled, never a long-lived task
that ran a turn inline; POST /api/chat runs its turn in one. paused_changed
carries {paused, since} only. The WebSocket guard runs before slash commands.
Docs say restart persistence is not tamper-proof yet. From the security
review of #918 (HIGH-1, MEDIUM-1, LOW-1, LOW-2, LOW-4, LOW-5).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…next round

From the re-review of #918.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…t, AppArmor)

Added to DATA_TRUSTED_FILES (the gate's always-denied floor and the boot
probe), the Seatbelt regex, the AppArmor profile and the shipped denied
paths (a new shipped list, so a config holding the previous one upgrades).
The daemon's temp file now carries pause.json as its prefix, so the floor
covers it. From the security review of #918 (MEDIUM-2), with Will's go.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…the Seatbelt regex set; docs say pause.json is floored

Both registries pin known shapes on purpose: a test file that calls
gate.evaluate by hand, and the regexes the Seatbelt profile may hold.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…om the owner's Telegram

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…e owner's Telegram

person_devices (device_store), set and cleared by /person allow|disallow from
the owner's own Telegram (cmd_person) and by nothing else: no REST route, so
neither the API token nor a device can set it. It is not the computer-use mark
and gives nothing but resume; a held device cannot carry it; a revoked one
never counts. DELETE /api/pause requires it. Two pause tests changed on
purpose (they resumed with the computer-use mark). /person is charted as
Telegram-only in the parity manifest and refused on the web. Will, 2026-11-12.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…-drift scanner reads tests too)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…s when the store cannot answer (review lows)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…device off the daemon's machine

From the review of the person mark (lows 2, 3, 4).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…on], /resume)

The owner only, in their own Telegram. /pause runs the same pause as PUT
/api/pause through the web app the daemon registered (the state alone when it
is not up); /resume clears it, tells every client and starts nothing. Refused
on the web and charted Telegram-only. Required before 0.9.21 so a pause can
always be lifted without a marked device (person-mark review, low 1).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…t the web app says it swept nothing (review lows)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
From the review of Telegram /pause and /resume (low 1; low 2 is the new
registration test).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
cde66e8 was pushed with this test still failing; this makes it pass.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… control characters)

Phone item 4a: the phone shows them on every screen.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ho or why (phone item 4b)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ver who or why

Phone item 4b: a chat-only (scoped) phone could not read /api/status. The
scoped allowlist and its pinned test gain the one route on purpose.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review of 4b: paused_response carried by and reason to every caller, so the
'never who or why' claim for scoped devices did not hold. It now takes the
request and adds who and why for an operator only; an unidentified caller gets
the scoped answer (fail closed). A comment says why read_file does not
re-sanitize.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…he Paused error names nobody (review lows)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…and trailing space; the Paused error names nobody

Second review of 4a/4b (lows).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…as paused (real child process)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The daemon tells every child it builds the exact pause file
(PROMETHEUS_PAUSE_FILE, via task_child_env). A child that was told and cannot
reach that file's folder is paused (fail closed); a readable folder with no
file is never-paused; a CLI run that was not told keeps the default path. The
child-env exact-match test gains the one key on purpose. Tested with real
child processes in the daemon's child environment. PR 2 must-fix (Will).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
OAraLabs and others added 15 commits November 13, 2026 13:27
AgentLoop.run_async re-enters itself once inside pause.run_as_turn (a
contextvar marks the task), so Telegram, Slack, Discord and the benchmark
commands get a turn task a pause cancels without touching their long-lived
poller. The body stays run_async's own, so the parity tests that read it are
unchanged. The red test's first version patched a helper that no longer
exists; it now checks the hand-off directly (fails without this change).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…d on

mark_job_run takes a status override ('skipped'); next_run moves on, so
nothing is saved up or re-dispatched every tick, and nothing is notified.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… by its caller's policy

Titles, compaction, extraction, the curator and synthesis all go through
LLMCallEnvelope.call: while paused it raises Paused, returns None or returns
a result carrying Paused, per on_failure; no silent_failures row.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ead_file's comment under its docstring

Review lows.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…open

A turn a spawned task runs, streamed side calls, Telegram /computer yes,
the curator tick, cron run-now, /api/code's operator answer, and the pause
file for model shells and grandchildren.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- A turn run from a task a turn spawned (a worker's re-engage) gets a
  turn task of its own: the in-turn mark is a weak reference to the
  turn's own task, not a flag every spawned task inherited.
- LLMCallEnvelope.stream refuses while paused, so session titles and
  conversation summaries ask no model; a pause in a round's stream is
  not a provider error and not a fallback case; a summary pass stops
  and counts no failure.
- ComputerTaskRunner.start refuses while paused (423 paused), so
  Telegram's /computer yes starts nothing.
- The curator's tick while paused is skipped, not failed; cron run-now
  answers 423; /api/code's 423 tells the operator who and why; model
  shells and grandchildren are told the same pause file; the docstring
  no longer says an empty mount reads as paused.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A pause landing while a desktop task looks for its app, a pause meeting
the compaction cascade, and a fail-open prompt hook while paused.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- ComputerTaskRunner.start checks the pause again after its last await,
  right before the task is listed: a pause landing while it looked for
  the app found nothing to stop, and the task then ran.
- A pause meeting the compaction cascade skips it instead of failing
  the pass.
- A prompt or agent hook that cannot be asked while paused blocks its
  event, whatever block_on_failure says (fail closed).
- CHANGELOG: which model-written shells are told the pause file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Gateway turns are cancelled at once; children stop at their next round;
the new 423s, the cron skip, the desktop door and side calls.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
# Conflicts:
#	CHANGELOG.md
#	docs/reference/routes.md
#	src/prometheus/web/ws_server.py
# Conflicts:
#	CHANGELOG.md
#	docs/reference/routes.md
@OAraLabs
OAraLabs changed the base branch from feat/pause-everything to main November 15, 2026 05:11
# Conflicts:
#	CHANGELOG.md
#	docs/guide/api.md
#	src/prometheus/engine/pause.py
@OAraLabs
OAraLabs marked this pull request as ready for review November 15, 2026 05:12
@OAraLabs
OAraLabs enabled auto-merge (squash) November 15, 2026 05:12
@OAraLabs
OAraLabs merged commit 587f187 into main Nov 15, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant