Skip to content

feat(pause): pause everything, PR 1: the state, PUT/DELETE /api/pause, the front doors and the loop - #918

Merged
OAraLabs merged 30 commits into
mainfrom
feat/pause-everything
Nov 15, 2026
Merged

OAraLabs merged 30 commits into
mainfrom
feat/pause-everything

Conversation

@OAraLabs

@OAraLabs OAraLabs commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Beacon iOS Piece 17, PR 1 of 3: pause everything. This PR covers the state, the routes, the front doors, the agent loop and the pause itself.

State: engine/pause.py

  • One daemon-wide state in <data>/pause.json, written atomically and fsync'd, mode 0600. Non-UTF-8 bytes and JSON nested too deep also count as unreadable, so as paused.
  • The daemon reads it first at boot, before any gateway, scheduler or turn exists. A daemon that was paused comes back paused.
  • A file that cannot be read or parsed means paused, with a reason saying so.
  • Child processes (workers, coding runs) read the same file, cached on its mtime.

PUT /api/pause: the operator pauses (the API token or an owner device; a scoped device gets 403 operator_only). In this order it:

  1. writes the state, so nothing new starts from here on;
  2. withdraws every pending approval, so approving one later is a 410;
  3. interrupts every bridge turn through its own Stop, and cancels every other task created for a turn (see below);
  4. stops desktop tasks;
  5. stops tasks and workers (TaskManager.stop_task; the process group goes);
  6. pauses coding runs through their control file, which keeps the work (that file is now written atomically);
  7. answers {paused, since, by, reason, stopped: {turns, desktop_tasks, tasks, coding_runs, approvals_withdrawn}} and broadcasts paused_changed.

Pausing twice sweeps twice and is still one pause, with the same since.

While paused

  • POST /api/chat/send, POST /api/chat, story dispatch, the Paperclip wake and /v1/chat/completions answer 423 {"error": "paused", since, by, reason} and keep nothing.
  • The WebSocket send gets an error frame of kind paused. In ws_server.py that is one guard at the top of _handle_send_message plus the capability line, nothing else.
  • run_loop refuses to start, and stops between rounds. That is the backstop for every door, and the only check in a child process.
  • Reads and every stop route still work.

DELETE /api/pause: only a person resumes.

  • A person is a device carrying the person mark, which the owner sets in their own Telegram: /person allow <device id>, removed with /person disallow <device id>, listed with /person.
  • The mark does nothing but allow resume. It is not the computer-use mark (that one gives desktop control), no REST route sets it, a held device cannot carry it, and a revoked one never counts. It lives in its own person_devices table, created on first use.
  • /person is charted Telegram-only in the parity manifest and refused on the web, where the speaker is a device.
  • The API token, an open daemon and any device without the mark, owner devices included, get 403 person_required. The token can mint an owner device on this machine; test_a_model_that_mints_its_own_owner_device_cannot_resume does exactly that and is refused.
  • The owner in Telegram: /pause [reason] and /resume, for the owner in their own Telegram and nobody else, so a pause can always be lifted even with no device marked. /pause runs the same pause as PUT /api/pause. Both are refused on the web and charted Telegram-only.
  • Resuming starts nothing. Interrupted turns stay interrupted. Coding runs stay paused until each is resumed (/api/code/{id}/resume).

pause.json is in the trusted-state floor. It joins the scheduled-jobs files in DATA_TRUSTED_FILES (the gate's always-denied floor and the boot probe), the Seatbelt regex, the AppArmor profile (deny @{HOME}/.prometheus/data/pause.json* wl,) and a new shipped denied-paths list, so a config holding the previous list upgrades. All four are by prefix: the daemon's temp file is now pause.json.<hex>.tmp. An installed AppArmor profile must be reloaded to get the rule; a separate, reviewed script does this on the mini (Will runs it). Red tests first: tests/test_pause_floor.py, which includes a real Seatbelt shell that cannot change, delete or create it but can still read it.

A pause holds even if it cannot be written. It is in force in the daemon before the file is written. If the write fails, the answer says persisted: false and the sweep still runs.

Only turn tasks are cancelled. These are tasks created for one turn (pause.run_as_turn, which POST /api/chat uses) or a /v1 request task. A long-lived task that happened to run a turn inline (a chat poller, the scheduler) is never cancelled; those turns stop at run_loop's round check. PR 2 wraps the chat gateways' turns in tasks of their own.

Clients: every client hears paused_changed {paused, since}; who paused and why are on /api/status, which is operator-only. No slash command runs over the WebSocket while paused.

Is the Mac app's token shell-readable? (Will asked PR 1 to say.) Beacon on the Mac keeps its credential under a Keychain-held safeStorage key, so a shell trying to read it raises a Keychain prompt on screen. Beacon on Linux without a keyring falls back to a fixed key, which a shell can read. Since the global token can mint an owner device anyway, resume no longer leans on this; see above.

Not in this PR

  • PR 2: cron, timers, side model calls, refusing new workers, subagents, coding and desktop tasks, the chat gateways' own replies, turns from Telegram, Slack, Discord and the scheduler run as tasks a pause can cancel, and a child that cannot read the pause state treats it as paused (fail closed), with a real-sandbox test.
  • PR 3: the push alert and the doctor row.
  • Until PR 2 lands, a Telegram turn during a pause is refused by the run_loop backstop, so its reply is an error rather than "paused".

Tests: red first, tests/test_pause_everything.py (33, review round included). Two existing tests shaped the code:

  • run_loop's copy stays its first statement;
  • the config-drift scanner reads any bare "device" string as a reader of a config key.

Gates: local Mac suite 15,992 passed (one timing test, the daemon's SIGTERM exit, failed under load and passes on its own), 0 failed (test_generated_reference and test_gate_manifest deselected: they fail only in a worktree). Ruff, the mypy gate and pip-audit are clean.

🤖 Generated with Claude Code

OAraLabs and others added 28 commits November 12, 2026 14:54
…front doors and the loop

engine/pause.py holds the state in <data>/pause.json (atomic, fsync'd, 0600;
unreadable = paused) and counts the turns running in this process.
PUT /api/pause (operator) writes it first, withdraws every pending approval,
interrupts bridge turns and cancels every other running turn, stops desktop
tasks, tasks and workers, and pauses coding runs through their control file
(now written atomically). DELETE /api/pause resumes for an owner device only,
never the API token. The chat routes, story dispatch, the Paperclip wake and
/v1 answer 423; the WebSocket send gets an error frame; run_loop refuses to
start and stops between rounds. The daemon reads the state first at boot.
/api/status gains paused; server_capabilities lists pause.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…' string for the drift scanner

test_run_loop_copies_before_it_reads_anything pins the copy as run_loop's
first statement; the config-drift scanner reads a bare 'device' constant as a
reader of whisper.device. The run_loop test passes a context with for_run.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…without the disk, odd files, turn tasks only, no name in the frame, slash commands refused

From the security review of #918.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… disk, and only turn tasks are cancelled

Resume uses PersonCheck, the bar hold lifts use: a device a person marked, not
held. An owner device alone is not a person, since the API token mints one
from this machine. The pause is in force before it is written; a failed write
answers persisted: false and the sweep still runs. Non-UTF-8 and deeply
nested pause files read as paused instead of raising. Only tasks created for a
turn (run_as_turn, a /v1 request task) are cancelled, never a long-lived task
that ran a turn inline; POST /api/chat runs its turn in one. paused_changed
carries {paused, since} only. The WebSocket guard runs before slash commands.
Docs say restart persistence is not tamper-proof yet. From the security
review of #918 (HIGH-1, MEDIUM-1, LOW-1, LOW-2, LOW-4, LOW-5).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…next round

From the re-review of #918.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…t, AppArmor)

Added to DATA_TRUSTED_FILES (the gate's always-denied floor and the boot
probe), the Seatbelt regex, the AppArmor profile and the shipped denied
paths (a new shipped list, so a config holding the previous one upgrades).
The daemon's temp file now carries pause.json as its prefix, so the floor
covers it. From the security review of #918 (MEDIUM-2), with Will's go.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…the Seatbelt regex set; docs say pause.json is floored

Both registries pin known shapes on purpose: a test file that calls
gate.evaluate by hand, and the regexes the Seatbelt profile may hold.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…om the owner's Telegram

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…e owner's Telegram

person_devices (device_store), set and cleared by /person allow|disallow from
the owner's own Telegram (cmd_person) and by nothing else: no REST route, so
neither the API token nor a device can set it. It is not the computer-use mark
and gives nothing but resume; a held device cannot carry it; a revoked one
never counts. DELETE /api/pause requires it. Two pause tests changed on
purpose (they resumed with the computer-use mark). /person is charted as
Telegram-only in the parity manifest and refused on the web. Will, 2026-11-12.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…-drift scanner reads tests too)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…s when the store cannot answer (review lows)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…device off the daemon's machine

From the review of the person mark (lows 2, 3, 4).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…on], /resume)

The owner only, in their own Telegram. /pause runs the same pause as PUT
/api/pause through the web app the daemon registered (the state alone when it
is not up); /resume clears it, tells every client and starts nothing. Refused
on the web and charted Telegram-only. Required before 0.9.21 so a pause can
always be lifted without a marked device (person-mark review, low 1).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…t the web app says it swept nothing (review lows)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
From the review of Telegram /pause and /resume (low 1; low 2 is the new
registration test).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
cde66e8 was pushed with this test still failing; this makes it pass.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… control characters)

Phone item 4a: the phone shows them on every screen.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ho or why (phone item 4b)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ver who or why

Phone item 4b: a chat-only (scoped) phone could not read /api/status. The
scoped allowlist and its pinned test gain the one route on purpose.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review of 4b: paused_response carried by and reason to every caller, so the
'never who or why' claim for scoped devices did not hold. It now takes the
request and adds who and why for an operator only; an unidentified caller gets
the scoped answer (fail closed). A comment says why read_file does not
re-sanitize.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…he Paused error names nobody (review lows)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…and trailing space; the Paused error names nobody

Second review of 4a/4b (lows).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ead_file's comment under its docstring

Review lows.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
# Conflicts:
#	CHANGELOG.md
#	docs/reference/routes.md
#	src/prometheus/web/ws_server.py
# Conflicts:
#	CHANGELOG.md
#	docs/reference/routes.md
@OAraLabs
OAraLabs marked this pull request as ready for review November 15, 2026 04:40
@OAraLabs
OAraLabs merged commit edc1964 into main Nov 15, 2026
14 checks passed
OAraLabs added a commit that referenced this pull request Nov 15, 2026
…rns cancellable, cron skipped, side calls and new work refused (#919)

* test(pause): red tests for pause everything, PR 1

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(pause): pause everything: the state, PUT/DELETE /api/pause, the front doors and the loop

engine/pause.py holds the state in <data>/pause.json (atomic, fsync'd, 0600;
unreadable = paused) and counts the turns running in this process.
PUT /api/pause (operator) writes it first, withdraws every pending approval,
interrupts bridge turns and cancels every other running turn, stops desktop
tasks, tasks and workers, and pauses coding runs through their control file
(now written atomically). DELETE /api/pause resumes for an owner device only,
never the API token. The chat routes, story dispatch, the Paperclip wake and
/v1 answer 423; the WebSocket send gets an error frame; run_loop refuses to
start and stops between rounds. The daemon reads the state first at boot.
/api/status gains paused; server_capabilities lists pause.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): the pause check follows the per-run copy; no bare 'device' string for the drift scanner

test_run_loop_copies_before_it_reads_anything pins the copy as run_loop's
first statement; the config-drift scanner reads a bare 'device' constant as a
reader of whisper.device. The run_loop test passes a context with for_run.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: resume needs a person (PersonCheck), a pause holds without the disk, odd files, turn tasks only, no name in the frame, slash commands refused

From the security review of #918.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): a person resumes (PersonCheck), a pause holds without the disk, and only turn tasks are cancelled

Resume uses PersonCheck, the bar hold lifts use: a device a person marked, not
held. An owner device alone is not a person, since the API token mints one
from this machine. The pause is in force before it is written; a failed write
answers persisted: false and the sweep still runs. Non-UTF-8 and deeply
nested pause files read as paused instead of raising. Only tasks created for a
turn (run_as_turn, a /v1 request task) are cancelled, never a long-lived task
that ran a turn inline; POST /api/chat runs its turn in one. paused_changed
carries {paused, since} only. The WebSocket guard runs before slash commands.
Docs say restart persistence is not tamper-proof yet. From the security
review of #918 (HIGH-1, MEDIUM-1, LOW-1, LOW-2, LOW-4, LOW-5).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(pause): say which turns a pause cancels and which stop at their next round

From the re-review of #918.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: pause.json sits in the trusted-state floor (gate, Seatbelt, AppArmor)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): pause.json sits in the trusted-state floor (gate, Seatbelt, AppArmor)

Added to DATA_TRUSTED_FILES (the gate's always-denied floor and the boot
probe), the Seatbelt regex, the AppArmor profile and the shipped denied
paths (a new shipped list, so a config holding the previous one upgrades).
The daemon's temp file now carries pause.json as its prefix, so the floor
covers it. From the security review of #918 (MEDIUM-2), with Will's go.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): register the floor test with the gate-call registry and the Seatbelt regex set; docs say pause.json is floored

Both registries pin known shapes on purpose: a test file that calls
gate.evaluate by hand, and the regexes the Seatbelt profile may hold.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: a separate person mark is what resumes; set only from the owner's Telegram

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(pause): a separate person mark is what resumes, set only from the owner's Telegram

person_devices (device_store), set and cleared by /person allow|disallow from
the owner's own Telegram (cmd_person) and by nothing else: no REST route, so
neither the API token nor a device can set it. It is not the computer-use mark
and gives nothing but resume; a held device cannot carry it; a revoked one
never counts. DELETE /api/pause requires it. Two pause tests changed on
purpose (they resumed with the computer-use mark). /person is charted as
Telegram-only in the parity manifest and refused on the web. Will, 2026-11-12.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): no bare 'device' key in the person-mark test (the config-drift scanner reads tests too)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: the /person handler closes its store; resume refuses when the store cannot answer (review lows)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): the /person handler closes its store; docs say to mark a device off the daemon's machine

From the review of the person mark (lows 2, 3, 4).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: the owner pauses and resumes from Telegram

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(pause): the owner pauses and resumes from Telegram (/pause [reason], /resume)

The owner only, in their own Telegram. /pause runs the same pause as PUT
/api/pause through the web app the daemon registered (the state alone when it
is not up); /resume clears it, tells every client and starts nothing. Refused
on the web and charted Telegram-only. Required before 0.9.21 so a pause can
always be lifted without a marked device (person-mark review, low 1).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: route registration wires chat pause; a pause without the web app says it swept nothing (review lows)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): a chat pause without the web app says it swept nothing

From the review of Telegram /pause and /resume (low 1; low 2 is the new
registration test).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): the no-web-app reply matches its test (case)

cde66e8 was pushed with this test still failing; this makes it pass.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: reason and who are one short line (phone item 4a)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): who paused and why are one short line (140 / 80 chars, no control characters)

Phone item 4a: the phone shows them on every screen.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: a scoped device can ask whether it is paused, not who or why (phone item 4b)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(pause): GET /api/pause answers {paused, since} to any device, never who or why

Phone item 4b: a chat-only (scoped) phone could not read /api/status. The
scoped allowlist and its pinned test gain the one route on purpose.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: a scoped device's 423 says since, not who or why (review of 4b)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): a scoped device's 423 says since, not who paused or why

Review of 4b: paused_response carried by and reason to every caller, so the
'never who or why' claim for scoped devices did not hold. It now takes the
request and adds who and why for an operator only; an unidentified caller gets
the scoped answer (fail closed). A comment says why read_file does not
re-sanitize.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: marks and invisible letters dropped, cap trimmed, the Paused error names nobody (review lows)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): who and why drop marks and invisible letters (NFC first) and trailing space; the Paused error names nobody

Second review of 4a/4b (lows).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: a child that cannot read the pause state treats it as paused (real child process)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): a child that cannot read the pause state treats it as paused

The daemon tells every child it builds the exact pause file
(PROMETHEUS_PAUSE_FILE, via task_child_env). A child that was told and cannot
reach that file's folder is paused (fail closed); a readable folder with no
file is never-paused; a CLI run that was not told keeps the default path. The
child-env exact-match test gains the one key on purpose. Tested with real
child processes in the daemon's child environment. PR 2 must-fix (Will).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: a gateway turn is cancelled by a pause and its poller survives

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): chat-gateway turns run as tasks a pause can cancel

AgentLoop.run_async re-enters itself once inside pause.run_as_turn (a
contextvar marks the task), so Telegram, Slack, Discord and the benchmark
commands get a turn task a pause cancels without touching their long-lived
poller. The body stays run_async's own, so the parity tests that read it are
unchanged. The red test's first version patched a helper that no longer
exists; it now checks the hand-off directly (fails without this change).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: a cron job due while paused is skipped, recorded and moved on

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): a cron job due while paused is skipped, recorded and moved on

mark_job_run takes a status override ('skipped'); next_run moves on, so
nothing is saved up or re-dispatched every tick, and nothing is notified.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: a side model call while paused reaches no model and fails by its policy

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): a side model call while paused reaches no model and fails by its caller's policy

Titles, compaction, extraction, the curator and synthesis all go through
LLMCallEnvelope.call: while paused it raises Paused, returns None or returns
a result carrying Paused, per on_failure; no silent_failures row.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red: no new coding run or desktop task while paused

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): no new coding run or desktop task starts while paused (423)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(pause): the CHANGELOG says what PR 2 adds

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(pause): the 423 docstring says an open daemon is the operator; read_file's comment under its docstring

Review lows.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red tests for the doors the second review of #919 found open

A turn a spawned task runs, streamed side calls, Telegram /computer yes,
the curator tick, cron run-now, /api/code's operator answer, and the pause
file for model shells and grandchildren.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): the doors the second review of #919 found open

- A turn run from a task a turn spawned (a worker's re-engage) gets a
  turn task of its own: the in-turn mark is a weak reference to the
  turn's own task, not a flag every spawned task inherited.
- LLMCallEnvelope.stream refuses while paused, so session titles and
  conversation summaries ask no model; a pause in a round's stream is
  not a provider error and not a fallback case; a summary pass stops
  and counts no failure.
- ComputerTaskRunner.start refuses while paused (423 paused), so
  Telegram's /computer yes starts nothing.
- The curator's tick while paused is skipped, not failed; cron run-now
  answers 423; /api/code's 423 tells the operator who and why; model
  shells and grandchildren are told the same pause file; the docstring
  no longer says an empty mount reads as paused.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(pause): red tests for round 2 of #919's review

A pause landing while a desktop task looks for its app, a pause meeting
the compaction cascade, and a fail-open prompt hook while paused.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pause): round 2 of #919's review

- ComputerTaskRunner.start checks the pause again after its last await,
  right before the task is listed: a pause landing while it looked for
  the app found nothing to stop, and the task then ran.
- A pause meeting the compaction cascade skips it instead of failing
  the pass.
- A prompt or agent hook that cannot be asked while paused blocks its
  event, whatever block_on_failure says (fail closed).
- CHANGELOG: which model-written shells are told the pause file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(pause): the API guide says what PR 2 changed

Gateway turns are cancelled at once; children stop at their next round;
the new 423s, the cron skip, the desktop door and side calls.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant