fix(mothership): take over runs whose controller died and re-run their interrupted tool by waleedlatif1 · Pull Request #9059 · simstudioai/sim

waleedlatif1 · 2026-10-15T00:30:31Z

Summary

The Sim process that admitted a Chat run is the run's controller. Sometimes that process stops mid-run: a blue/green deploy terminates the old task set, or the process crashes. If no browser is attached when that happens, the run is stranded:

Recovery never starts. It only ran from a browser reconnect.
The tool result is lost. The in-flight Sim tool died with its owner and never recorded a result.
The worker gives up. It waited out the call and then ended the run as detached.

This PR lets Sim take such runs over by itself:

Takeover. Every Sim process looks for abandoned runs every 20s. An abandoned run is an interactive run with its recovery intent saved, no chat lock, and one minute idle. Each one is taken over through the same recovery a reconnect runs (recoverStreamRun, shared by the two use cases).
Access is re-checked. The takeover acts as the run's user through a new delegated operation, mothership.runs.takeover. It goes through the same authorization funnel and capability (copilot.use) as a reconnect, so the user's current access is checked again.
Exactly one controller. The chat lock and the controller-token claim admit one controller. A live controller is never taken over, and a reconnect or another process racing the takeover loses cleanly.
Revoked access. The run is settled as an error under the fenced chat lock and the guarded controller token, and the worker is told to stop it.
Interrupted tools are re-run.
- A Sim tool whose owner died before recording a result is claimed again under a new owner token and run again. "Died" means the call is still running, unsettled, and its execution lease has lapsed.
- A workflow tool is re-run by the server; the browser cannot pick it up.
- The new owner token fences the dead owner out, because both its renewals and its result require the token and an unexpired lease.
- Each start is counted in a new execution_attempts column. After three starts the call fails with a clear message, so a tool that kills its executor cannot loop.
- Each re-run is a fresh execution: a new owner token, a new service meter id, and a new workflow execution id. Each attempt therefore bills once.
The takeover waits for leased tools. While one of the run's Sim tools still holds its lease, the takeover waits. A dead owner's lease lapses within a minute, and after that the call can be re-run instead of being reported as interrupted.
Draining processes don't adopt runs. A process stops scanning on SIGTERM, so it never takes over a run it is about to abandon.

No worker change is needed. The worker already re-hands every parked call, including one that was in flight, to a newly attached leg.

Type of Change

Bug fix

Testing

abandoned-runs.integration.ts runs against real Postgres and Redis. It goes through the production takeover, the reconnect route and the chat lifecycle, with a stub worker. It covers:

a controller dies mid-tool → takeover → the tool is re-run → the run completes
a live controller is never taken over
a still-leased tool delays the takeover until its lease lapses
a browser reconnect racing the takeover starts one controller and one re-run
revoked access → the run is settled as an error and the worker is stopped
a tool that has already died 3 times fails with a clear message
for workflow tools: only the server may pick up an interrupted run, and an exhausted one is failed before it is offered

Fails before the fix: with the reclaim and live-lease guards reverted, 4 of the 8 tests fail. With only the live-lease guard reverted, its test fails. The existing recover-stream.test.ts passes. The migration is backward-compatible (check:migrations).

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added
No new warnings introduced

…r interrupted tool A Chat run is driven by the Sim process that admitted it. When that process stops mid-run (a deploy's old task set is terminated, or a crash) and no browser is attached, nothing takes the run over: recovery only ran from a browser reconnect, the in-flight tool's result was never recorded, and the worker eventually ended the run as detached. - Every Sim process now looks every 20s for interactive runs whose chat lock is gone and takes each over through the same recovery a reconnect runs, acting as the run's user under a delegated principal that re-checks that user's current access. The chat lock and controller token still admit exactly one controller, so a racing reconnect or another process loses. - A user who lost access gets the run settled as an error and the worker is told to stop it. - A Sim tool whose owner died before recording a result (its execution lease lapsed while the call was still running) is claimed again under a new owner token and run again, by the server for a workflow tool. Each start is counted; after three, the call fails with a clear message instead. - A takeover waits while one of the run's Sim tools still holds its lease.

vercel · 2026-10-15T00:30:35Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 15, 2026 6:34am UTC

cubic-dev-ai

All reported issues were addressed across 12 files

_{Turn on auto-fix | Re-trigger cubic}

…the schema mock

… dispose chat pubsub in tests

greptile-apps · 2026-10-15T00:37:47Z

Confidence Score: 5/5

[High impact] The PR appears safe to merge; no new blocking issue was found.

Summary

Adds automatic recovery for Chat runs whose controller died, with bounded tool retries and fresh ownership checks.

Sim takes over abandoned Chat runs without waiting for a reconnect.
Interrupted tools can restart under a new owner up to three times.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Scan idle runs] --> B{Chat lock held?}
  B -->|Yes| C[Skip run]
  B -->|No| D[Check current user access]
  D -->|Denied| E[Settle run and stop worker]
  D -->|Allowed| F{Tool lease still live?}
  F -->|Yes| G[Wait for a later scan]
  F -->|No| H[Acquire chat lock]
  H --> I[Check shutdown and claim controller]
  I --> J[Attach to worker again]
  J --> K[Re-run interrupted tools within attempt limit]

_{Reviews (6) · Last reviewed commit: "test(mothership): find the attempt cap i..." · Reviewed by Greptile}

…errupted re-run

waleedlatif1 · 2026-10-15T00:52:23Z

@greptileai review

waleedlatif1 · 2026-10-15T00:52:24Z

@cubic-dev-ai review

cubic-dev-ai · 2026-10-15T00:52:30Z

@cubic-dev-ai review

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

…ure table

cubic-dev-ai

1 issue found and verified against the latest diff

Confidence score: 3/5

In abandoned-runs.integration.ts, an unset TEST_REDIS_URL becomes the invalid string "undefined", so environment validation fails before the Redis-gated suite can skip. Assign REDIS_URL only when url is set.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/mothership/request/application/abandoned-runs.integration.ts">

<violation number="1" location="apps/sim/lib/mothership/request/application/abandoned-runs.integration.ts:50">
P2: When `TEST_REDIS_URL` is unset, this assignment turns `undefined` into the invalid URL string `"undefined"`, so environment validation fails before the Redis-gated suite can skip. Assign `REDIS_URL` only when `url` is present, as the existing desktop-executor integration suite does.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

…ands, decline takeovers on shutdown - The abandoned-run scan resumes from a durable cursor and examines a bounded window per round, so runs whose controller is alive cannot hide later runs. - A re-run's earlier attempts' workbench commands are handed to the workbench recovery to stop before the re-run may use the workbench; each recorded command now names the attempt that started it. - Shutdown aborts a takeover still in flight at its controller claim. - Only an authorization denial settles a run as revoked; other takeover failures leave it recoverable, as a failed reconnect does. - The suite sets REDIS_URL only when a test Redis is configured.

waleedlatif1 · 2026-10-15T01:46:36Z

@greptileai review

waleedlatif1 · 2026-10-15T01:46:37Z

@cubic-dev-ai review

cubic-dev-ai · 2026-10-15T01:46:42Z

@cubic-dev-ai review

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 15 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

…imant never started

waleedlatif1 · 2026-10-15T02:02:38Z

@greptileai review

waleedlatif1 · 2026-10-15T02:02:39Z

@cubic-dev-ai review

cubic-dev-ai · 2026-10-15T02:02:46Z

@cubic-dev-ai review

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 15 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

- Add the `mothership-abandoned-run-takeover-disabled` AppConfig kill switch, checked once per scan; off by default, so takeovers run unless it is on. - Stop a controller whose chat lock lapsed while every renewal threw, so a controller partitioned from Redis cannot keep driving a run taken over. - Never revoke another call awaiting its re-run from a sibling's wait or workbench access; the sibling's workbench access still stops that call's dead attempt's commands. - Treat a workspace access loss found inside recovery like an authorization denial, by error type; other recovery failures stay recoverable. - Cover organization Assistant takeover for a member and a removed member.

… SQL operand

waleedlatif1 · 2026-10-15T06:26:39Z

@greptileai review

waleedlatif1 · 2026-10-15T06:26:40Z

@cubic-dev-ai review

cubic-dev-ai · 2026-10-15T06:26:45Z

@cubic-dev-ai review

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 21 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

…e-run predicates as conditions, revoke access mid-recovery in the test

…ction

waleedlatif1 · 2026-10-15T06:50:00Z

@greptileai review

waleedlatif1 · 2026-10-15T06:50:02Z

@cubic-dev-ai review

cubic-dev-ai · 2026-10-15T06:50:09Z

@cubic-dev-ai review

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

waleedlatif1 · 2026-10-15T06:52:19Z

@greptileai review

waleedlatif1 · 2026-10-15T06:52:20Z

@cubic-dev-ai review

cubic-dev-ai · 2026-10-15T06:52:26Z

@cubic-dev-ai review

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 22 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai Bot reviewed Oct 15, 2026

View reviewed changes

fix(mothership): import the takeover by absolute path and regenerate …

03d1411

…the schema mock

vercel Bot temporarily deployed to Preview October 15, 2026 00:34 Inactive

waleedlatif1 mentioned this pull request Oct 15, 2026

v0.9.22: docs overhaul, geo and mcp improvements #9040

Merged

fix(mothership): page past live controllers, stop a scan on shutdown,…

6401695

… dispose chat pubsub in tests

vercel Bot temporarily deployed to Preview October 15, 2026 00:36 Inactive

greptile-apps Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/async-runs/orphaned-runs.ts Outdated

Comment thread apps/sim/lib/mothership/async-runs/repository.ts

Comment thread apps/sim/lib/mothership/request/application/abandoned-runs.ts Outdated

test(mothership): assert the claim admits a first execution or an int…

9d06cff

…errupted re-run

vercel Bot temporarily deployed to Preview October 15, 2026 00:46 Inactive

test(mothership): add execution_attempts to the browser download fixt…

cb879b9

…ure table

vercel Bot temporarily deployed to Preview October 15, 2026 00:55 Inactive

cubic-dev-ai Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/application/abandoned-runs.ts

Comment thread apps/sim/lib/mothership/request/application/abandoned-runs.integration.ts Outdated

waleedlatif1 added 2 commits October 14, 2026 18:30

fix(mothership): keep the takeover stop handler module-private

79f0d27

vercel Bot temporarily deployed to Preview October 15, 2026 01:33 Inactive

cubic-dev-ai Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/async-runs/repository.ts Outdated

fix(mothership): let the server re-claim a workflow call its dead cla…

18f650f

…imant never started

vercel Bot temporarily deployed to Preview October 15, 2026 01:53 Inactive

cubic-dev-ai Bot reviewed Oct 15, 2026

View reviewed changes

vercel Bot temporarily deployed to Preview October 15, 2026 06:11 Inactive

fix(mothership): negate the re-runnable predicate without an optional…

fe57d22

… SQL operand

vercel Bot temporarily deployed to Preview October 15, 2026 06:13 Inactive

cubic-dev-ai Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/application/abandoned-runs.integration.ts Outdated

Comment thread apps/sim/lib/mothership/request/session/abort.ts

fix(mothership): time the lock lapse from its acquisition, type the r…

e333fdb

…e-run predicates as conditions, revoke access mid-recovery in the test

vercel Bot temporarily deployed to Preview October 15, 2026 06:33 Inactive

test(mothership): find the attempt cap inside the typed re-run conjun…

a6cd60a

…ction

vercel Bot temporarily deployed to Preview October 15, 2026 06:34 Inactive

cubic-dev-ai Bot reviewed Oct 15, 2026

View reviewed changes

waleedlatif1 merged commit 2e98120 into staging Oct 15, 2026
56 checks passed

waleedlatif1 mentioned this pull request Oct 15, 2026

v0.9.23: project files foundation, ci fixes #9078

Merged

waleedlatif1 deleted the fix/deploy-drain-inflight-tools branch October 15, 2026 08:48

Conversation

waleedlatif1 commented Oct 15, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 15, 2026

Uh oh!

waleedlatif1 commented Oct 15, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 15, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 15, 2026

Uh oh!

waleedlatif1 commented Oct 15, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 15, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 15, 2026

Uh oh!

waleedlatif1 commented Oct 15, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 15, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 15, 2026

Uh oh!

waleedlatif1 commented Oct 15, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 15, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 15, 2026

Uh oh!

waleedlatif1 commented Oct 15, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 15, 2026

Uh oh!

waleedlatif1 commented Oct 15, 2026

Uh oh!

waleedlatif1 commented Oct 15, 2026

vercel Bot commented Oct 15, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 15, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading