Skip to content

fix(microvm-worker): state that the refusal ended - #817

Merged
ParidelPooya merged 2 commits into
mainfrom
fix/microvm-worker-unknown-outcome-warning
Oct 23, 2026
Merged

ParidelPooya merged 2 commits into
mainfrom
fix/microvm-worker-unknown-outcome-warning

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #806.

Summary

The warning for an unknown outcome of the worker's own SuspendMicrovm call no longer says that jobs are refused once the refusal has already ended. It now states that the refusal ended during the call, that the worker accepts jobs, and when it tries the call again.

Why

  1. Without a terminate hook, the warning said: "The worker refuses jobs until the MicroVM resumes, or for 30 seconds."
  2. A resume hook during the call ends the refusal. The 30-second limit of a suspend hook's refusal can also end during the call.
  3. After an unknown outcome, the worker starts a new refusal only while _suspending is still set.
  4. So in these sequences a job right after the warning gets HTTP 202. The fix(microvm-worker): wait at least 30 s after a suspend refused for now #799 review measured this in three sequences: a resume hook, a suspend hook then a resume hook, and a suspend hook whose 30 seconds ended during the call.
  5. So the warning stated a refusal that had already ended.

Changes

  • _failed_call_warning reads suspending for an unknown outcome. The caller reads it under the lock, as before.
  • Refusal ended and the worker is not closed: "the suspend call for the idle MicroVM had an unknown outcome. The refusal of jobs already ended during the call, on a resume hook or after 30 seconds. So the worker accepts jobs. It tries the call again one idle time after the call and every job end."
  • Refusal ended after close(): "…So the worker accepts jobs. The worker is closed, so it does not try the call again." A closed worker starts no idle time.
  • A job can start during the call after a resume hook. A running job cancels the idle time. So the next call comes one idle time after the call and every job end, not at the outcome. The text says so.
  • README item 8 and the listener docstring item 8 now end with: "A resume hook during the call, or the end of a suspend hook's 30 seconds during the call, has already ended the refusal. The worker then accepts jobs at once, and tries the call again one idle time after the call and every job end."
  • The terminating text and the normal unknown text are unchanged. Docstring items 8 to 10 of _failed_call_warning describe the cases.
  • The decision-block comment now says that the text matches the decision for a rejection and a refusal for now. For an unknown outcome, the second locked block decides whether the 30-second refusal starts. The window between the two blocks existed on main and is tracked in MicroVM worker: the unknown-outcome warning and its refusal are decided in two locked blocks #820. This PR does not change it.
  • Behavior is unchanged. The texts and conditions match the JavaScript change in fix(microvm-worker): state that jobs are accepted after an unknown outcome whose refusal ended aws-durable-execution-sdk-js#1069, for MicroVM worker: the unknown-outcome warning says jobs are refused after the refusal ended aws-durable-execution-sdk-js#1056.

Testing

  • The three sequences, with and without a job. Each test asserts the full text logged once, and the next call one idle time after the call or the job end. The job cases also assert a 202 for a job right after the warning.
  • A job that starts during the call after a resume hook and still runs at the outcome. A second job gets 202. No call starts while a job runs. The next call comes one idle time after the last job ends.
  • The case of close() and a resume hook during the call. It logs the closed text, a job gets 202, and no call follows.
  • Guards that keep the old text while the refusal lasts: no hook, a suspend hook still in effect, and close() alone. Each gets 503 until 30 s after the call, then 202. Also a resume-then-terminate case that keeps the terminating text.
  • Mutation check on the first commit: removing, inverting, or reordering the new branch fails 1 to 17 tests.
  • hatch run dev-microvm-worker:cov: 450 passed, worker.py at 100 %. Typecheck, hatch fmt --check, and commit lint pass. Stress run with -n 8, twice: 450 passed.

The worker logs a warning when its own SuspendMicrovm call has an
unknown outcome. Without a terminate hook, the warning said: "The
worker refuses jobs until the MicroVM resumes, or for 30 seconds."

1. A resume hook during the call ends the refusal of jobs.
2. The 30-second limit of a suspend hook's refusal can also end during
   the call.
3. After an unknown outcome, the worker starts a new refusal only while
   `_suspending` is still set.
4. So after these sequences, the worker accepts jobs. A job right after
   the warning gets HTTP 202.
5. So the warning stated a refusal that had already ended.

`_failed_call_warning` now reads `suspending` for an unknown outcome.
The caller reads it under the lock, as before.

- When the refusal already ended, the warning says so. It also says
  that the worker accepts jobs and that the idle time starts again.
- After close() during the call, the same case says that the worker is
  closed and does not try the call again. A closed worker starts no
  idle time.
- The terminating text and the normal unknown text are unchanged.
- Behavior is unchanged.

The texts and conditions match the JavaScript worker's fix for
aws/aws-durable-execution-sdk-js#1056.

Fixes #806
The unknown-outcome warning for a refusal that ended during the call
said: "So the worker accepts jobs, and the idle time starts again."

1. A resume hook during the call ends the refusal of jobs.
2. A job can then start during the call.
3. A running job cancels the idle time.
4. So the idle time does not start at the outcome. It starts when the
   call and every job have ended.
5. So the text now says: "So the worker accepts jobs. It tries the call
   again one idle time after the call and every job end." The text for
   a refusal for now after a suspend hook ends the same way.

A new test covers this sequence. A job runs at the outcome. The next
call comes one idle time after the last job ends.

README item 8 and the listener docstring item 8 now state this case.
The `_failed_call_warning` docstring item 8 says the same.

The comment in the decision block said that the text matches the
decision whatever another thread does. That holds for a rejection and
a refusal for now, because the same block decides. For an unknown
outcome, the block at the end of `_call_suspend` decides whether the
30-second refusal starts. The comment now says so. Issue #820 tracks
the window between the two blocks.

Behavior is unchanged. The texts match the JavaScript worker's change
in aws/aws-durable-execution-sdk-js#1069.
ParidelPooya added a commit to aws/aws-durable-execution-sdk-js that referenced this pull request Oct 23, 2026
…wn outcome whose refusal ended

The ended text of the unknown-outcome warning said: "So the worker
accepts jobs, and the idle time starts again."

1. A resume hook during the call ends the refusal of jobs.
2. So a job can start during the call.
3. startIdleTime starts no idle time while a job runs.
4. So the idle time starts when the call and every job have ended, not
   when the warning is logged.
5. So the text now says: "So the worker accepts jobs. It tries the call
   again one idle time after the call and every job end." The
   refused-for-now text for the same state uses the same sentence.

README item 8 and the listener JSDoc item 8 now state this case. A resume
hook or the end of a suspend hook's 30 seconds during the call has already
ended the refusal. The worker then accepts jobs at once.

The second line of item 10 in the failedCallWarning JSDoc is now aligned
under its text.

Three new tests cover sequences that had no test:

- A job still runs at the outcome. The next call comes one idle time
  after the job ends.
- A resume hook and then a suspend hook arrive during the call. The
  warning keeps the refusal text.
- Two suspend hooks arrive during the call, and the call ends during the
  second hook's refusal. The warning keeps the refusal text.

Behavior is unchanged. aws/aws-durable-execution-sdk-python#817 makes the
same change.

Refs #1056

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@ParidelPooya
ParidelPooya marked this pull request as ready for review October 23, 2026 20:31
@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 20:31 — with GitHub Actions Inactive
@ParidelPooya
ParidelPooya merged commit 44b9f4d into main Oct 23, 2026
24 checks passed
@ParidelPooya
ParidelPooya deleted the fix/microvm-worker-unknown-outcome-warning branch October 23, 2026 20:31
ParidelPooya added a commit to aws/aws-durable-execution-sdk-js that referenced this pull request Oct 23, 2026
…tcome whose refusal ended (#1069)

Fixes #1056.

## Summary

The MicroVM worker logs a warning when its own SuspendMicrovm call has
an unknown outcome. Without a `terminate` hook, the warning said that
the worker refuses jobs. That was false when the refusal had already
ended during the call. This PR changes only which text the worker logs.
The behavior is unchanged.

## Why

1. The warning said: "The worker refuses jobs until the MicroVM resumes,
or for 30 seconds."
2. A `resume` hook during the call ends the refusal of jobs.
3. The 30-second limit of a `suspend` hook's refusal can also end during
the call. This needs a client from `createMicrovmsClient` that runs past
the worker's 10-second abort.
4. After an unknown outcome, the worker starts a new refusal only while
`suspending` is still set.
5. So in these sequences the worker accepts jobs. A job right after the
warning gets HTTP 202.
6. The likely sequence is the first one: the worker's own call suspended
the MicroVM, and its answer was lost across the freeze.

## Changes

- `failedCallWarning` reads `suspending` for an unknown outcome. The
catch block calls it before the unknown branch acts on the state.
- When the refusal already ended, the warning says: "the suspend call
for the idle MicroVM had an unknown outcome. The refusal of jobs already
ended during the call, on a resume hook or after 30 seconds. So the
worker accepts jobs. It tries the call again one idle time after the
call and every job end."
- The idle time cannot start while a job runs. A job can start during
the call after a `resume` hook. So the text names the call and every job
end, as the refused-for-now text for the same state does.
- A closed worker starts no idle time. So after `close()` during the
call, the same case says: "… So the worker accepts jobs. The worker is
closed, so it does not try the call again."
- The terminating text and the normal unknown text are unchanged.
- The JSDoc of `failedCallWarning` lists the new cases.
- README item 8 and the listener JSDoc item 8 end with: "A `resume` hook
during the call, or the end of a `suspend` hook's 30 seconds during the
call, has already ended the refusal. The worker then accepts jobs at
once, and tries the call again one idle time after the call and every
job end."
- The Python worker has the same text.
aws/aws-durable-execution-sdk-python#817 makes the same change, with
identical texts and conditions.

## Testing

- 12 new tests in `auto-suspend.test.ts`. The three sequences from the
issue each assert the full text, one warning, HTTP 202 for a job right
after, and the next call one idle time after that job ends.
- Other tests cover the idle time that starts at the call end, the
closed variant, a `resume` hook and then a `terminate` hook, and a guard
that keeps the old text with no hook, with a `suspend` hook still in
effect, and after `close()` alone.
- A job that still runs at the outcome: the warning has the new text,
and the next call comes one idle time after the job ends.
- A `resume` hook and then a `suspend` hook during the call, and two
`suspend` hooks with the call ending during the second refusal: both
keep the refusal text and refuse jobs for 30 seconds after the call.
- 6 of the new tests fail on `main`. With the old texts expected, all
tests pass on `main`. So calls, statuses, and timings are unchanged.
- Removing or inverting each new condition fails at least one test.
- `npm test` in the package passes 292 tests. Coverage rose from 96.57 %
to 96.59 % of statements, and from 92.53 % to 92.61 % of branches.
- `npm run build`, the package lint, the root `npm run lint`, and `npm
run format:check` pass.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

This branch was previously deployed

1 inactive deployment
ai-pr-review-runtime — 6e648e92 Deployed Oct 23, 2026 by ParidelPooya via notify / Generate notification summary #264
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MicroVM worker: the unknown-outcome warning says jobs are refused after the refusal ended

2 participants