Skip to content

fix(idempotency): avoid running durable executions twice - #5873

Merged
dreamorosi merged 3 commits into
aws-powertools:mainfrom
vishwakt:fix/5867-idempotency-durable-ownership
Nov 1, 2026
Merged

dreamorosi merged 3 commits into
aws-powertools:mainfrom
vishwakt:fix/5867-idempotency-durable-ownership

Conversation

@vishwakt

@vishwakt vishwakt commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A durable execution suspended past its first invocation kept the in-progress expiry of that invocation, so its record looked dead and another execution with the same payload could take it over. The replay branch also continued past any existing record, including another execution's lock or a completed result. This PR stores no in-progress expiry for durable executions and records the execution ARN as the owner of the record, so an execution only continues with its own record.

It also fixes a second bug: on main, a crash before the first checkpoint fails the whole durable execution, because the retry runs in ExecutionMode and is rejected by its own record.

Changes

  • makeIdempotent passes context.executionContext.durableExecutionArn to the handler. Durable executions save their in-progress record without an in-progress expiry, and saveInProgress no longer logs the registerLambdaContext warning when an owner is provided.
  • IdempotencyRecord has an optional owner. saveInProgress and saveSuccess accept it through the existing options object ({ identity, owner }), so the owner stays on the completed record too.
  • The handler checks the existing record whenever it hits IdempotencyItemAlreadyExistsError. It continues only with an INPROGRESS record owned by the same execution. A completed record returns the stored result, and any other record is handled like before, for example IdempotencyAlreadyInProgressError.
  • The ownership check doesn't depend on replay mode, because the SDK only switches to ReplayMode once more than one operation is checkpointed. A retry after a crash before the first checkpoint runs in ExecutionMode, and on main it is rejected by its own record, failing the execution. It now continues with its own record.
  • Records without an owner, written by earlier versions during a rolling deploy, keep today's replay behaviour, so an execution that started on the old version isn't locked out by its own record.
  • The owner attribute name is configurable through ownerAttr (default owner), added to PERSISTENCE_ATTRIBUTE_KEY_MAPPINGS and read and written by both the DynamoDB and cache persistence layers. No change was needed to their locking conditions, since both already treat a record without an in-progress expiry as locked until it expires.
  • Docs: new "Durable functions" section in docs/features/idempotency.md with a snippet, the recommendation to set expiresAfterSeconds to at least the execution timeout, a note on records that stay locked when an execution ends without cleanup, a note for custom persistence stores, and ownerAttr in both attribute tables. examples/snippets gets @aws/durable-execution-sdk-js as a dev dependency so the snippet type-checks.
  • Tests: a timeline test models the probe from the issue end to end with the real CachePersistenceLayer and an in-memory client. Execution A ships the order and is suspended, B starts with the same payload after A's first invocation would have timed out and is rejected, A resumes and completes, and B then gets the stored result, with the work running once. Other tests cover a foreign lock during replay, own, foreign and ownerless completed records during replay, a retry before the first checkpoint, and the owner round trip in both layers. All the new tests fail against main.

Issue number: closes #5867


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

A durable execution can be suspended past its first invocation, so the
in-progress expiry taken from that invocation made its record look dead and
let another execution with the same payload take it over. The replay branch
also continued past any existing record, including a completed one.

Durable executions now store no in-progress expiry and record their
execution ARN as the owner of the record. An execution continues only with
its own in-progress record, so a retry before the first checkpoint works too,
and a completed record returns the stored result. Records without an owner,
written by earlier versions, keep the previous replay behaviour.

The owner attribute name is configurable through ownerAttr in both the
DynamoDB and cache persistence layers, and the docs get a section on
durable functions.
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/XL PRs between 500-999 LOC, often PRs that grown with feedback label Oct 30, 2026
@svozza

svozza commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Thanks @vishwakt! I ran the probe from #5867 against this branch on Lambda (Node.js 24, eu-west-1), using main as a control for each scenario. Every scenario behaves as intended on this branch:

Scenario This PR main
DynamoDB: B starts with the same payload while A is suspended, then C starts after both finish A runs once. B fails with IdempotencyAlreadyInProgressError without entering the handler. C gets A's stored result. A and B both run to completion, and C gets B's result
Cache (ElastiCache Serverless, Valkey 8): same scenario same as DynamoDB same as DynamoDB
Crash after acquiring the record, before the first checkpoint the retry continues with its own record and the execution succeeds the retry is rejected by its own record, and the execution fails
Execution started on main, resumed on this branch before its record's deadline takes the ownerless fallback during replay and completes once —
Execution started on main, resumed on this branch after its record's deadline re-acquires the record and completes once —

One thing the deploy scenarios showed: a resumed durable invocation runs whatever code is on $LATEST at the time, not the code the execution started on. So the ownerless fallback is reachable in practice, as you intended.

A couple of small suggestions:

  1. Mention the second fix in the description. On main, a crash before the first checkpoint fails the whole durable execution, because the retry runs in ExecutionMode and is rejected by its own record. This PR fixes that too, which is worth stating.
  2. Note what happens when an execution stops without cleanup. Durable records no longer have an in-progress expiry. If an execution reaches its executionTimeout or is stopped, the record stays locked until expiresAfterSeconds. Combined with the recommendation to set that to at least the execution timeout, this could be a long time. A sentence next to that recommendation would help.

@vishwakt

Copy link
Copy Markdown
Contributor Author

Thanks @svozza for running the probe on both layers and the deploy scenarios, that's really reassuring. Good to know the ownerless fallback is reachable in practice because resumed invocations run on $LATEST.

Both suggestions are addressed:

  1. The description now calls out the second fix: on main, a crash before the first checkpoint fails the whole durable execution, because the retry runs in ExecutionMode and is rejected by its own record.
  2. Next to the expiresAfterSeconds recommendation, the docs now say that if an execution ends without cleaning up its record (it reaches its execution timeout or is stopped), the record stays INPROGRESS until it expires and other executions with the same payload get an IdempotencyAlreadyInProgressError. They also say that deleting the record from the persistence store lets the payload run again sooner.

@dreamorosi
dreamorosi requested a review from svozza October 30, 2026 19:52
@dreamorosi
dreamorosi merged commit 9608c05 into aws-powertools:main Nov 1, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL PRs between 500-999 LOC, often PRs that grown with feedback

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Idempotency runs durable executions twice when one is suspended past its first invocation

3 participants