Skip to content

fix(circuit_breaker): start a fresh probe success streak on each probe election - #8646

Merged
leandrodamascena merged 2 commits into
aws-powertools:developfrom
unknownhackerworld:fix/circuit-breaker-stale-probe-successes
Oct 20, 2026
Merged

leandrodamascena merged 2 commits into
aws-powertools:developfrom
unknownhackerworld:fix/circuit-breaker-stale-probe-successes

Conversation

@unknownhackerworld

Copy link
Copy Markdown
Contributor

Issue number: closes #8645

Summary

Changes

The half-open probe success streak lives in the module-level _LOCAL_SUCCESSES dict in aws_lambda_powertools/utilities/circuit_breaker/base.py. Before this change it was reset in only two places:

  • _call_probe, when a probe fails in this environment
  • _call_probe, when this environment reaches success_threshold and closes the circuit

Nothing reset it when a worker won a new probe election in handle(). So a streak from an earlier recovery window carried into the next one. With success_threshold=3:

  1. Env A wins the election and gets 2 successful probes.
  2. Env A goes idle, its probe lease lapses, env B takes over, B's probe fails and the circuit reopens. A's counter is still 2, because B runs in a different execution environment.
  3. In the next window env A wins the election again, and its first success brings the count to 3 and closes the circuit.

That breaks the documented rule ("If success_threshold consecutive probes succeed, the circuit closes") and sends full traffic back to a downstream that just failed a probe.

The fix adds a small _reset_probe_successes() helper, which sets _LOCAL_SUCCESSES[name] = 0 under _COUNTERS_LOCK. handle() calls it right after try_acquire_half_open succeeds, on both paths:

  • the OPEN path (recovery_timeout elapsed, fresh election)
  • the HALF_OPEN expired-lease takeover path

The thread that already owns the lease skips the election (record.half_open_owner == _probe_owner_id()), so a streak within one window is unchanged. Only a newly won election starts from zero. This matches how _LAST_OBSERVED_STATE already throws away a stale failure streak after an external close (test_stale_local_failures_reset_on_external_close).

The lock is held only for the dict write, never across persistence I/O or callbacks, as the existing comment on _COUNTERS_LOCK requires.

Tradeoffs

  • On a lease takeover, the new owner now needs the full success_threshold successes and no longer inherits the old owner's partial streak. That means a few more probes before closing. It follows the "correctness over recovery latency" choice from fix(circuit_breaker): make probe election per-thread and synchronize local counters #8323, which noted the inherited streak as pre-existing and left it unchanged.
  • No changes to the persistence schema, the DynamoDB layer, the public API or the docs. The docs already describe the intended behaviour.

Alternatives considered

  • Reset the success streak whenever this env sees OPEN or CLOSED in handle(), like _LAST_OBSERVED_STATE does for failures. This misses the lease takeover path, where the record goes from HALF_OPEN (old owner) to HALF_OPEN (new owner) without passing through OPEN or CLOSED. Tying the reset to winning the election covers both paths in one place.
  • Persist the success count in the store. This would make the count shared across environments, but it adds a write per probe and changes the record schema. That is a much bigger change for the same guarantee.

User experience

Before (success_threshold=3, after the scenario above):

call()  # first probe of the new recovery window
store.db["c"].state  # CLOSED after one probe

After:

call()  # first probe of the new recovery window
store.db["c"].state  # HALF_OPEN, 1 of 3
call(); call()
store.db["c"].state  # CLOSED after 3 consecutive successes

Single-environment functions that never lose their lease behave the same as before.

Tests

Two new regression tests in tests/functional/circuit_breaker/test_circuit_breaker.py, next to the existing bug regressions:

  • test_new_probe_election_does_not_reuse_successes_from_an_earlier_recovery_window: 2 successes, then an external reopen, then a new election. The new window needs its own 3 successes.
  • test_probe_lease_takeover_starts_a_fresh_success_streak: a takeover of an expired lease in an environment that has a leftover streak of 2. The first success counts as 1 and the circuit stays HALF_OPEN.

Both fail on develop (the circuit closes after one probe) and pass with this change. Locally:

  • pytest tests/functional/circuit_breaker tests/unit/circuit_breaker: 85 passed
  • ruff check and ruff format --check on the changed files: clean
  • mypy aws_lambda_powertools/utilities/circuit_breaker/base.py: no issues

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

…e election


_LOCAL_SUCCESSES is module-level and was only reset when a probe failed
in the same environment or when this environment closed the circuit. A
streak left from an earlier recovery window (this environment probed,
lost its lease, and another environment reopened the circuit) counted
towards the next window, so the circuit could close after fewer than
success_threshold probes.

Reset the streak right after this worker wins try_acquire_half_open, on
both the OPEN path and the expired-lease takeover path. Add regression
tests for both paths.

Closes aws-powertools#8645
@unknownhackerworld
unknownhackerworld requested a review from a team as a code owner October 20, 2026 05:02
@unknownhackerworld
unknownhackerworld requested review from svozza and a balanced review from Copilot and removed request for Copilot October 20, 2026 05:02
@boring-cyborg

boring-cyborg Bot commented Oct 20, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@boring-cyborg boring-cyborg Bot added the tests label Oct 20, 2026
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 20, 2026
Copilot AI balanced review requested due to automatic review settings October 20, 2026 09:36

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@sonarqubecloud

Copy link
Copy Markdown

@leandrodamascena leandrodamascena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @unknownhackerworld, this is great work! I ran it through the same multi-environment setup as yesterday, including 4 processes against a real DynamoDB table, and the circuit never closed early anymore, while develop still did. The two tests cover exactly the two election paths. Approving now, this will wait here a bit until the release, no need to rebase or touch anything, and you'll get the credit in the release notes. Thank you!

@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.93%. Comparing base (d82a695) to head (f586a9e).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #8646   +/-   ##
========================================
  Coverage    96.93%   96.93%           
========================================
  Files          318      318           
  Lines        16031    16036    +5     
  Branches      1423     1423           
========================================
+ Hits         15540    15545    +5     
  Misses         349      349           
  Partials       142      142           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@leandrodamascena
leandrodamascena merged commit 4fe0d6d into aws-powertools:develop Oct 20, 2026
14 checks passed
@boring-cyborg

boring-cyborg Bot commented Oct 20, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

@powertools-for-aws-oss-automation

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: circuit breaker can close after fewer than success_threshold probes when an environment wins a new probe election

3 participants