Repository navigation
Conversation
When a leader leaves the leader state, through shutdown, a forced resignation or an elapsed reelection deadline, publish the loss of leadership before trying to resign. Resigning against an unreachable database can take several seconds, and if another client wins the lease meanwhile, both clients could run maintenance once the database is reachable again.
hcrosse
force-pushed
the
publish-leadership-loss-before-resign
branch
from
November 14, 2026 02:41
fc0258a to
f149f74
Compare
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues block approval of the leadership-loss notification ordering change.
No open issues 🎉
What changed in this PR
Updates Go’s leader-election cleanup to publish lost leadership before attempting resignation, closing the notification delay left after #1213 so maintenance can stop sooner when the database is unreachable.
Changes:
- Publishes leadership loss before resignation during leader-state cleanup.
- Adds a regression test verifying leadership is cleared before resignation begins.
- Documents the fix in the changelog.
| File | Description |
|---|---|
internal/leadership/elector.go |
Moves the lost-leadership notification ahead of resignation. |
internal/leadership/elector_test.go |
Adds a resignation mock hook and an ordering regression test. |
CHANGELOG.md |
Records the leadership-notification timing fix. |
Review effort: Balanced
Contributor
|
Ah, thank you for submitting. It looks like the elector steps down on time, but it doesn't tell the maintenance services until the resign loop finishes, which can take ~13s when the database is unreachable. They keep running past lease expiry and can overlap with a new leader if the connection recovers. Will take a close look at this soon and get this resolved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publish lost leadership before trying to resign on shutdown, forced resignation or reelection deadline. Resignation currently delays the notification, so if the database becomes reachable after another client wins the lease, both clients can run maintenance. This closes the gap after #1213.
Rust, Java and JavaScript already stop leader work before resigning. Ruby runs maintenance on the same loop thread, so it needs no change.
The regression test fails without the change and passes with it. A reproduction that cuts the leader's Postgres connection until another client wins the lease sends 7–10 maintenance statements after reconnection without the fix, and none with it.
make -k testpasses on unmodified master and the patched tree on Linux. golangci-lint 2.14.0 reports 0 issues in every module.Regression test output without the fix
From master
edc210f8with only the new test applied:With the fix,
go test ./internal/leadership/...andgo test -race -run TestElectorRunLeaderState ./internal/leadership/...pass.