Skip to content

[go] Publish the loss of leadership before resigning - #1600

Open
hcrosse wants to merge 1 commit into
riverqueue:masterfrom
hcrosse:publish-leadership-loss-before-resign
Open

hcrosse wants to merge 1 commit into
riverqueue:masterfrom
hcrosse:publish-leadership-loss-before-resign

Conversation

@hcrosse

@hcrosse hcrosse commented Nov 14, 2026

Copy link
Copy Markdown

Publish lost leadership before trying to resign on shutdown, forced resignation or reelection deadline. Resignation currently delays the notification, so if the database becomes reachable after another client wins the lease, both clients can run maintenance. This closes the gap after #1213.

Rust, Java and JavaScript already stop leader work before resigning. Ruby runs maintenance on the same loop thread, so it needs no change.

The regression test fails without the change and passes with it. A reproduction that cuts the leader's Postgres connection until another client wins the lease sends 7–10 maintenance statements after reconnection without the fix, and none with it. make -k test passes on unmodified master and the patched tree on Linux. golangci-lint 2.14.0 reports 0 issues in every module.

Regression test output without the fix

From master edc210f8 with only the new test applied:

elector_test.go:367: Error: Not equal:
    expected: []bool{false}
    actual  : []bool{true}
--- FAIL: TestElectorRunLeaderState (0.00s)
    --- FAIL: TestElectorRunLeaderState/PublishesLostLeadershipBeforeResigning (1.44s)

With the fix, go test ./internal/leadership/... and go test -race -run TestElectorRunLeaderState ./internal/leadership/... pass.

Copilot AI balanced review requested due to automatic review settings November 14, 2026 02:41
When a leader leaves the leader state, through shutdown, a forced
resignation or an elapsed reelection deadline, publish the loss of
leadership before trying to resign. Resigning against an unreachable
database can take several seconds, and if another client wins the
lease meanwhile, both clients could run maintenance once the database
is reachable again.
@hcrosse
hcrosse force-pushed the publish-leadership-loss-before-resign branch from fc0258a to f149f74 Compare November 14, 2026 02:41

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the leadership-loss notification ordering change.

No open issues 🎉

What changed in this PR

Updates Go’s leader-election cleanup to publish lost leadership before attempting resignation, closing the notification delay left after #1213 so maintenance can stop sooner when the database is unreachable.

Changes:

  • Publishes leadership loss before resignation during leader-state cleanup.
  • Adds a regression test verifying leadership is cleared before resignation begins.
  • Documents the fix in the changelog.
File Description
internal/​leadership/​elector.go Moves the lost-leadership notification ahead of resignation.
internal/​leadership/​elector_test.go Adds a resignation mock hook and an ordering regression test.
CHANGELOG.md Records the leadership-notification timing fix.

Review effort: Balanced

@bgentry

bgentry commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Ah, thank you for submitting. It looks like the elector steps down on time, but it doesn't tell the maintenance services until the resign loop finishes, which can take ~13s when the database is unreachable. They keep running past lease expiry and can overlap with a new leader if the connection recovers.

Will take a close look at this soon and get this resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants