Skip to content

Retry on a deadlock the update of the cluster id of a node registering again - #195

Merged
sghf merged 1 commit into
opensvc:mainfrom
cvaroqui:node-register-retry
Nov 18, 2026
Merged

sghf merged 1 commit into
opensvc:mainfrom
cvaroqui:node-register-retry

Conversation

@cvaroqui

Copy link
Copy Markdown
Member

A node registering again with a new cluster id could fail its registration with a 500 cannot update node: the update of its cluster_id ran out of the executor retrying a statement on a deadlock.

Cause

POST /auth/node updates the cluster_id of a node already known when it brings a new one (server/handlers/post_auth_node.go), as every om3 QA run does, a cluster rebuilt under the same node names. UpdateNodeFields ran its UPDATE with oDb.DB.ExecContext, the connection pool directly, while InsertNode and most writes go through oDb.ExecContext, which runs the statement in a transaction and retries it up to 9 times on a MariaDB deadlock (error 1213). The worker writing the nodes rows of the first feeds of the peer node deadlocks the update, which then fails at once.

Seen in about one om3 QA run in four on 2026-11-18 (om3 PR #1232, test 020.03 "registers at the first try"): on a first node registering, and on a second node one second after its peer registered and sent its first asset, package, disk, sysreport and checks feeds. The other runs of the same batch registered at the first try, the collector up.

Change

  • UpdateNodeFields runs its UPDATE through oDb.ExecContext; within a transaction (HasTx) the executor runs it on that transaction, as before.
  • A test checks the update goes through the executor and sends the expected statement; it fails on the former code.

Not confirmed

The collector log was not read: the line failed to update node cluster id at those times would tell the error, a deadlock or a lock wait timeout. A lock wait timeout is not retried by the executor either.

…ster id a node registering again sets

UpdateNodeFields ran its UPDATE on the connection pool, out of the
ExecContext executor retrying a statement on a deadlock. A node registering
again, with the new cluster id of a cluster rebuilt under the same node
names, updates its cluster_id while the worker may write the nodes rows of
the first feeds of its peer: the deadlock failed the registration at once
with a 500 "cannot update node", in about one QA run in four of om3 PR
1232 (2026-11-18, test 020.03).
@sghf
sghf merged commit 5d544af into opensvc:main Nov 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants