Repository navigation
Retry on a deadlock the update of the cluster id of a node registering again - #195
Merged
Merged
Conversation
…ster id a node registering again sets UpdateNodeFields ran its UPDATE on the connection pool, out of the ExecContext executor retrying a statement on a deadlock. A node registering again, with the new cluster id of a cluster rebuilt under the same node names, updates its cluster_id while the worker may write the nodes rows of the first feeds of its peer: the deadlock failed the registration at once with a 500 "cannot update node", in about one QA run in four of om3 PR 1232 (2026-11-18, test 020.03).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A node registering again with a new cluster id could fail its registration with a 500
cannot update node: the update of itscluster_idran out of the executor retrying a statement on a deadlock.Cause
POST /auth/nodeupdates thecluster_idof a node already known when it brings a new one (server/handlers/post_auth_node.go), as every om3 QA run does, a cluster rebuilt under the same node names.UpdateNodeFieldsran itsUPDATEwithoDb.DB.ExecContext, the connection pool directly, whileInsertNodeand most writes go throughoDb.ExecContext, which runs the statement in a transaction and retries it up to 9 times on a MariaDB deadlock (error 1213). The worker writing thenodesrows of the first feeds of the peer node deadlocks the update, which then fails at once.Seen in about one om3 QA run in four on 2026-11-18 (om3 PR #1232, test 020.03 "registers at the first try"): on a first node registering, and on a second node one second after its peer registered and sent its first asset, package, disk, sysreport and checks feeds. The other runs of the same batch registered at the first try, the collector up.
Change
UpdateNodeFieldsruns itsUPDATEthroughoDb.ExecContext; within a transaction (HasTx) the executor runs it on that transaction, as before.Not confirmed
The collector log was not read: the line
failed to update node cluster idat those times would tell the error, a deadlock or a lock wait timeout. A lock wait timeout is not retried by the executor either.