Skip to content

feat: report the last task status when a task times out - #239

Open
barnabasbusa wants to merge 1 commit into
masterfrom
bbusa/timeout-last-status
Open

barnabasbusa wants to merge 1 commit into
masterfrom
bbusa/timeout-last-status

Conversation

@barnabasbusa

@barnabasbusa barnabasbusa commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

When a check task runs out of time, by its own timeout or its parent's deadline, the only result was context deadline exceeded, so a failed run had to be dug out of the task logs to learn what never passed. This appends the task's last progress message to the timeout error (task timed out: context deadline exceeded, last status: Waiting for finality, finalized epoch 3 < 1000 (current epoch 5)) for every task, and makes check_consensus_finality report the condition that failed and check_consensus_sync_status name the clients that are not synced. Verified on a local devnet.

Examples

A check that never passes inside its timeout (from a local devnet run):

- name: check_consensus_finality
  timeout: 2m
  config:
    minFinalizedEpochs: 1000
before: context deadline exceeded
after:  task timed out: context deadline exceeded, last status: Waiting for finality, finalized epoch 3 < 1000 (current epoch 5) (attempt 6)

The same applies when the parent's deadline cancels the task, e.g. a check inside run_tasks_concurrent with a timeout:

before: context deadline exceeded
after:  context deadline exceeded, last status: Waiting for sync... 2/3, not synced: 3-geth-lodestar-prysm (attempt 12)

check_consensus_finality with failOnCheckMiss:

before: finality check failed
after:  finality check failed: 5 unfinalized epochs > 3 (finalized epoch 9, current epoch 14)

@barnabasbusa
barnabasbusa force-pushed the bbusa/timeout-last-status branch from c0ce7d5 to ba7f2aa Compare October 13, 2026 10:57
@barnabasbusa
barnabasbusa changed the base branch from master to bbusa/pin-staticcheck October 13, 2026 10:57
@barnabasbusa
barnabasbusa force-pushed the bbusa/timeout-last-status branch from ba7f2aa to ef8fcf7 Compare October 13, 2026 11:14
@barnabasbusa
barnabasbusa changed the base branch from bbusa/pin-staticcheck to bbusa/go-1.26 October 13, 2026 11:14
@barnabasbusa
barnabasbusa force-pushed the bbusa/timeout-last-status branch from ef8fcf7 to 7d83089 Compare October 13, 2026 11:16

@redpandabot redpandabot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change wraps task execution errors with a timeoutError helper that appends the task's last progress message when the task's own timer fired or the error wraps context.DeadlineExceeded, adds a lock-protected GetProgressMessage accessor, and makes the finality and sync-status checks report their failure conditions and unsynced clients. I traced every return path and lock interaction and found the logic, locking and error semantics consistent; the new unit test matches the helper's branches.


Reviewed 5 changed file(s) @ 7d830895 — no blocking issues found.
"RTFM."

Base automatically changed from bbusa/go-1.26 to master October 13, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants