feat: add beacon API error check playbook

[barnabasbusa]

Participation checks pass beacon API breaks between a beacon node and another team's validator client as long as votes still land. This dev playbook watches every validator client's beacon API traffic through its rpc-snooper metrics (clSnooperMetrics from ethereum-package#1502, route labels from rpc-snooper#59) and fails as soon as any request gets a 4xx or 5xx response, naming the pair, route and status. Stacked on the check_http_metrics aggregation PR. On a local Lodestar/Prysm crossed devnet it reported 3-geth-lodestar-prysm: POST /eth/v2/beacon/pool/attestations status 400 x8 from the Prysm VC.

Example

ethereum-package config (needs ethereum-package#1502 for clSnooperMetrics):

participants:
  - cl_type: prysm
    vc_type: lodestar
    snooper_enabled: true
  - cl_type: lodestar
    vc_type: prysm
    snooper_enabled: true
additional_services:
  - assertoor
assertoor_params:
  tests:
    - file: https://raw.githubusercontent.com/ethpandaops/assertoor/master/playbooks/dev/beacon-api-error-check.yaml
      config:
        watchDuration: 30m

Result from the local Lodestar/Prysm crossed devnet:

success  Check if all clients are ready
success  Skip startup epochs
failure  Watch beacon API responses — background task completed unexpectedly
failure  Beacon API errors per client pair — matrix task execution failed (1 failures)
failure  No beacon API errors for 3-geth-lodestar-prysm — assertions failed: beacon_api_errors: 1 of 3 series (want eq 0): {method="POST",server="cl-3-lodestar-geth:4000",status="400",uri="/eth/v2/beacon/pool/attestations",...} 8

[redpandabot]

Adds a dev playbook that loops over ethereum-package's clSnooperMetrics (name/url per validator-client↔beacon-node pair) and runs check_http_metrics against each snooper's ngx_request_count{status=~"[45].."} counter in delta+each mode, failing the run on any new 4xx/5xx. The config maps cleanly onto the task schemas: clSnooperMetrics elements do carry name/url, the metric and status label exist in rpc-snooper, ngx_request_count is a counter, and aggregate: each + missingSeries: pass + delta is the documented pattern for exactly this use (newly-appearing error series count from zero, pre-existing errors are excluded). With an empty clSnooperMetrics list the matrix finishes immediately and onBackgroundComplete: fail surfaces a failure rather than a silent pass, so the misconfiguration case is safe. I found no functional defect in the changed files.

---

_{Reviewed 2 changed file(s) @ 2a898aa1 — no blocking issues found.
"If debugging is the process of removing software bugs, then programming must be the process of putting them in." — Edsger Dijkstra}

[redpandabot]

Summary

Adds a dev playbook that watches each validator client's beacon API traffic via rpc-snooper metrics and fails on any 4xx/5xx, using the check_http_metrics aggregation from the stacked PR. The wiring is correct (metric name, labels, var paths and task options all verified against the task sources and the ethereum-package/rpc-snooper checkouts), but the fail-fast configuration still lets infrastructure blips abort the watch: a single failed scrape and a snooper counter reset both false-fail, and missingMetric: pass can false-pass on a stale snooper image.

Issues

• 🟡 playbooks/dev/beacon-api-error-check.yaml:51 — Any single scrape hiccup fails the whole 30m watchdog — see the thread on that line
• 🟡 playbooks/dev/beacon-api-error-check.yaml:60 — Counter reset after a snooper restart false-fails the watch — see the thread on that line
• 🟡 playbooks/dev/beacon-api-error-check.yaml:52 — missingMetric: pass silently greens a snooper without the metric — see the thread on that line

---

_{Reviewed @ 2e95be01
"If debugging is the process of removing software bugs, then programming must be the process of putting them in." — Edsger Dijkstra}

[redpandabot]

🟡 Counter reset after a snooper restart false-fails the watch

resetBehavior defaults to fail and the playbook doesn't set it. If the snooper container restarts during the 30m window, ngx_request_count drops below the recorded baseline and evaluateSeries raises 'counter reset detected', failing the watchdog though no API errors occurred; resetBehavior: rebaseline would make the check survive restarts.

[redpandabot]

🟡 missingMetric: pass silently greens a snooper without the metric

rpc-snooper registers ngx_request_count in init(), so a missing family means a stale/misbehaving snooper image, not 'no errors'. With missingMetric: pass and missingSeries: pass the playbook reports success in that case; consider missingMetric: fail so absence of data is surfaced instead of passing vacuously.