feat: aggregate and regex-match series in check_http_metrics

[barnabasbusa]

check_http_metrics required every assertion to match exactly one series, so labelled counters such as error responses per route and status could not be checked at all. This adds labelPatterns (regex label matchers, e.g. status: "[45]..") and aggregate: sum compares the sum of all matching series and each requires every series to pass and names the ones that do not, e.g. {status="400",uri="/eth/v2/beacon/pool/attestations"} 8. In delta mode the baselines are kept per series and a series that appears during the run counts from zero, so an error counter that only gets created by the first error is still caught, and the failure details go into the progress and error messages.

Examples

Fail on any new error response of a labelled counter, naming every series that moved:

- name: check_http_metrics
  config:
    url: "http://cl-3-lodestar-geth:8008/metrics"
    continueOnPass: true
    failOnCheckMiss: true
    missingMetric: pass
    missingSeries: pass
    assertions:
    - name: rest_errors
      metric: lodestar_api_rest_errors_total
      aggregate: each
      mode: delta
      operator: eq
      value: 0

Regex label match, 4xx/5xx responses through an rpc-snooper (output from a local devnet run):

    assertions:
    - name: beacon_api_errors
      metric: ngx_request_count
      labelPatterns:
        status: "[45].."
      aggregate: each
      mode: delta
      operator: eq
      value: 0

assertions failed: beacon_api_errors: 1 of 3 series (want eq 0): {method="POST",status="400",uri="/eth/v2/beacon/pool/attestations",...} 8

Sum across series with a budget:

    - name: few_errors
      metric: ngx_request_count
      labelPatterns:
        status: "5.."
      aggregate: sum
      mode: delta
      operator: lte
      value: 5

assertions failed: few_errors: sum 6 (want lte 5): {route="a",status="500"} 2, {route="b",status="503"} 4

Without aggregate the selector still has to match exactly one series, now also usable with labelPatterns.

[redpandabot]

Summary

Adds labelPatterns regex selectors and sum/each aggregation with per-series delta baselines to check_http_metrics. The feature logic, regex anchoring, and tests look sound, but the two issues flagged in the previous review are both still present in the rewritten history: the first scrape of an aggregate delta assertion bypasses missingMetric/missingSeries handling, and the baselines output goes stale after a per-series rebaseline.

Issues

• 🟡 pkg/tasks/check_http_metrics/task.go:536 — aggregate delta ignores missingMetric/missingSeries on the first scrape — see the thread on that line
• 🟡 pkg/tasks/check_http_metrics/task.go:572 — baselines output goes stale after an aggregate rebaseline — see the thread on that line

---

_{Reviewed @ 37abb783
"If you can't measure it, you can't improve it." — Peter Drucker}

[redpandabot]

🟡 aggregate delta ignores missingMetric/missingSeries on the first scrape

In evaluateSeries, the delta baseline is recorded and waiting returned at lines 536-544 before the mf == nil / zero-series checks, so missingMetric/missingSeries are never consulted on the first poll. The single-series path calls handleMissing immediately, so with aggregate: each|sum, missingMetric: fail or missingSeries: fail does not fail on the first scrape but one poll later; recording the empty baseline and honoring fail are not mutually exclusive.

[redpandabot]

🟢 baselines output goes stale after an aggregate rebaseline

On ResetBehaviorRebaseline the per-series map is updated in place (line 573), but the assertion-level t.baselines[a.Name] written at line 538 is never refreshed, so the exposed baselines output keeps reporting the pre-reset sum. The single-series path updates both.