Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions pages/validators/_meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,7 @@
"genvm-configuration": "GenVM Configuration",
"greybox-setup": "Greybox Setup",
"upgrade": "Upgrade Guide",
"state-recovery": "State Recovery & Re-sync",
"rollup-outage": "Rollup / Sequencer Outage",
"changelog": "Changelog"
}
4 changes: 4 additions & 0 deletions pages/validators/monitoring.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,10 @@ curl -s http://localhost:9153/metrics | grep genlayer_node_
For production validators, we recommend setting up a complete monitoring stack with Prometheus and Grafana. This enables real-time visibility into your validator's performance and helps identify issues before they impact your validator's operation.
</Callout>

<Callout type="info">
Monitor the `/health` condition states for `chain.rpc` and `chain.websocket`, the age of the latest L2 head, and `blocksBehind` from `gen_syncing`. See [Rollup / Sequencer Outage](/validators/rollup-outage) to distinguish a provider failure from a chain-wide stall, or [State Recovery & Re-sync](/validators/state-recovery) when only local replay is behind.
</Callout>

## Logs and Metrics Forwarding

You can forward your logs and metrics to external systems for centralized monitoring and alerting by using the service `alloy` provided in the `docker-compose.yaml` from the extracted tarball.
Expand Down
194 changes: 194 additions & 0 deletions pages/validators/rollup-outage.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
---
description: "Diagnose Bradbury Phase 2 connectivity and sequencer incidents, compare Orbit chain heads, and verify validator recovery."
---

import { Callout } from "nextra-theme-docs";

# Rollup / Sequencer Outage: Incident Response

A validator follows GenLayer Chain over WebSocket and submits votes and receipts through its HTTP RPC. Bradbury Phase 2 runs on Arbitrum Orbit with AnyTrust data availability and settles to Ethereum Sepolia. A sequencer stall can stop new L2 blocks even while the RPC serves past blocks. A single provider or a local connection can also fail while the chain keeps advancing.

<Callout type="info" emoji="💡">
Keep the node running while you establish the scope. A chain-wide halt prevents the consensus contracts from recording new actions while the chain is stopped. If only your node loses connectivity, the chain can continue and your validator can miss duties. Check the chain head and your validator’s status again when service returns.
</Callout>

This runbook follows the incident lifecycle: **detect → communicate → wait & verify → resume → post-incident.**

---

## Earlier Bradbury incident

The July 1, 2026 Bradbury incident predates the Phase 2 Orbit deployment. Operators coordinated recovery when validators lost access to the underlying chain. For a current incident, verify the network identity, transports and L2 head before deciding whether to escalate a chain-wide outage.

<Callout type="info">
Two distinct things happen and it helps to keep them apart. **Not-active** is a **node-side** state: a node that cannot reach the chain stops being "synced" and voluntarily skips its duties (it will not vote on stale state). **Quarantine** is an **on-chain** consensus state, driven by missed activity windows (idleness) or appeal outcomes — mechanisms that require the chain to be *advancing*. In a total rollup outage the chain is frozen, so the outage window itself does not manufacture fresh quarantines; the exposure comes at the **edges** — as the chain resumes and idleness is assessed — and from **asymmetric** failures where your node is unreachable while the chain keeps moving.
</Callout>

<Callout type="warning" emoji="⚠️">
After connectivity returns, check the validator’s on-chain status. A node catching up can resume duties once it is synced and eligible. An on-chain quarantine has its own release conditions.
</Callout>

---

## 1. Detect

Run the transport checks separately and compare the latest L2 block number and timestamp with an independent, verified Phase 2 endpoint or the chain explorer on [Networks](/developers/networks#testnet-bradbury). A stalled head across providers and a failed connection to one provider call for different responses.

### Run doctor

`doctor` connects to the configured GenLayer Chain RPC and WebSocket URLs with a short timeout and reports each explicitly:

```bash copy
./bin/genlayernode doctor
```

During an outage you will see failures like:

```
✗ GenLayer Chain RPC: Failed to connect to <your-rpc-url>
Error: <dial/connection error>
Fix: Ensure GenLayer Chain is running and accessible at the configured URL
✗ GenLayer Chain WebSocket: Failed to connect to <your-ws-url>
...
Some GenLayer Chain checks failed!
```

A timeout identifies a connectivity failure from this node. Verify the endpoint, Conduit key, DNS and network identity before attributing it to the sequencer.

### Check the health endpoint

On Bradbury node v0.6.0-rc.5, inspect `conditions["chain.rpc"].state` (HTTP) and `conditions["chain.websocket"].state` (subscriptions) in the [operations health response](https://github.com/genlayerlabs/genlayer-node/blob/v0.6.0-rc.5/docs/api/ops/health.md). Each condition reports its own failure reason. `chain.sync` and `gen_syncing` show whether local processing is keeping up.

```bash copy
curl -s http://localhost:9153/health
```

Monitor both condition states. A transport can be degraded while `/health` still returns HTTP 200. Compare independent L2 heads even when both transports are healthy.

### Read the logs

The node's rollup transport classifies connection/timeout failures as **transient** and retries them with backoff rather than crashing. During an outage the recurring log lines are about the connection, not consensus:

- `Error checking zksync connection` and `Failed to reconnect zksync.` are literal messages from the [v0.6.0-rc.5 WebSocket keep-alive loop](https://github.com/genlayerlabs/genlayer-node/blob/v0.6.0-rc.5/internal/node/infrastructure/zksync/keepalive.go). The legacy wording refers to the configured GenLayer Chain connection.
- `redialed ws rpc client` / `ws rpc redial failed` as the WebSocket RPC client re-dials a stale connection.
- `retrying RPC request due to timeout` (with `attempt` / `max_attempts`) from the HTTP RPC retry path.
- `[TRANSACT] Gas bump retry hit transient RPC error, retaining budget` when a pending submission cannot land — the node backs off and **keeps** its replacement budget rather than burning it.

<Callout type="info">
Timeouts, connection failures, and provider degradation are treated as **transient**: they are retried, they do **not** burn the transaction replacement budget, and they do **not** create nonce gaps. Blockchain reverts and client-side mistakes are classified separately and never trip the breaker — so a rollup outage is handled as "wait for the endpoint to come back," exactly as intended.
</Callout>

<Callout type="info" emoji="📊">
The transport-layer **circuit breaker** is observable through **metrics, not logs**. Watch `genlayer_rollup_rpc_circuit_state` (0 = closed, 1 = half-open, 2 = open) and `genlayer_rollup_rpc_circuit_transitions_total` on your metrics endpoint. When gated, calls return a `transport circuit breaker is open` transient error internally. Its thresholds are currently compiled-in constants and cannot be retuned from `config.yaml`.
</Callout>

### How the node protects itself while the L2 is down

You do not need to intervene to prevent damage — the transport layer already backs off intelligently:

| Mechanism | Behavior during an outage |
|---|---|
| **Retry classification** | Connection/timeout errors are marked transient and retried with backoff; they do not exhaust the replacement budget or leave nonce gaps. |
| **Adaptive throughput** | As the failure rate climbs, request concurrency is cut and delays grow (near-full speed → moderate backoff → one-at-a-time with long waits), so the node stops hammering a dead or degraded endpoint. |
| **Circuit breaker** | After consecutive provider failures the breaker **opens** and gates calls, moves to **half-open** to probe, and **closes** again on success. A write-only fault gates only submissions but keeps reads flowing; a total fault gates everything because the endpoint is presumed dead. |
| **Duty gating** | When the node can no longer track the chain head it stops being "synced," **skips its consensus duties, and pauses its sync pipeline** — it will not vote on stale state. This is the node going *not-active*; it is protective, not a fault. |
| **Keep-alive / reconnect** | The WebSocket connection and RPC client are monitored and automatically re-established when the endpoint returns. |

<Callout type="warning" emoji="🚨">
**Do not treat a rollup outage as a local-node problem.** Deleting your chain database, re-running the wizard, or re-staking will not help — the L2 is down for everyone — and a needless database wipe forces a slow [full re-sync](/validators/state-recovery#path-b--full-re-sync-from-genesis-trustless) once the network returns. Confirm the outage is upstream first.
</Callout>

---

## 2. Communicate

A rollup outage is a network event. Coordinated communication prevents every operator from independently debugging the same shared failure.

- **Confirm scope.** Compare the reported HTTP and WebSocket conditions, head progress at an independent provider and reports from other validators. If only one endpoint is affected, contact its operator. Escalate a shared stall through the network’s operator channel.
- **Report in the operator channel.** Include the UTC start time, network, last observed L2 block, node version and redacted `doctor` or log output. Omit provider credentials.
- **Watch for a status update.** The Foundation team coordinates the rollup restart. Expect an acknowledgement of the outage, and later a resume signal. Do not act on rumors — wait for the coordinated all-clear.

<Callout type="info">
Share the affected transport and time range when comparing telemetry with other operators. See [Monitoring & Telemetry](/validators/monitoring) for metrics forwarding.
</Callout>

---

## 3. Wait & Verify

While the rollup is down there is no useful consensus work to do. The goal of this phase is to keep the node **ready** so it rejoins cleanly the moment the L2 returns.

- **Leave the node running.** Its transport keeps retrying and reconnecting automatically. Stopping it gains nothing and risks a cold start later.
- **Do not modify the database, config, or stake.** Nothing about your node caused the outage; changing things now only adds a second variable to debug.
- **Keep watching detection signals.** Poll `/health`, rerun `doctor` and confirm that fresh L2 heads arrive. A reachable endpoint may still be serving the last block from before the stall.
- **Confirm the outage is genuinely upstream, not you.** If *other* validators recover and you do not, re-check your own RPC/WS URLs, firewall, and DNS — at that point it may be a local connectivity problem masquerading as an outage. See [Node State Recovery & Re-sync](/validators/state-recovery) if your node fell behind but the network is up.

---

## 4. Resume

Recovery is designed to be **hands-off on the node side**.

### The node reconnects on its own

Once the rollup RPC and WebSocket are reachable again:

1. The keep-alive loop re-establishes the WebSocket subscription.
2. The transport circuit breaker probes (**half-open**) and, on a successful call, returns to **closed** — normal request flow resumes.
3. The node catches up any blocks it missed during the outage. Track progress with `gen_syncing`:

```bash copy
curl -s -X POST http://localhost:9151 \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"gen_syncing","params":[],"id":1}'
```

Wait for `blocksBehind` to return to `0`. If the outage was long and your node fell far behind, follow [Node State Recovery & Re-sync](/validators/state-recovery) to bring it back to the tip.

### Re-entering the active set

Validators that were moved to **quarantined / not-active** during the outage rejoin through the network's normal activation path — **priming**, not re-staking. Your stake and identity are untouched.

```bash copy
# Where does my validator stand after the outage?
genlayer staking active-validators
genlayer staking quarantined-validators
genlayer staking validator-info --validator 0xYourValidatorWallet...
```

- Validators must be **primed** each epoch (`validatorPrime()`) to be eligible for selection. Your node **retries priming on its own** until it succeeds — while the chain is down you will see log lines like `Validator could not prime for current epoch. It will not validate until primed. Retrying prime.`, and once the chain is back and the previous epoch has finalized, priming lands and duties resume automatically. See [Validator Priming](/understand-genlayer-protocol/core-concepts/optimistic-democracy/staking#epoch-activation-and-priming).
- Quarantine is managed **on-chain by the consensus contracts**, not by a node-side flag you can clear. A temporary idleness quarantine has an on-chain end epoch and lifts by itself once that epoch passes; lifting one early is an administrative / tribunal action, **not** an operator self-service command. Follow the Foundation's coordinated resume — do not try to force re-activation unilaterally.

<Callout type="warning" emoji="⚠️">
Because re-activation is coordinated across the whole validator set, resume the way the Foundation directs. Independently re-staking or wiping state does not speed up your return and can leave you worse off (a needless full re-sync).
</Callout>

---

## 5. Post-Incident

- **Verify participation.** Confirm the node is producing the expected consensus activity in [monitoring](/validators/monitoring) and that `gen_syncing` reports `blocksBehind: 0`.
- **Confirm you are in the active set.** `genlayer staking active-validators` should list your validator again; it should no longer appear in `quarantined-validators`.
- **Improve detection.** Alert on `chain.rpc` and `chain.websocket` condition states, stale L2 heads and a growing `blocksBehind`. Each signal covers a different failure mode.
- **Keep a timeline.** Record when you detected the outage, what the errors were, and when you recovered — it shortens triage next time and helps the Foundation correlate reports.

---

## Quick Reference

| Situation | Action |
|---|---|
| `doctor` shows RPC and WebSocket failures | Compare with an independent provider and other operators to establish the scope. |
| `doctor` shows "Not configured" | Configuration problem, not an outage — fix the URL |
| Only your node fails; others are fine | Local connectivity issue — check your firewall/DNS/RPC, then [State Recovery](/validators/state-recovery) |
| Rollup back, `blocksBehind` shrinking | Normal — the node is catching up; wait |
| Rollup back, `blocksBehind` stays high | Check whether local sync is advancing and follow [State Recovery](/validators/state-recovery). |
| Validator shows as not-active or quarantined after recovery | Check sync, epoch priming and the on-chain quarantine status before taking action. |

## Related

- [Monitoring & Telemetry](/validators/monitoring) — alert on `/health` and sync lag so outages page you
- [Node State Recovery & Re-sync](/validators/state-recovery) — bring a node that fell behind back to the tip
- [Setup Guide](/validators/setup-guide) — network RPC/WebSocket and consensus configuration
- [Slashing](/understand-genlayer-protocol/core-concepts/optimistic-democracy/slashing) — what slashing actually penalizes (and why an outage does not)
- [Staking](/understand-genlayer-protocol/core-concepts/optimistic-democracy/staking) — priming, the active set, and quarantine
Loading
Loading