diff --git a/CLAUDE.md b/CLAUDE.md index b32eec1df..ef68b3779 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -45,7 +45,7 @@ crates/ └─ test-fixtures/ # Spec-fixture loading (prod dep of rpc's Hive test driver) net/ ├─ api/ # Actor protocol traits wiring BlockChain ↔ P2P - ├─ p2p/ # libp2p: gossipsub + req-resp (Status, BlocksByRoot, BlocksByRange) + ├─ p2p/ # libp2p: gossipsub + req-resp (Status, BlocksByRoot, BlocksByRange, gloas envelopes) │ ├─ src/gossipsub/ # Topic encoding, message handling │ ├─ src/req_resp/ # Request/response codec and handlers │ └─ src/metrics.rs # Peer connection/disconnection tracking @@ -334,13 +334,28 @@ actual_slot = finalized_slot + 1 + relative_index - Beacon wire: `validate_messages()` is on, so every beacon message waits for a verdict (~4.2s before gossipsub's cache evicts it). Rules in `state_transition::beacon::gossip` (cheap half inline, stateful half on a bounded `spawn_blocking` task); plumbing in `p2p/src/beacon/verdict.rs`. Lean gossip still auto-forwards - Data columns: every check runs in p2p. A column gossip did not accept (`Queue`/`Overloaded`), every fetched column, and parked columns replayed after their parent imports go through `column::chain_checks` in `p2p/src/beacon/column_checks.rs`. The chain actor stores what it gets unchecked; only debug builds re-run `chain_checks` there - Beacon subscribes seven global topics plus two node-id-derived subnet families: custody - columns and backbone attestation subnets. `beacon_aggregate_and_proof` and + columns and backbone attestation subnets. Gloas digests add two more topics, + `execution_payload` and `payload_attestation_message` (`BeaconTopics::for_fork`; earlier + digests never carry them). Gloas has its own rules for `beacon_block`, + `data_column_sidecar` (fork enum `DataColumnSidecar`, fork from the topic's digest), + aggregates (`SignedAggregateAndProof::Gloas`, aggregation-bits length bounded before + expansion) and attestations (`verify_attestation_payload_status`). Deliberate + departures: with no bad-block cache a block seen without a post-state is queued or + ignored where the spec rejects; an envelope whose block state is not cached is queued and + forwarded to the actor (which verifies every envelope) rather than dropped; payload votes + are applied without a second signature check. `beacon_aggregate_and_proof` and `beacon_attestation_{subnet_id}` validate in p2p like blocks/columns, with their own permit pool. The actor applies only accepted aggregates (gossip's, plus the Beacon API's own validated `aggregate_and_proofs` submissions), attesting indices already verified, held one slot per `validate_on_attestation`'s `current_slot >= data.slot + 1`; subnet attestations are verified and relayed but never applied. See [`docs/beacon_wire.md`](docs/beacon_wire.md) + - Fork-digest rollover happens at runtime (`ForkSchedule`, `beacon::transition`): the next + digest's topics are joined `SUBSCRIBE_LEAD_EPOCHS` before its boundary and the previous + one's left `UNSUBSCRIBE_LAG_EPOCHS` after, blob-parameter (BPO) boundaries included. + **Known gap**: ethrex's `DiscoveryServer` cannot replace the served ENR at runtime, so its + `eth2` entry keeps the startup digest until a restart; peers discovering the node after a + boundary reject the record - **Req/Resp**: Status, BlocksByRoot, BlocksByRange (snappy frame compression + varint length) - Beacon adds `beacon_blocks_by_{range,root}/2` alongside its Status/Ping/MetaData/Goodbye set. Both serve from the checkpoint-anchored store, and `build_status` advertises it @@ -350,6 +365,11 @@ actual_slot = finalized_slot + 1 + relative_index the chain actor, since the store's head trails a delivered batch by the whole actor mailbox) and the store's head (which gossip imports move and the watermark never sees). Fetched blocks reach the actor as `BlockSource::Sync`. + - Gloas adds `execution_payload_envelopes_by_{range,root}/1`. Server: canonical FULL + envelopes only, and the head's envelope only when the head node is FULL. Client: by root + on `needs_envelope` (same dedup and retry ladder as block fetches), by range after a + gloas span's blocks were handed to the actor, with range sync holding the next batch + until the envelope answer is forwarded (the actor keeps unmatched envelopes only briefly) Their chunks carry four `` (the block's own epoch's fork digest), which is why `BeaconWire` and the codec carry `genesis_validators_root`. See [`docs/beacon_wire.md`](docs/beacon_wire.md) @@ -452,20 +472,23 @@ against them. `beacon::tests::the_built_in_network_derives_what_it_always_did` pins the parsed mainnet config to `Config::mainnet()`, so the file and the Rust constant cannot drift apart. Sepolia's config schedules gloas: its `GLOAS_*` keys are claimed (`ForkName::Gloas` exists, and `Config` carries the -fork's schedule, timing and churn fields), and its state transition and fork -choice exist, but the live follower refuses gloas blocks -(`process_or_pend_block`) and a gloas anchor (`refuse_unfollowable_fork` in -`main.rs`), since nothing delivers payload envelopes or payload attestations to -it, so a Sepolia follower stops tracking the chain at `GLOAS_FORK_EPOCH` -regardless. `network::warn_if_unfollowed_fork_scheduled` (shared by -`BuiltInNetwork::resolve` and a loaded `NetworkDir`) warns about this -explicitly at startup, once per scheduled fork for which `ForkName::is_followed` -answers no (the one place that says which forks the node follows), separately -from the ignored-keys warning, which now only ever names heze's keys and +fork's schedule, timing and churn fields), and the live follower follows it +(`ForkName::Gloas.is_followed()`): the chain actor imports gloas blocks and +their payload envelopes, the p2p layer delivers both and the payload votes, and +the follower crosses `GLOAS_FORK_EPOCH` without a restart (fork-digest +rollover, see Networking). `refuse_unfollowable_fork` in `main.rs` and +`network::warn_if_unfollowed_fork_scheduled` (shared by +`BuiltInNetwork::resolve` and a loaded `NetworkDir`) are now inert guards for +the next fork the node cannot follow: the first refuses such an anchor with +`UnsupportedFork`, the second warns once per scheduled fork for which +`ForkName::is_followed` answers no (the one place that says which forks the +node follows). The ignored-keys warning only ever names heze's keys and `GAS_LIMIT_SCHEDULE`/`INCLUSION_LIST_DUE_BPS`. Platåberget (the Glamsterdam testnet) is already past its `GLOAS_FORK_EPOCH`, -so its checkpoint anchor is a gloas state that `refuse_unfollowable_fork` -refuses; its entry is there for when the live follower takes gloas blocks. Its +so its checkpoint anchor is a gloas state: it is accepted, starts with no +payload known (head EMPTY), and its first FULL child is held until the anchor's +envelope arrives (best-effort from the checkpoint URL's +`execution_payload_envelopes` endpoint, else the by-root request). Its file says `CONFIG_NAME: 'testnet'` (a placeholder for Prysm), so `BuiltInNetwork::resolve` sets every built-in's `CONFIG_NAME` to its `--network` name. @@ -663,6 +686,11 @@ transitions are in `ethlambda-types`, per the section above. Nothing above tests inside the module, are `#[cfg_attr(not(feature = ...), ignore)]` so they report as ignored rather than silently absent. Turning the feature on is what `make test-beacon` does, and it runs `--lib` too so those 15 are not missed. + `ethlambda-blockchain` has its own `beacon-spec-tests` feature gating the + fixture-driven chain-actor tests (the gloas envelope cases in its `--lib` + target, which read `consensus-spec-tests`); `make test-beacon-mainnet` runs + them, and without the feature they report as ignored, so a plain `cargo test` + needs no download. - Every fixture case is its own test, named `////`, so a failure names the case and not the suite around it. The spec binary therefore supplies its own harness (`harness = false`), since a case is only @@ -726,9 +754,33 @@ transitions are in `ethlambda-types`, per the section above. Nothing above suites. - **Gloas** lives in `stf/gloas.rs`, `stf/epoch/gloas.rs`, `helpers/gloas.rs`, the gloas section of `fork_choice.rs`, and `containers/gloas.rs` in - `ethlambda-types`. **The live follower does not follow gloas**: it refuses - gloas blocks and a gloas anchor until envelopes and payload attestations are - delivered to it. `get_head_node` is one bottom-up walk for every fork, with + `ethlambda-types`. **The live follower follows gloas.** The chain actor's + envelope path is `beacon_envelope.rs`: an envelope waits for its block's + post-state, then for every sampled column when the bid has commitments + (data availability moves from the block to the envelope), then goes to + `on_execution_payload_envelope`; a block whose parent is FULL with an + unverified payload is held until that envelope verifies. All queues are + capped, aged out and swept at finality, a missing parent envelope is re-asked + once a slot (`FetchRequest.needs_envelope`, asked at once when the block is + held, whatever its source, since `BlockSource::Sync` also covers by-root + answers), an envelope for a block that is stored but not imported is checked + against that block's bid and held outside the per-arrival-slot cap + (`awaiting_import`, which range catch-up needs), and `beacon_columns.rs` bounds + the parking of gloas column sidecars (they carry no signature). Fork-choice + events are timed at arrival, not at the slot tick, and the head's payload + status is kept with its root (`Store::head_payload_status`, recomputed after + a restart). The node stays a follower: the validator-client endpoints still + refuse a gloas epoch (`refuse_validator_duties_from_gloas`). + With `--execution-endpoint`, an envelope that passes the consensus checks is + judged by `engine_newPayloadV5` (INVALID with a null `latestValidHash` is a + hash mismatch that refuses only that envelope; INVALID with one condemns the + payload via the execution-chain walk; an engine error keeps the envelope and + retries once a slot, probing with the oldest), and a gloas head sends + `engine_forkchoiceUpdatedV4` with the custody columns (head hash by the head + node's FULL or EMPTY status, finalized and safe from the checkpoint blocks' + `bid.parent_block_hash`, safe being a fallback for fast confirmation). See + [`docs/beacon_engine.md`](docs/beacon_engine.md). + `get_head_node` is one bottom-up walk for every fork, with the current slot's fork selecting the payload rules and the boost gate, so a follower on any network that schedules gloas (Sepolia does) pays work proportional to its votes plus blocks every slot and decodes no block in the @@ -823,11 +875,20 @@ snapshot (`States`) + diff (`StateDiffs`) pairs; `BlockRoots` and `LiveChain` index by slot for range serving and fork choice. Attestations and gossip signatures are not persisted; they live in in-memory `Store` buffers consumed during the tick pipeline. See [`docs/data_storage.md`](docs/data_storage.md) -for the full reference: what each of the eleven tables holds and how it's +for the full reference: what each of the thirteen tables holds and how it's keyed, the snapshot/diff reconstruction algorithm, the block-import write sequence, pruning rules, what never changes at runtime, and startup/restore behavior. +- `ExecutionPayloadEnvelopes` (verified gloas envelopes, keyed by slot and + root) and `BlockTimeliness` (the gloas part of `block_timeliness`) are + reloaded on resume for the unfinalized window; the PTC vote vectors are + reseeded empty and each payload gets a verdict seeded for the execution + client to promote. Adding them did not bump `DB_VERSION`. +- A restart of a beacon data directory currently hits + `SpecAssert("block_root in store.unrealized_justifications")` in head + computation until lambdaclass/ethlambda#628 (persists unrealized + justifications) lands; that is not gloas-specific. - `BlockProof` is the only pruned block table (below the finalized boundary); `get_signed_block` returns `None` for a pruned finalized block. - A `StateDiff` omits `config` and `validators`, trusting they never mutate; diff --git a/Cargo.lock b/Cargo.lock index 10236a9b3..ed5443c8a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1991,6 +1991,7 @@ dependencies = [ "libssz-types 0.3.0 (git+https://github.com/lambdaclass/libssz?rev=5cb14378ffdadcb51b387c003567e2630d0c7073)", "serde", "serde_json", + "snap", "spawned-concurrency", "thiserror 2.0.18", "tokio", diff --git a/Makefile b/Makefile index f8fe3c95d..10796aff3 100644 --- a/Makefile +++ b/Makefile @@ -55,6 +55,7 @@ test-beacon: test-beacon-mainnet test-beacon-minimal ## 🧪 Run the Beacon Chai test-beacon-mainnet: consensus-spec-tests cryptography-specs ## 🧪 Run the Beacon Chain spec tests, mainnet preset $(BEACON_TEST) --features beacon-spec-tests + cargo test -p ethlambda-blockchain --lib --profile release-fast --features beacon-spec-tests test-beacon-minimal: consensus-spec-tests cryptography-specs ## 🧪 Run the Beacon Chain spec tests, minimal preset $(BEACON_TEST) --features beacon-spec-tests,preset-minimal diff --git a/README.md b/README.md index e0fb368bd..7b139b5d6 100644 --- a/README.md +++ b/README.md @@ -132,9 +132,11 @@ provider. See [`ethlambda beacon`](#ethlambda-beacon--the-ethereum-beacon-chain) below for the remaining flags. -`--network plataberget` (the Glamsterdam testnet) is built in as well, but -Platåberget has run gloas since its `GLOAS_FORK_EPOCH` and the follower does -not take gloas blocks or a gloas anchor yet, so the node cannot follow it yet. +`--network plataberget` (the Glamsterdam testnet) is built in as well. It has +run gloas since its `GLOAS_FORK_EPOCH`, which the follower follows: a gloas +checkpoint anchor is accepted, and payload envelopes are fetched and imported. +It stays a follower, so the validator-client endpoints still refuse a gloas +epoch. ### Lean consensus devnet diff --git a/bin/ethlambda/Cargo.toml b/bin/ethlambda/Cargo.toml index 1e239cae3..59b38642d 100644 --- a/bin/ethlambda/Cargo.toml +++ b/bin/ethlambda/Cargo.toml @@ -64,7 +64,7 @@ libc.workspace = true # `test-util` for `#[tokio::test(start_paused = true)]`: the checkpoint-sync # tests would otherwise wait out the real retry backoff. Dev-only, so the # feature never reaches the shipped binary. -tokio = { workspace = true, features = ["test-util"] } +tokio = { workspace = true, features = ["test-util", "net", "io-util", "macros", "rt"] } # Keygen writes a directory of key files and the benchmark corpus writes a # key set, so their tests need a throwaway directory. tempfile = "3" diff --git a/bin/ethlambda/src/beacon.rs b/bin/ethlambda/src/beacon.rs index 0528e5724..4af0cb8e0 100644 --- a/bin/ethlambda/src/beacon.rs +++ b/bin/ethlambda/src/beacon.rs @@ -24,11 +24,10 @@ //! all. `node`'s checkpoint sync is untouched: a lean node fetches a //! *finalized* anchor, which genuinely has no local source. +use ethlambda_p2p::beacon::fork_schedule::ForkSchedule; use ethlambda_p2p::beacon::swarm::BeaconWireConfig; use ethlambda_types::beacon::config::Config; -use ethlambda_types::beacon::constants::FAR_FUTURE_EPOCH; use ethlambda_types::beacon::containers::BeaconState; -use ethlambda_types::beacon::fork_digest::{compute_fork_digest, next_fork_boundary}; use ethlambda_types::beacon::preset; use ethlambda_types::beacon::primitives::{Epoch, Root}; use ethlambda_types::enr::EnrForkId; @@ -47,22 +46,11 @@ pub fn time_at_epoch(config: &Config, genesis_time: u64, epoch: Epoch) -> u64 { genesis_time + epoch * config.seconds_per_slot * preset::SLOTS_PER_EPOCH } -/// The `eth2` ENR entry for this chain at this epoch. -/// -/// `next_fork_*` point at the next boundary that moves the digest, which -/// includes blob-parameter-only forks. Peers tolerate a difference here by -/// design: only `fork_digest` has to match. +/// The `eth2` ENR entry for this chain at this epoch, from the same schedule +/// the running node follows, so the record it starts with and the one it would +/// switch to name the same boundaries. pub fn enr_fork_id(config: &Config, genesis_validators_root: Root, epoch: Epoch) -> EnrForkId { - let fork_digest = compute_fork_digest(config, genesis_validators_root, epoch); - // With no boundary ahead, the spec says to repeat the current fork's own - // version and name `FAR_FUTURE_EPOCH` as the epoch it activates at. - let boundary = next_fork_boundary(config, epoch); - let named_epoch = boundary.unwrap_or(epoch); - EnrForkId { - fork_digest, - next_fork_version: config.fork_version(config.fork_at_epoch(named_epoch)), - next_fork_epoch: boundary.unwrap_or(FAR_FUTURE_EPOCH), - } + ForkSchedule::new(config, genesis_validators_root).enr_fork_id(epoch) } /// Ethereum mainnet's genesis `BeaconState`, SSZ-encoded: a test fixture. @@ -187,28 +175,32 @@ pub fn wire_params( ); ethlambda_p2p::metrics::set_beacon_fork_digest(&digest_hex); - // The digest is computed once. Crossing a boundary while running strands - // this node on topic names nobody publishes to, so say when that is. + // The digest moves at every fork and blob-schedule boundary, and the p2p + // actor follows it without a restart (`ethlambda_p2p::beacon::transition`): + // the next digest's topics are joined ahead of the boundary, the switch + // happens at it, and the old topics are left after it, per + // `SUBSCRIBE_LEAD_EPOCHS` and `UNSUBSCRIBE_LAG_EPOCHS`. Say what is coming. // - // The gloas boundary gets its own message rather than the generic one: a - // restart there recomputes the digest and resubscribes to the right - // topics, the same as any other boundary, but it does not make this - // build any more able to import a gloas block or cross a gloas epoch - // boundary (the chain actor refuses every gloas block, since nothing - // delivers payload envelopes or payload attestations to it yet). - // Suggesting a restart "crosses" it would read as a stall a restart - // clears, which this is not. - match next_fork_boundary(&chain, epoch) { - Some(boundary) if boundary == chain.gloas_fork_epoch => warn!( - boundary_epoch = boundary, - boundary_unix_time = time_at_epoch(&chain, genesis.genesis_time, boundary), - "The fork digest changes at this boundary, but this build cannot follow past it: \ - a restart resubscribes to the right topics, not past the gloas gap itself" + // A fork the node does not follow keeps its own warning: crossing its + // digest works like any other, but the node stops tracking the chain + // there regardless. No fork is in that state today, since gloas is + // followed, so the warning is the guard for the next one. + let schedule = ForkSchedule::new(&chain, genesis.genesis_validators_root); + match schedule.next_boundary_after(epoch) { + Some(next) if !next.fork.is_followed() => warn!( + boundary_epoch = next.activation_epoch, + boundary_unix_time = time_at_epoch(&chain, genesis.genesis_time, next.activation_epoch), + fork = next.fork.as_str(), + fork_digest = %hex::encode(next.digest), + "The fork digest changes at this boundary and the node will switch topics, but this \ + build cannot follow the fork itself" ), - Some(boundary) => info!( - boundary_epoch = boundary, - boundary_unix_time = time_at_epoch(&chain, genesis.genesis_time, boundary), - "The fork digest changes at this boundary; restart the node to cross it" + Some(next) => info!( + boundary_epoch = next.activation_epoch, + boundary_unix_time = time_at_epoch(&chain, genesis.genesis_time, next.activation_epoch), + fork = next.fork.as_str(), + fork_digest = %hex::encode(next.digest), + "The fork digest changes at this boundary; the node will cross it without a restart" ), None => info!("No fork or blob-schedule boundary is scheduled"), } @@ -293,7 +285,7 @@ pub fn wire_params( #[cfg(test)] mod tests { use super::*; - use ethlambda_types::beacon::constants::CUSTODY_REQUIREMENT; + use ethlambda_types::beacon::constants::{CUSTODY_REQUIREMENT, FAR_FUTURE_EPOCH}; use ethlambda_types::beacon::fork::ForkName; /// Mainnet's genesis, 2020-12-01 12:00:23 UTC. diff --git a/bin/ethlambda/src/checkpoint_sync.rs b/bin/ethlambda/src/checkpoint_sync.rs index 62f84bf48..8c4467b21 100644 --- a/bin/ethlambda/src/checkpoint_sync.rs +++ b/bin/ethlambda/src/checkpoint_sync.rs @@ -25,7 +25,7 @@ use std::time::Duration; use ethlambda_p2p::beacon::decode; use ethlambda_types::beacon::config::Config; -use ethlambda_types::beacon::containers::{BeaconState, SignedBeaconBlock}; +use ethlambda_types::beacon::containers::{BeaconState, SignedBeaconBlock, gloas}; use ethlambda_types::beacon::preset; use ethlambda_types::block::SignedBlock; use ethlambda_types::genesis::{GenesisMismatch, verify_state_genesis}; @@ -61,6 +61,12 @@ fn beacon_block_path(slot: u64) -> String { format!("/eth/v2/beacon/blocks/{slot}") } +/// Path of the execution-payload-envelope endpoint on a Beacon API server, for +/// the block at `root`. +fn beacon_envelope_path(root: H256) -> String { + format!("/eth/v1/beacon/execution_payload_envelopes/0x{root:x}") +} + /// Maximum attempts to refetch the anchor pair if the state and block roots don't match. const MAX_ANCHOR_FETCH_ATTEMPTS: u32 = 3; @@ -700,6 +706,77 @@ pub async fn fetch_beacon_anchor( Ok((state, block)) } +/// Why a peer did not hand over the anchor's envelope. +/// +/// `Unknown` is not "the payload is empty": a peer answers 404 both for a +/// payload it never saw and one that was withheld, and a checkpoint source +/// cannot tell the two apart for us either. +#[derive(Debug, thiserror::Error)] +enum EnvelopeFetchError { + #[error("the peer does not know the anchor's envelope")] + Unknown, + #[error("request failed: {0}")] + Request(#[from] reqwest::Error), + #[error("SSZ deserialization failed: {0:?}")] + Decode(DecodeError), + #[error("the envelope is for another block")] + WrongBlock, +} + +/// Fetch the execution payload envelope of the block at `anchor_root` from one +/// peer. +async fn fetch_envelope( + client: &Client, + base_url: &str, + anchor_root: H256, +) -> Result { + let url = format!("{base_url}{}", beacon_envelope_path(anchor_root)); + let response = client + .get(&url) + .header("Accept", "application/octet-stream") + .send() + .await?; + if response.status() == reqwest::StatusCode::NOT_FOUND { + return Err(EnvelopeFetchError::Unknown); + } + let bytes = response.error_for_status()?.bytes().await?; + let envelope = gloas::SignedExecutionPayloadEnvelope::from_ssz_bytes(&bytes) + .map_err(EnvelopeFetchError::Decode)?; + if envelope.message.beacon_block_root != anchor_root { + return Err(EnvelopeFetchError::WrongBlock); + } + Ok(envelope) +} + +/// Best-effort fetch of a gloas anchor's execution payload envelope, so the +/// node can serve and build on the anchor's payload without waiting for a FULL +/// child to make the req/resp by-root path fetch it. +/// +/// Tries each URL in order and returns the first envelope that names +/// `anchor_root`. Every failure is logged at `info!` and skipped, including a +/// 404: it means the peer does not know the payload, which is not evidence the +/// payload is empty, so the caller must not conclude that. Verification is the +/// chain actor's job; this only transports. +pub async fn fetch_anchor_envelope( + urls: &[String], + anchor_root: H256, +) -> Option { + let client = build_client() + .inspect_err(|err| info!(%err, "Skipping the anchor envelope fetch")) + .ok()?; + for url in urls { + let base_url = trim_trailing_slash(url); + match fetch_envelope(&client, base_url, anchor_root).await { + Ok(envelope) => { + info!(%url, "Fetched the anchor's execution payload envelope"); + return Some(envelope); + } + Err(err) => info!(%url, %err, "No anchor execution payload envelope from this peer"), + } + } + None +} + /// Try each checkpoint URL in order, then retry the whole round, exactly as /// the lean path does. The two loops are shared rather than duplicated: /// `try_urls_in_order` and the fixed backoff mean the same thing on both @@ -1289,3 +1366,131 @@ mod tests { )); } } + +/// The anchor-envelope fast path, against a one-shot HTTP server. +#[cfg(test)] +mod anchor_envelope_tests { + use super::*; + use libssz::SszEncode as _; + use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; + use tokio::net::TcpListener; + + /// Serves `status` and `body` to every request until dropped, and returns + /// the base URL. The request line is captured on the channel. + async fn serve( + status: &'static str, + body: Vec, + ) -> (String, tokio::sync::mpsc::UnboundedReceiver) { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let url = format!("http://{}", listener.local_addr().unwrap()); + let (tx, rx) = tokio::sync::mpsc::unbounded_channel(); + tokio::spawn(async move { + loop { + let Ok((mut socket, _)) = listener.accept().await else { + return; + }; + let mut buffer = vec![0u8; 4096]; + let read = socket.read(&mut buffer).await.unwrap_or(0); + let request = String::from_utf8_lossy(&buffer[..read]); + let line = request.lines().next().unwrap_or_default().to_owned(); + let _ = tx.send(line); + let head = format!( + "HTTP/1.1 {status}\r\ncontent-length: {}\r\nconnection: close\r\n\r\n", + body.len() + ); + let _ = socket.write_all(head.as_bytes()).await; + let _ = socket.write_all(&body).await; + let _ = socket.shutdown().await; + } + }); + (url, rx) + } + + fn envelope_for(root: H256) -> gloas::SignedExecutionPayloadEnvelope { + let payload = gloas::ExecutionPayload { + parent_hash: Default::default(), + fee_recipient: Default::default(), + state_root: Default::default(), + receipts_root: Default::default(), + logs_bloom: vec![0u8; preset::BYTES_PER_LOGS_BLOOM] + .try_into() + .expect("built at exactly BYTES_PER_LOGS_BLOOM"), + prev_randao: Default::default(), + block_number: 7, + gas_limit: 0, + gas_used: 0, + timestamp: 0, + extra_data: Default::default(), + base_fee_per_gas: Default::default(), + block_hash: Default::default(), + transactions: Default::default(), + withdrawals: Default::default(), + blob_gas_used: 0, + excess_blob_gas: 0, + block_access_list: Default::default(), + slot_number: 0, + }; + gloas::SignedExecutionPayloadEnvelope { + message: gloas::ExecutionPayloadEnvelope { + payload, + execution_requests: Default::default(), + builder_index: 3, + beacon_block_root: root, + parent_beacon_block_root: H256::ZERO, + }, + signature: Default::default(), + } + } + + #[tokio::test] + async fn the_envelope_is_fetched_from_the_peers_endpoint() { + let root = H256::from([7u8; 32]); + let envelope = envelope_for(root); + let (url, mut requests) = serve("200 OK", envelope.to_ssz()).await; + + let fetched = fetch_anchor_envelope(&[url], root).await; + + assert_eq!(fetched, Some(envelope)); + let line = requests.recv().await.unwrap(); + assert!( + line.starts_with(&format!( + "GET /eth/v1/beacon/execution_payload_envelopes/0x{root:x} " + )), + "unexpected request: {line}" + ); + } + + #[tokio::test] + async fn a_404_is_unknown_and_falls_through_to_the_next_peer() { + let root = H256::from([7u8; 32]); + let envelope = envelope_for(root); + let (unknown, _) = serve("404 Not Found", Vec::new()).await; + let (knows, _) = serve("200 OK", envelope.to_ssz()).await; + + assert_eq!( + fetch_anchor_envelope(std::slice::from_ref(&unknown), root).await, + None, + "a 404 is not an envelope" + ); + assert_eq!( + fetch_anchor_envelope(&[unknown, knows], root).await, + Some(envelope) + ); + } + + #[tokio::test] + async fn a_failing_or_wrong_envelope_is_ignored() { + let root = H256::from([7u8; 32]); + let (broken, _) = serve("500 Internal Server Error", Vec::new()).await; + let (garbage, _) = serve("200 OK", vec![1, 2, 3]).await; + let (other, _) = serve("200 OK", envelope_for(H256::from([8u8; 32])).to_ssz()).await; + + assert_eq!(fetch_anchor_envelope(&[broken], root).await, None); + assert_eq!(fetch_anchor_envelope(&[garbage], root).await, None); + assert_eq!( + fetch_anchor_envelope(&[other], root).await, + None, + "an envelope for another block must not be handed over" + ); + } +} diff --git a/bin/ethlambda/src/main.rs b/bin/ethlambda/src/main.rs index f0f698a69..0f3a26b39 100644 --- a/bin/ethlambda/src/main.rs +++ b/bin/ethlambda/src/main.rs @@ -47,7 +47,8 @@ use ethlambda_crypto::signature::ValidatorSecretKey; use ethlambda_engine::types::ClientVersionV1; use ethlambda_engine::{EngineClient, JwtSecret}; use ethlambda_network_api::{ - InitBlockChain, InitP2P, ToBlockChainToP2PRef, ToP2PToBlockChainRef, ToRpcToP2PRef, + BlockArrival, InitBlockChain, InitP2P, ToBlockChainToP2PRef, ToP2PToBlockChainRef, + ToRpcToP2PRef, }; use ethlambda_p2p::{ LeanWireConfig, P2P, PeerId, SwarmConfig, WireConfig, attestation_subscription_subnets, @@ -206,6 +207,9 @@ enum ChainActor { /// `--execution-endpoint` was not given. engine: Option, safe_slots_to_import_optimistically: u64, + /// The checkpoint-sync URLs to ask for a gloas anchor's execution + /// payload envelope; empty when there is nothing to ask. + anchor_envelope_urls: Vec, }, } @@ -632,8 +636,10 @@ async fn run_node(options: Options) -> eyre::Result<()> { // A handshake failure is not a reason to refuse to run: the // execution client may simply be starting up, and every // call that matters has its own retry ladder. + let gloas_scheduled = store.config().gloas_fork_epoch + != ethlambda_types::beacon::constants::FAR_FUTURE_EPOCH; let _ = client - .handshake(&ours) + .handshake(&ours, gloas_scheduled) .await .inspect_err(|err| warn!(%err, "Engine API handshake failed")); @@ -665,6 +671,7 @@ async fn run_node(options: Options) -> eyre::Result<()> { engine, safe_slots_to_import_optimistically: mainnet .safe_slots_to_import_optimistically, + anchor_envelope_urls: clean_checkpoint_urls.clone(), }, } } @@ -775,6 +782,7 @@ async fn run_node(options: Options) -> eyre::Result<()> { let _ = served.inspect_err(|err| error!(%err, "RPC server failed")); }); + let mut anchor_envelope_fetch: Option<(Vec, H256)> = None; let blockchain = match setup.chain { ChainActor::Lean(validator_keys, config) => { BlockChain::spawn(setup.store, validator_keys, config, events) @@ -783,8 +791,12 @@ async fn run_node(options: Options) -> eyre::Result<()> { custody_columns, engine, safe_slots_to_import_optimistically, + anchor_envelope_urls, } => { check_custody_set(&custody_columns)?; + anchor_envelope_fetch = missing_anchor_envelope_root(&setup.store) + .filter(|_| !anchor_envelope_urls.is_empty()) + .map(|root| (anchor_envelope_urls, root)); BlockChain::spawn_beacon( setup.store, sync_status, @@ -815,6 +827,23 @@ async fn run_node(options: Options) -> eyre::Result<()> { }) .inspect_err(|err| error!(%err, "Failed to send InitBlockChain — actors not wired"))?; + // After both `Init*` sends, so the actor has its p2p handle when an + // envelope parks waiting for columns. Off the startup path: the fetch is best-effort and a slow peer must not + // delay the node. The envelope goes through the same chain-actor entry a + // gossiped one does, so the bid check, the column check and the + // verification all apply to it. + if let Some((urls, anchor_root)) = anchor_envelope_fetch { + let chain = blockchain.actor_ref().to_p2p_to_block_chain_ref(); + tokio::spawn(async move { + if let Some(envelope) = checkpoint_sync::fetch_anchor_envelope(&urls, anchor_root).await + { + let _ = chain + .new_execution_payload_envelope(Box::new(envelope), BlockArrival::now()) + .inspect_err(|err| warn!(%err, "Failed to hand over the anchor's envelope")); + } + }); + } + wait_for_shutdown(RunningNode { p2p, blockchain, @@ -825,6 +854,25 @@ async fn run_node(options: Options) -> eyre::Result<()> { Ok(()) } +/// The root of the anchor block when it is a gloas block whose execution +/// payload envelope the store does not hold. +/// +/// The anchor is the finalized block. A node that checkpoint-synced onto a +/// gloas block has the block and state but not the payload its children may +/// build on; asking the checkpoint source for it saves waiting for a FULL +/// child to make the by-root request fetch it. +fn missing_anchor_envelope_root(store: &Store) -> Option { + let root = store.latest_finalized().ok()?.root; + let block = store.get_signed_block(&root).ok()??; + // Gloas or any later fork (lean sorts last, and is not a beacon fork). + let fork = block.fork_name(); + if fork < ForkName::Gloas || fork == ForkName::Lean { + return None; + } + let held = store.get_execution_payload_envelope(&root).ok()?.is_some(); + (!held).then_some(root) +} + /// Reject a beacon node with no custody set before it spawns. /// /// `data_availability_for` used to refuse this per block. It cannot any more: @@ -1751,19 +1799,16 @@ fn first_config_difference( None } -/// Refuses an anchor state in `fork` if this node cannot follow that fork yet, -/// currently gloas. +/// Refuses an anchor state in `fork` if this node cannot follow that fork. /// -/// `fork_choice::get_forkchoice_store` accepts a gloas anchor, since fork choice -/// itself handles the fork. This node's wiring does not: nothing delivers -/// payload envelopes or payload attestations to the chain actor, and -/// `process_or_pend_block` refuses every gloas block, so a follower anchored -/// here would sit at its anchor forever, looking alive while importing -/// nothing. Refusing at startup reports the real reason instead. Checked ahead -/// of any store construction, on both anchor sources (a loaded network's -/// genesis state can schedule `GLOAS_FORK_EPOCH: 0`, and a checkpoint provider -/// can serve a gloas finalized state), so a rejected anchor writes nothing to -/// the data directory. +/// Every fork is followed today, gloas included, so this lets every anchor +/// through. A gloas anchor starts with no payload known (`payloads` is empty +/// in `fork_choice::get_forkchoice_store`), so its head is EMPTY and the +/// first FULL child is held until the actor fetches the parent's envelope. +/// The guard stays for the next fork the node cannot follow: checked ahead of +/// any store construction, on both anchor sources (a loaded network's genesis +/// state and a checkpoint provider's finalized state), so a rejected anchor +/// writes nothing to the data directory. /// /// Keeps the refusal distinguishable from a peer serving a mismatched anchor /// pair: reporting both as `AnchorPairingMismatch` would tell an operator to @@ -2259,24 +2304,15 @@ mod tests { assert!(validate_aggregate_subnet_ids(Some(&[]), 4).is_ok()); } - /// Fork choice accepts a gloas anchor, so the node has to be the one to - /// refuse it: nothing here delivers the payload envelopes a gloas chain - /// needs. Every fork this node does follow must still be let through. + /// Gloas is followed: the envelopes and payload attestations a gloas chain + /// needs are delivered, so no fork is refused as an anchor. #[test] - fn startup_refuses_a_gloas_anchor_and_only_a_gloas_anchor() { - assert!(matches!( - refuse_unfollowable_fork(ForkName::Gloas), - Err(checkpoint_sync::CheckpointSyncError::UnsupportedFork { - fork: ForkName::Gloas - }) - )); + fn startup_accepts_an_anchor_at_every_beacon_fork() { for fork in ForkName::ALL { - if fork != ForkName::Gloas { - assert!( - refuse_unfollowable_fork(fork).is_ok(), - "{fork} must stay followable" - ); - } + assert!( + refuse_unfollowable_fork(fork).is_ok(), + "{fork} must be followable" + ); } } @@ -2792,11 +2828,10 @@ validators: } /// A loaded network can schedule gloas at epoch 0, which makes its own - /// genesis state a gloas one. Fork choice accepts that anchor, so startup - /// has to be what refuses it, with the reason that names the fork, and - /// before anything is written to the data directory. + /// genesis state a gloas one. That is a legitimate anchor now that gloas + /// is followed: the store is built, at the genesis slot. #[tokio::test] - async fn a_gloas_genesis_is_refused_before_the_store_is_built() { + async fn a_gloas_genesis_becomes_an_anchor() { use ethlambda_state_transition::beacon::config::Config; use ethlambda_state_transition::beacon::upgrade::upgrade_state; @@ -2837,6 +2872,10 @@ validators: state = upgrade_state(&state, fork, &config).unwrap(); } assert_eq!(state.fork_name(), ForkName::Gloas); + // A real gloas genesis commits to gloas's empty body; the upgrade + // chain above leaves phase0's in the header. + state.latest_block_header_mut().body_root = + gloas::BeaconBlockBody::empty().hash_tree_root(); let dir = tempfile::tempdir().unwrap(); std::fs::write(dir.path().join("config.yaml"), config_text).unwrap(); @@ -2845,23 +2884,13 @@ validators: let source = network::NetworkSource::Loaded(Box::new(loaded)); let backend: Arc = Arc::new(InMemoryBackend::new()); - // `Store` is not `Debug`, so take the error by pattern. - let Err(err) = fetch_initial_beacon_state(&[], backend.clone(), &source).await else { - panic!("a gloas genesis must not become an anchor"); - }; - assert!( - matches!( - err, - checkpoint_sync::CheckpointSyncError::UnsupportedFork { - fork: ForkName::Gloas - } - ), - "the refusal must name the fork: {err}" - ); - assert!( - Store::from_db_state(backend).unwrap().is_none(), - "a refused anchor leaves the directory empty" - ); + let store = fetch_initial_beacon_state(&[], backend, &source) + .await + .expect("a gloas genesis anchors"); + let (head_slot, _) = store + .beacon_head() + .expect("an anchored directory has a head"); + assert_eq!(head_slot, 0, "a genesis anchor is at slot 0"); } /// A changed fork epoch leaves genesis time and the validators root @@ -3334,3 +3363,107 @@ validators: assert_eq!(fallback, mainnet_source.bootnodes()); } } + +#[cfg(test)] +mod anchor_envelope_root_tests { + use super::*; + use ethlambda_storage::backend::InMemoryBackend; + use ethlambda_types::beacon::config::Config; + use ethlambda_types::beacon::containers::gloas; + use ethlambda_types::checkpoint::Checkpoint; + + fn anchored_at(block: SignedBeaconBlock) -> (Store, H256, u64) { + let slot = block.slot(); + let root = block.message_hash_tree_root(); + let mut store = Store::init_beacon( + Arc::new(InMemoryBackend::default()), + 1_606_824_023, + Config::mainnet(), + root, + Checkpoint { root, slot }, + slot, + ); + store.insert_signed_block(root, block).expect("insert"); + (store, root, slot) + } + + fn gloas_block(slot: u64) -> SignedBeaconBlock { + SignedBeaconBlock::Gloas(gloas::SignedBeaconBlock { + message: gloas::BeaconBlock { + slot, + proposer_index: 0, + parent_root: H256::ZERO, + state_root: H256::ZERO, + body: Default::default(), + }, + signature: Default::default(), + }) + } + + #[test] + fn a_gloas_anchor_without_its_envelope_is_asked_about() { + let (store, root, _) = anchored_at(gloas_block(64)); + assert_eq!(missing_anchor_envelope_root(&store), Some(root)); + } + + #[test] + fn a_gloas_anchor_that_holds_its_envelope_is_not() { + let (mut store, root, slot) = anchored_at(gloas_block(64)); + let mut envelope = test_envelope(); + envelope.message.beacon_block_root = root; + store.insert_verified_payload(slot, &envelope); + assert_eq!(missing_anchor_envelope_root(&store), None); + } + + #[test] + fn a_pre_gloas_anchor_has_no_envelope_to_ask_for() { + let block = SignedBeaconBlock::Phase0(phase0::SignedBeaconBlock { + message: phase0::BeaconBlock { + slot: 64, + proposer_index: 0, + parent_root: H256::ZERO, + state_root: H256::ZERO, + body: phase0::BeaconBlockBody::default(), + }, + signature: Default::default(), + }); + let (store, _, _) = anchored_at(block); + assert_eq!(missing_anchor_envelope_root(&store), None); + } + + fn test_envelope() -> gloas::SignedExecutionPayloadEnvelope { + let payload = gloas::ExecutionPayload { + parent_hash: Default::default(), + fee_recipient: Default::default(), + state_root: Default::default(), + receipts_root: Default::default(), + logs_bloom: vec![0u8; ethlambda_types::beacon::preset::BYTES_PER_LOGS_BLOOM] + .try_into() + .expect("built at exactly BYTES_PER_LOGS_BLOOM"), + prev_randao: Default::default(), + block_number: 7, + gas_limit: 0, + gas_used: 0, + timestamp: 0, + extra_data: Default::default(), + base_fee_per_gas: Default::default(), + block_hash: Default::default(), + transactions: Default::default(), + withdrawals: Default::default(), + blob_gas_used: 0, + excess_blob_gas: 0, + block_access_list: Default::default(), + slot_number: 0, + }; + gloas::SignedExecutionPayloadEnvelope { + message: gloas::ExecutionPayloadEnvelope { + payload, + execution_requests: Default::default(), + builder_index: 3, + beacon_block_root: H256::ZERO, + parent_beacon_block_root: H256::ZERO, + }, + signature: Default::default(), + } + } +} diff --git a/bin/ethlambda/src/network/built_in.rs b/bin/ethlambda/src/network/built_in.rs index 8133a765d..615be8b3b 100644 --- a/bin/ethlambda/src/network/built_in.rs +++ b/bin/ethlambda/src/network/built_in.rs @@ -67,13 +67,11 @@ impl BuiltInNetwork { .wrap_err_with(|| format!("the built-in {} config.yaml did not parse", self.name()))?; super::check_preset(parsed.config.preset_base.as_str())?; super::check_constants(&parsed.config)?; - // A built-in config can carry a fork whose *keys* this build claims - // but whose state transition it does not implement yet (gloas), or - // one whose keys it does not even claim (heze). The ignored-keys - // warning below only ever said the second half of that; it used to - // say both, back when GLOAS_* were unclaimed too. `warn_if_unfollowed_fork_scheduled` - // says the first half explicitly instead of leaving it to be - // discovered as a stall. + // A built-in config can carry a fork whose keys this build does not + // even claim (heze), which the ignored-keys warning below says. A fork + // whose keys it claims but whose rules it does not follow is + // `warn_if_unfollowed_fork_scheduled`'s to say; none exists today, + // since gloas is followed. parsed.warn_about_ignored_keys(); let genesis = chain.genesis(); diff --git a/bin/ethlambda/src/network/mod.rs b/bin/ethlambda/src/network/mod.rs index 570fdb40b..90d4c8d49 100644 --- a/bin/ethlambda/src/network/mod.rs +++ b/bin/ethlambda/src/network/mod.rs @@ -197,6 +197,7 @@ pub(crate) fn check_constants(config: &Config) -> Result<(), ConstantsMismatch> maximum_gossip_clock_disparity == constants::MAXIMUM_GOSSIP_CLOCK_DISPARITY, max_request_blocks == protocols::MAX_REQUEST_BLOCKS, max_request_blocks_deneb == protocols::MAX_REQUEST_BLOCKS_DENEB, + max_request_payloads == protocols::MAX_REQUEST_PAYLOADS, max_request_data_column_sidecars == protocols::max_request_data_column_sidecars(), max_payload_size == ethlambda_p2p::MAX_PAYLOAD_SIZE as u64, message_domain_invalid_snappy == ethlambda_p2p::MESSAGE_DOMAIN_INVALID_SNAPPY, @@ -246,8 +247,9 @@ fn derive_genesis_fields(config: &mut Config, genesis_time: u64) { /// config parsed. A loaded network reaches this the same as a built-in one, /// and can even schedule such a fork at epoch 0, in which case its own genesis /// state already decodes as that fork and startup refuses it outright -/// (`refuse_unfollowable_fork` in `main.rs`, since fork choice itself accepts -/// a gloas anchor); this warning fires first either way. +/// (`refuse_unfollowable_fork` in `main.rs`); this warning fires first either +/// way. Every fork `ForkName` models is followed today, gloas included, so +/// this is the guard for the next fork the node cannot follow. fn warn_if_unfollowed_fork_scheduled(network: &str, config: &Config) { let unfollowed = ethlambda_types::beacon::fork::ForkName::ALL .into_iter() @@ -460,6 +462,15 @@ mod tests { ); } + #[test] + fn a_config_that_changes_the_envelope_request_ceiling_is_refused() { + let mut config = Config::mainnet(); + config.max_request_payloads = 64; + + let err = check_constants(&config).unwrap_err().to_string(); + assert!(err.contains("MAX_REQUEST_PAYLOADS is 64"), "got {err}"); + } + #[test] fn an_absent_preset_base_is_a_distinct_error_from_a_mismatch() { // `Config::preset_base` defaults an absent PRESET_BASE to "". diff --git a/crates/blockchain/Cargo.toml b/crates/blockchain/Cargo.toml index 99607acf8..f6aaa1136 100644 --- a/crates/blockchain/Cargo.toml +++ b/crates/blockchain/Cargo.toml @@ -38,8 +38,17 @@ hex.workspace = true hex = { workspace = true } libssz.workspace = true libssz-types.workspace = true +# The mock execution client the payload-validity tests talk to over a socket. +tokio = { workspace = true, features = ["io-util", "macros", "net", "rt"] } datatest-stable = "0.3.3" serde_json.workspace = true +snap = "1.1.1" + +[features] +# Run the fork choice fixture cases the actor tests read from +# `consensus-spec-tests`. Off by default so `cargo test` needs no fixture +# download; `make test-beacon-mainnet` turns it on. +beacon-spec-tests = [] [[test]] name = "forkchoice_spectests" diff --git a/crates/blockchain/src/beacon_columns.rs b/crates/blockchain/src/beacon_columns.rs new file mode 100644 index 000000000..88d035a2a --- /dev/null +++ b/crates/blockchain/src/beacon_columns.rs @@ -0,0 +1,935 @@ +//! The chain actor's parking lot for data column sidecars the p2p layer's +//! checks could not judge yet. +//! +//! A sidecar is judged against a block's post-state: its parent's for a fulu +//! sidecar, its own block's for a gloas one. Until that state exists the +//! sidecar's bytes wait in `Table::PendingDataColumns`, and only their keys are +//! held in `BlockChainServer::sidecars_awaiting_parent`. +//! +//! # What bounds this +//! +//! A fulu sidecar carries a signed header, which the p2p checks verify against +//! the head state before it is ever sent here. A gloas sidecar carries no +//! signature at all, so a peer can invent `(slot, root, index)` triples as fast +//! as it can send them. Gloas sidecars are therefore parked only when their shape +//! passes +//! [`is_parkable_gloas`](ethlambda_state_transition::beacon::gossip::column::is_parkable_gloas), +//! and at most [`MAX_PARKED_GLOAS_ROOTS_PER_SLOT`] distinct roots per slot. + +use std::collections::{HashMap, HashSet}; + +use ethlambda_state_transition::beacon::gossip::column::{ + is_parkable_gloas, judgeable_block_gloas, +}; +use ethlambda_types::ShortRoot; +use ethlambda_types::beacon::containers::DataColumnSidecar; +use ethlambda_types::primitives::H256; +use tracing::{debug, error, info, trace}; + +use crate::{BlockChainServer, metrics}; + +/// Where a parked data column sidecar was put. +/// +/// The three fields are exactly `Table::PendingDataColumns`'s key, so reading +/// the sidecar back needs nothing else. `slot` is also what the finality sweep +/// compares against. +#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)] +pub(crate) struct ParkedColumn { + pub(crate) slot: u64, + pub(crate) block_root: H256, + pub(crate) index: u64, +} + +/// How many distinct gloas block roots may have sidecars parked at one slot. +/// +/// A slot has one canonical block, and a short fork or an equivocating +/// proposer adds a few more, so a handful covers every honest case. A gloas +/// sidecar names its block by an unsigned root, so past this cap the extra +/// roots at a slot are ones nobody can vouch for, and the node stops keeping +/// rows for them. +pub(crate) const MAX_PARKED_GLOAS_ROOTS_PER_SLOT: usize = 4; + +/// The bookkeeping that bounds the parking lot, beside the map it bounds. +#[derive(Default)] +pub(crate) struct ColumnParking { + /// How many sidecars are parked. Kept as a running count, updated where + /// one is parked, drained or evicted, so publishing it is not a walk over + /// every parked root. + pub(crate) count: usize, + /// The gloas roots with sidecars parked, by the slot they were parked at. + pub(crate) gloas_roots: HashMap>, + /// The slot each root in [`Self::gloas_roots`] was parked at, kept as its + /// own map rather than read back off any parked sidecar: the parked set + /// for a root is keyed by the awaited root alone, so a fulu sidecar whose + /// parent is a gloas root shares it, and its slot says nothing about the + /// gloas root's. + pub(crate) gloas_root_slots: HashMap, +} + +/// The root whose post-state `sidecar` cannot be checked without. +/// +/// A fulu sidecar is verified against its parent's state (its proposer and +/// signature live there). A gloas sidecar carries neither, but reads its +/// commitments from its own block's bid, so it waits for that block instead. +pub(crate) fn awaited_root(sidecar: &DataColumnSidecar) -> H256 { + match sidecar { + DataColumnSidecar::Fulu(sidecar) => sidecar.signed_block_header.message.parent_root, + DataColumnSidecar::Gloas(sidecar) => sidecar.beacon_block_root, + } +} + +impl BlockChainServer { + /// Park sidecars the chain checks found no post-state for, or send them + /// straight back to be checked if the block they wait on has one by now. + /// + /// The second case is a race this actor has to close, because the checks + /// run elsewhere: the p2p layer looked for the post-state, found none and + /// sent these, and if the block imported in between, its + /// [`Self::drain_sidecars_awaiting_parent`] has already run and will not + /// run again, so a sidecar parked now would wait for nothing until + /// finality evicts it. Asked with the same predicate the checks use + /// ([`Self::is_ready_to_judge`]), so a sidecar sent back is one they will + /// find everything for. + pub(crate) fn park_data_columns(&mut self, sidecars: Vec) { + let config = self.store.config(); + let mut ready = Vec::new(); + for sidecar in sidecars { + if let DataColumnSidecar::Gloas(gloas) = &sidecar + && !is_parkable_gloas(&config, &self.custody_columns, gloas) + { + trace!( + slot = gloas.slot, + column = gloas.index, + "Dropping a gloas data column sidecar not shaped to be parked" + ); + continue; + } + let awaited = awaited_root(&sidecar); + if self.is_ready_to_judge(&sidecar) { + ready.push(sidecar); + continue; + } + self.queue_sidecar_awaiting_parent(awaited, sidecar); + } + self.send_data_columns_for_checks(ready); + } + + /// Whether the p2p checks can judge `sidecar` now: a fulu sidecar needs its + /// parent's post-state, a gloas one its block and that block's post-state. + /// + /// The gloas half is [`judgeable_block_gloas`], the same call the p2p + /// check makes first, so this actor and that one can never disagree about + /// a root that has a state but no block, which would bounce its sidecars + /// between them for ever. + fn is_ready_to_judge(&self, sidecar: &DataColumnSidecar) -> bool { + match sidecar { + DataColumnSidecar::Fulu(_) => { + matches!(self.store.get_state(&awaited_root(sidecar)), Ok(Some(_))) + } + DataColumnSidecar::Gloas(gloas) => { + judgeable_block_gloas(&self.store, &gloas.beacon_block_root).is_ok() + } + } + } + + /// Hand sidecars to the p2p layer's chain checks, which send back the + /// ones that pass through `new_data_column_sidecars`. + pub(crate) fn send_data_columns_for_checks(&self, sidecars: Vec) { + if sidecars.is_empty() { + return; + } + let Some(ref p2p) = self.p2p else { + return; + }; + let _ = p2p.check_data_column_sidecars(sidecars).inspect_err( + |err| error!(%err, "Failed to send data column sidecars to the p2p layer for checks"), + ); + } + + /// Park `sidecar` against the parent root it could not be checked against. + /// + /// Counted as `queued_for_parent` rather than as a rejection: nothing about + /// the sidecar has been judged yet, and conflating the two is what made the + /// deadlock invisible in the metrics (every column read as + /// `rejected{reason="unknown_parent"}` while the real fault was upstream). + /// + /// Nothing is refused here. The queue used to hold whole sidecars and so + /// carried a count cap, which a follower behind the tip hit constantly: + /// it receives gossip for the tip continuously, so the queue filled with + /// sidecars for blocks it would not reach for minutes and then refused + /// the ones for the block it was about to import. Measured on the eth-4 + /// follower a hundred slots behind, the queue sat pinned at its cap and + /// dropped 2,561 sidecars in ten minutes while the chain ground through + /// by-root lookups for slots whose columns gossip had already delivered + /// and this function had thrown away. Evicting the furthest-ahead entry + /// instead of the newest fixed which sidecar was lost, not that one was. + /// + /// The cap is gone now that the sidecars live in + /// `Table::PendingDataColumns` and only their keys are held here, so what + /// grows is disk rather than this actor's memory. + /// [`Self::evict_sidecars_awaiting_parent_at_or_below_finality`] is what + /// bounds it, which bounds how *long* an entry lives but not how fast + /// they arrive: the chain checks do not require `parent_root` to name a + /// block this node knows. Every sidecar reaching here, gossiped or + /// fetched, has had its header's signature checked against the head state + /// by those checks (`queue_unless_forged`, in + /// `ethlambda_state_transition::beacon::gossip::column`), but only when a + /// head state is already cached *and* the header's `proposer_index` names + /// a validator in it: with no cached head state, or a proposer index that + /// names none (`u64::MAX`, say), that check is skipped and a made-up + /// header still reaches here and parks a row. A peer exploiting either + /// gap can still park rows as fast as it can invent a slot, proposer and + /// index, until finality catches up. + /// + /// A gloas sidecar has no header to check at all, so it is bounded + /// instead: every sidecar parked under a root must share that root's slot, + /// and a slot keeps at most [`MAX_PARKED_GLOAS_ROOTS_PER_SLOT`] roots. + pub(crate) fn queue_sidecar_awaiting_parent( + &mut self, + awaited: H256, + sidecar: DataColumnSidecar, + ) { + let parked = ParkedColumn { + slot: sidecar.slot(), + block_root: sidecar.block_root(), + index: sidecar.index(), + }; + + // A re-delivery of something already parked. The by-root and by-range + // fetch paths skip gossip's seen cache entirely, so they never touch + // the p2p actor's `SeenColumns` (which in any case only records an + // Accept, never a park); a re-delivery reaching here is ordinary, and + // without this check the same column would take a second slot in the + // queue and leave a stale key behind after the first replay took its + // row. Asked before the write rather than left to the set below, + // because the write is what costs. + if self + .sidecars_awaiting_parent + .get(&awaited) + .is_some_and(|parked_columns| parked_columns.contains(&parked)) + { + return; + } + + if matches!(sidecar, DataColumnSidecar::Gloas(_)) + && !self.gloas_root_fits(awaited, parked.slot) + { + trace!( + slot = parked.slot, + block_root = %ShortRoot(&awaited.0), + "Dropping a gloas data column sidecar: its root does not fit the slot's bound" + ); + return; + } + + // The bytes go to disk before the key goes in the map, so a failed + // write leaves no key pointing at a row that is not there. + if let Err(err) = self.store.put_pending_data_column(&sidecar) { + error!(%err, "Failed to park a data column sidecar"); + return; + } + + trace!( + slot = parked.slot, + column = parked.index, + awaited = %ShortRoot(&awaited.0), + "Queueing a data column sidecar until the block it is judged against has a post-state" + ); + if matches!(sidecar, DataColumnSidecar::Gloas(_)) { + self.column_parking + .gloas_roots + .entry(parked.slot) + .or_default() + .insert(awaited); + self.column_parking + .gloas_root_slots + .insert(awaited, parked.slot); + } + let inserted = self + .sidecars_awaiting_parent + .entry(awaited) + .or_default() + .insert(parked); + if inserted { + self.column_parking.count += 1; + } + self.publish_sidecars_awaiting_parent(); + } + + /// Whether a gloas sidecar for `root` at `slot` may be parked: the root is + /// not already parked at another slot, and this slot has room for it. + /// + /// A root this actor already holds as a pending or held block is exempt + /// from the per-slot cap: an honest block is known, so + /// [`MAX_PARKED_GLOAS_ROOTS_PER_SLOT`] forged roots at its slot cannot shut + /// its columns out. It still has to match the slot + /// it was first parked at. + fn gloas_root_fits(&self, root: H256, slot: u64) -> bool { + if self + .column_parking + .gloas_root_slots + .get(&root) + .is_some_and(|&parked_slot| parked_slot != slot) + { + return false; + } + self.column_parking + .gloas_roots + .get(&slot) + .is_none_or(|roots| { + roots.contains(&root) + || roots.len() < MAX_PARKED_GLOAS_ROOTS_PER_SLOT + || self.holds_block(&root) + }) + } + + /// Whether this actor holds the block `root` names as pending (missing a + /// parent) or held (for columns, or for its parent's payload). + fn holds_block(&self, root: &H256) -> bool { + self.pending_block_parents.contains_key(root) + || self.blocks_awaiting_columns.contains_key(root) + || self.envelopes.is_block_held(root) + } + + /// Republish how many sidecars are parked, from the running count. + pub(crate) fn publish_sidecars_awaiting_parent(&self) { + metrics::set_sidecars_awaiting_parent(self.column_parking.count as u64); + } + + /// Send every sidecar parked against `block_root` back to the p2p layer's + /// chain checks, now that it has a post-state to be checked against (for a + /// gloas sidecar, now that it is imported itself). + /// + /// Called from the one arm that means "this root now has a post-state". + /// The ones that pass come back through `new_data_column_sidecars` as a + /// new message, so nothing here re-enters the import path. + pub(crate) fn drain_sidecars_awaiting_parent(&mut self, block_root: H256) { + let Some(parked_columns) = self.sidecars_awaiting_parent.remove(&block_root) else { + return; + }; + debug!( + awaited = %ShortRoot(&block_root.0), + count = parked_columns.len(), + "Replaying data column sidecars whose awaited block just imported" + ); + self.column_parking.count = self + .column_parking + .count + .saturating_sub(parked_columns.len()); + if let Some(slot) = self.column_parking.gloas_root_slots.remove(&block_root) + && let Some(roots) = self.column_parking.gloas_roots.get_mut(&slot) + { + roots.remove(&block_root); + if roots.is_empty() { + self.column_parking.gloas_roots.remove(&slot); + } + } + self.publish_sidecars_awaiting_parent(); + + let mut sidecars = Vec::with_capacity(parked_columns.len()); + for parked in parked_columns { + // Taken, not read: the row has served its purpose either way. A + // replay that passes is written to `DataColumns`, and one that + // fails a check has been judged, so neither leaves anything worth + // keeping here. + let sidecar = match self.store.take_pending_data_column( + parked.slot, + &parked.block_root, + parked.index, + ) { + Ok(Some(sidecar)) => sidecar, + Ok(None) => { + error!( + slot = parked.slot, + column = parked.index, + block_root = %ShortRoot(&parked.block_root.0), + "A parked data column sidecar has no row to replay from" + ); + continue; + } + Err(err) => { + error!(%err, "Failed to read back a parked data column sidecar"); + continue; + } + }; + sidecars.push(sidecar); + } + self.send_data_columns_for_checks(sidecars); + } + + /// Drop parked sidecars whose block finality has superseded. + /// + /// The counterpart of [`Self::evict_held_blocks_at_or_below_finality`] and + /// run beside it, for the same reason: a parent root that never arrives + /// would otherwise pin its children's sidecars for this node's whole + /// uptime. A sidecar at or below the finalized slot can never be needed + /// again, since the chain checks would drop it outright now. + pub(crate) fn evict_sidecars_awaiting_parent_at_or_below_finality(&mut self) { + let finalized_slot = self + .store + .latest_finalized() + .expect("finalized checkpoint exists") + .slot; + + // Swept on every call, not only when a row is dropped: a root whose + // sidecars were all replayed or refused leaves its bookkeeping behind, + // and nothing else would clear it. + self.column_parking + .gloas_roots + .retain(|slot, _| *slot > finalized_slot); + self.column_parking + .gloas_root_slots + .retain(|_, slot| *slot > finalized_slot); + + if self.sidecars_awaiting_parent.is_empty() { + return; + } + + let mut dropped: Vec = Vec::new(); + self.sidecars_awaiting_parent.retain(|_, parked_columns| { + parked_columns.retain(|parked| { + let keep = parked.slot > finalized_slot; + if !keep { + dropped.push(*parked); + } + keep + }); + !parked_columns.is_empty() + }); + + if !dropped.is_empty() { + self.column_parking.count = self.column_parking.count.saturating_sub(dropped.len()); + info!( + finalized_slot, + count = dropped.len(), + "Evicting parked data column sidecars that finality has superseded" + ); + // The rows go with the keys, so a key dropped from the map never + // leaves its bytes on disk with nothing left to read them. + let keys = dropped + .iter() + .map(|parked| (parked.slot, parked.block_root, parked.index)); + let _ = self + .store + .delete_pending_data_column_sidecars(keys) + .inspect_err(|err| error!(%err, "Failed to drop parked data column sidecars")); + self.publish_sidecars_awaiting_parent(); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::tests::{ + GENESIS_TIME, bare_block, bare_state, beacon_server, beacon_server_recording, beacon_store, + beacon_store_at_slot_10, gloas_sidecar_at, gloas_store_at_slot_10, sidecar_at, + }; + use ethlambda_types::primitives::HashTreeRoot as _; + + #[test] + fn a_data_column_sidecar_naming_a_parent_with_no_state_is_parked_not_stored() { + // `beacon_store` writes no anchor state (see its own doc comment), so + // any parent root at all is unknown here, including the anchor's own. + let (mut server, p2p) = beacon_server_recording(beacon_store_at_slot_10()); + let parent_root = H256::repeat_byte(9); + let sidecar = sidecar_at(10, parent_root); + let block_root = sidecar.signed_block_header.message.hash_tree_root(); + + server.park_data_columns(vec![DataColumnSidecar::Fulu(sidecar)]); + + // Not stored: it has not been checked, so it has not been accepted. + assert_eq!( + server + .store + .data_column_indices_for(10, &block_root) + .unwrap(), + Vec::::new() + ); + // But kept, under the parent it is waiting on. Dropping it here is + // what deadlocks a follower running the availability gate: a held + // block writes no post-state, so every sidecar of every child of it + // lands in exactly this branch. + assert_eq!( + server + .sidecars_awaiting_parent + .get(&parent_root) + .map(HashSet::len), + Some(1) + ); + assert!(p2p.checks.lock().unwrap().is_empty()); + } + + #[test] + fn a_sidecar_whose_parent_imported_meanwhile_goes_back_for_checks_not_into_the_queue() { + // The race the p2p layer's checks open: they found no parent state, + // then the parent imported and drained its (still empty) queue before + // this message arrived. Parking it now would strand it until + // finality, since that parent never drains again. + let (mut server, p2p) = beacon_server_recording(beacon_store_at_slot_10()); + let parent_root = H256::repeat_byte(9); + server + .store + .insert_state(parent_root, bare_state()) + .expect("insert"); + let sidecar = sidecar_at(10, parent_root); + + server.park_data_columns(vec![DataColumnSidecar::Fulu(sidecar.clone())]); + + assert!(server.sidecars_awaiting_parent.is_empty()); + assert_eq!( + *p2p.checks.lock().unwrap(), + vec![vec![DataColumnSidecar::Fulu(sidecar)]] + ); + } + + #[test] + fn a_parked_sidecar_goes_back_for_checks_once_its_parent_gains_a_post_state() { + // The deadlock this closes, in miniature: while the parent has no + // post-state every sidecar under it parks, and if parking were the end + // of the story the queue would only ever grow. What breaks the cycle + // is that gaining a post-state releases them, to the p2p layer's + // checks, since this actor no longer judges a sidecar itself. + let (mut server, p2p) = beacon_server_recording(beacon_store_at_slot_10()); + let parent_root = H256::repeat_byte(9); + let sidecar = sidecar_at(10, parent_root); + let block_root = sidecar.signed_block_header.message.hash_tree_root(); + + server.park_data_columns(vec![DataColumnSidecar::Fulu(sidecar.clone())]); + assert!(server.sidecars_awaiting_parent.contains_key(&parent_root)); + + server.drain_sidecars_awaiting_parent(parent_root); + + // Gone from the queue and from `PendingDataColumns`, and handed to + // the checks exactly as it was parked. + assert!(!server.sidecars_awaiting_parent.contains_key(&parent_root)); + assert!( + server + .store + .take_pending_data_column_sidecar(10, &block_root, 0) + .expect("DB read should succeed") + .is_none() + ); + assert_eq!( + *p2p.checks.lock().unwrap(), + vec![vec![DataColumnSidecar::Fulu(sidecar)]] + ); + } + + /// A gloas-active server with a recording p2p ref, sampling columns 0..8. + fn gloas_server() -> (BlockChainServer, std::sync::Arc) { + let (mut server, p2p) = beacon_server_recording(gloas_store_at_slot_10()); + server.custody_columns = (0..8).collect(); + (server, p2p) + } + + /// The count the parking lot publishes is the sum of what it holds. + fn assert_count_matches(server: &BlockChainServer) { + let held: usize = server + .sidecars_awaiting_parent + .values() + .map(HashSet::len) + .sum(); + assert_eq!(server.column_parking.count, held); + } + + #[test] + fn a_gloas_sidecar_for_an_unimported_block_parks_on_its_own_block_and_replays_when_it_imports() + { + // A gloas sidecar is judged against its own block (the commitments are + // in its bid), so it waits on that block's post-state where a fulu one + // waits on its parent's. + let (mut server, p2p) = gloas_server(); + let block_root = H256::repeat_byte(5); + let sidecar = gloas_sidecar_at(10, block_root, 3); + + server.park_data_columns(vec![sidecar.clone()]); + + assert_eq!( + server.sidecars_awaiting_parent.get(&block_root), + Some(&HashSet::from([ParkedColumn { + slot: 10, + block_root, + index: 3, + }])) + ); + assert_eq!(server.column_parking.count, 1); + // Parked and not custodied. + assert!(!server.store.has_data_column(10, &block_root, 3)); + assert!(p2p.checks.lock().unwrap().is_empty()); + + // The block imports: what was waiting on it goes back for checks, as + // it was parked. + server.drain_sidecars_awaiting_parent(block_root); + + assert!(server.sidecars_awaiting_parent.is_empty()); + assert_eq!(server.column_parking.count, 0); + assert!(server.column_parking.gloas_roots.is_empty()); + assert!( + server + .store + .take_pending_data_column(10, &block_root, 3) + .expect("DB read should succeed") + .is_none() + ); + assert_eq!(*p2p.checks.lock().unwrap(), vec![vec![sidecar]]); + } + + #[test] + fn a_gloas_sidecar_whose_block_has_a_state_goes_straight_back_for_checks() { + let (mut server, p2p) = gloas_server(); + let block_root = H256::repeat_byte(5); + server + .store + .insert_signed_block(block_root, bare_block(10, H256::ZERO)) + .expect("insert block"); + server + .store + .insert_state(block_root, bare_state()) + .expect("insert"); + let sidecar = gloas_sidecar_at(10, block_root, 3); + + server.park_data_columns(vec![sidecar.clone()]); + + assert!(server.sidecars_awaiting_parent.is_empty()); + assert_eq!(*p2p.checks.lock().unwrap(), vec![vec![sidecar]]); + } + + /// The p2p check will not judge a sidecar whose block is missing, so this + /// actor must not call it ready either: a state with no block would send it + /// back and forth between the two for ever. + #[test] + fn a_gloas_sidecar_whose_root_has_a_state_but_no_block_is_parked_not_bounced() { + let (mut server, p2p) = gloas_server(); + let block_root = H256::repeat_byte(5); + server + .store + .insert_state(block_root, bare_state()) + .expect("insert"); + + server.park_data_columns(vec![gloas_sidecar_at(10, block_root, 3)]); + + assert!(server.sidecars_awaiting_parent.contains_key(&block_root)); + assert!(p2p.checks.lock().unwrap().is_empty()); + } + + #[test] + fn a_gloas_sidecar_is_parked_and_counted_awaiting_its_parent() { + // Gloas blocks import, so a parked row has a block to drain it. + let (mut server, p2p) = gloas_server(); + let block_root = H256::repeat_byte(5); + + server.park_data_columns(vec![gloas_sidecar_at(10, block_root, 3)]); + + assert!(server.sidecars_awaiting_parent.contains_key(&block_root)); + assert_eq!(server.column_parking.count, 1); + assert!(p2p.checks.lock().unwrap().is_empty()); + } + + #[test] + fn a_fulu_sidecar_naming_a_gloas_root_as_parent_does_not_decide_that_roots_slot() { + // The parked set is keyed by the awaited root alone, so a forged fulu + // sidecar whose parent is the gloas root R shares R's set. R's slot + // must come from its own record, not from whichever element the set + // yields first. + let (mut server, _p2p) = gloas_server(); + let root = H256::repeat_byte(5); + server.park_data_columns(vec![DataColumnSidecar::Fulu(sidecar_at(7, root))]); + + server.park_data_columns(vec![ + gloas_sidecar_at(10, root, 0), + gloas_sidecar_at(10, root, 1), + ]); + + assert_eq!(server.column_parking.gloas_root_slots.get(&root), Some(&10)); + assert_eq!(server.sidecars_awaiting_parent[&root].len(), 3); + + // Draining the root frees its bookkeeping even though the first parked + // element was the fulu one. + server.drain_sidecars_awaiting_parent(root); + assert!(server.column_parking.gloas_roots.is_empty()); + assert!(server.column_parking.gloas_root_slots.is_empty()); + } + + #[test] + fn a_gloas_root_parked_at_one_slot_refuses_a_sidecar_naming_another() { + let (mut server, _p2p) = gloas_server(); + let root = H256::repeat_byte(5); + server.park_data_columns(vec![DataColumnSidecar::Fulu(sidecar_at(7, root))]); + server.park_data_columns(vec![gloas_sidecar_at(10, root, 0)]); + + server.park_data_columns(vec![gloas_sidecar_at(9, root, 1)]); + + assert!( + !server.sidecars_awaiting_parent[&root] + .iter() + .any(|parked| parked.slot == 9) + ); + assert_count_matches(&server); + } + + #[test] + fn eviction_clears_gloas_bookkeeping_even_when_no_sidecar_is_left_to_drop() { + // A root whose sidecars were all replayed leaves its entries behind; + // the sweep must not return early on an empty parking map. + let mut server = beacon_server(beacon_store(GENESIS_TIME, 10)); + let root = H256::repeat_byte(5); + server + .column_parking + .gloas_roots + .insert(10, HashSet::from([root])); + server.column_parking.gloas_root_slots.insert(root, 10); + assert!(server.sidecars_awaiting_parent.is_empty()); + + server.evict_sidecars_awaiting_parent_at_or_below_finality(); + + assert!(server.column_parking.gloas_roots.is_empty()); + assert!(server.column_parking.gloas_root_slots.is_empty()); + } + + #[test] + fn a_root_the_actor_already_holds_as_a_block_is_exempt_from_the_slot_cap() { + let (mut server, _p2p) = gloas_server(); + for byte in 1..=MAX_PARKED_GLOAS_ROOTS_PER_SLOT as u8 { + server.park_data_columns(vec![gloas_sidecar_at(10, H256::repeat_byte(byte), 0)]); + } + let honest = H256::repeat_byte(0xaa); + let forged = H256::repeat_byte(0xbb); + // The honest block is pending on a parent this node does not have. + server.pending_block_parents.insert(honest, H256::ZERO); + + server.park_data_columns(vec![ + gloas_sidecar_at(10, forged, 0), + gloas_sidecar_at(10, honest, 0), + ]); + + assert!(!server.sidecars_awaiting_parent.contains_key(&forged)); + assert!(server.sidecars_awaiting_parent.contains_key(&honest)); + assert_count_matches(&server); + } + + #[test] + fn a_misshapen_or_uncustodied_gloas_sidecar_is_dropped_before_anything_is_written() { + let (mut server, _p2p) = gloas_server(); + let block_root = H256::repeat_byte(5); + // Not a column this node samples. + let uncustodied = gloas_sidecar_at(10, block_root, 9); + // No cells at all. + let mut empty = gloas_sidecar_at(10, block_root, 3); + if let DataColumnSidecar::Gloas(sidecar) = &mut empty { + sidecar.column = Default::default(); + sidecar.kzg_proofs = Default::default(); + } + + server.park_data_columns(vec![uncustodied, empty]); + + assert!(server.sidecars_awaiting_parent.is_empty()); + assert_eq!(server.column_parking.count, 0); + for index in [3, 9] { + assert!( + server + .store + .take_pending_data_column(10, &block_root, index) + .expect("DB read should succeed") + .is_none() + ); + } + } + + #[test] + fn a_slot_parks_no_more_gloas_roots_than_the_cap() { + let (mut server, _p2p) = gloas_server(); + let roots: Vec = (1..=MAX_PARKED_GLOAS_ROOTS_PER_SLOT as u8 + 1) + .map(H256::repeat_byte) + .collect(); + for &root in &roots { + server.park_data_columns(vec![gloas_sidecar_at(10, root, 0)]); + } + + let (admitted, refused) = roots.split_at(MAX_PARKED_GLOAS_ROOTS_PER_SLOT); + for root in admitted { + assert!(server.sidecars_awaiting_parent.contains_key(root)); + } + for root in refused { + assert!(!server.sidecars_awaiting_parent.contains_key(root)); + assert!( + server + .store + .take_pending_data_column(10, root, 0) + .expect("DB read should succeed") + .is_none(), + "a refused root leaves no row on disk" + ); + } + assert_eq!(server.column_parking.count, MAX_PARKED_GLOAS_ROOTS_PER_SLOT); + + // A root already parked keeps taking its other columns, and another + // slot has a cap of its own. + server.park_data_columns(vec![ + gloas_sidecar_at(10, admitted[0], 1), + gloas_sidecar_at(9, refused[0], 0), + ]); + assert_eq!(server.sidecars_awaiting_parent[&admitted[0]].len(), 2); + assert!(server.sidecars_awaiting_parent.contains_key(&refused[0])); + assert_count_matches(&server); + } + + #[test] + fn every_sidecar_parked_for_a_gloas_root_shares_its_slot() { + let (mut server, _p2p) = gloas_server(); + let root = H256::repeat_byte(5); + server.park_data_columns(vec![ + gloas_sidecar_at(10, root, 0), + gloas_sidecar_at(9, root, 1), + ]); + + assert_eq!( + server.sidecars_awaiting_parent.get(&root), + Some(&HashSet::from([ParkedColumn { + slot: 10, + block_root: root, + index: 0, + }])) + ); + assert!( + server + .store + .take_pending_data_column(9, &root, 1) + .expect("DB read should succeed") + .is_none() + ); + assert_count_matches(&server); + } + + #[test] + fn eviction_frees_the_gloas_roots_and_the_count_of_a_finalized_slot() { + // Populated directly, as `parked_sidecars_are_dropped_once_finality_ + // passes_their_slot` is: the finalized slot is fixed at init. + let mut server = beacon_server(beacon_store(GENESIS_TIME, 10)); + let root = H256::repeat_byte(5); + server.sidecars_awaiting_parent.insert( + root, + HashSet::from([ParkedColumn { + slot: 10, + block_root: root, + index: 0, + }]), + ); + server.column_parking.count = 1; + server + .column_parking + .gloas_roots + .insert(10, HashSet::from([root])); + + server.evict_sidecars_awaiting_parent_at_or_below_finality(); + + assert!(server.sidecars_awaiting_parent.is_empty()); + assert_eq!(server.column_parking.count, 0); + assert!(server.column_parking.gloas_roots.is_empty()); + } + + #[test] + fn a_parked_sidecar_holds_its_bytes_on_disk_and_not_in_the_queue() { + // The queue's size is chosen by whoever is gossiping, so what it holds + // per entry is the thing that has to stay small: a key, not a cell per + // blob. + let mut server = beacon_server(beacon_store_at_slot_10()); + let parent_root = H256::repeat_byte(9); + let sidecar = sidecar_at(10, parent_root); + let block_root = sidecar.signed_block_header.message.hash_tree_root(); + + server.park_data_columns(vec![DataColumnSidecar::Fulu(sidecar)]); + + assert_eq!( + server.sidecars_awaiting_parent.get(&parent_root), + Some(&HashSet::from([ParkedColumn { + slot: 10, + block_root, + index: 0, + }])) + ); + assert!( + server + .store + .take_pending_data_column_sidecar(10, &block_root, 0) + .expect("DB read should succeed") + .is_some(), + "the sidecar's bytes belong in PendingDataColumns" + ); + } + + #[test] + fn a_parked_sidecar_does_not_satisfy_the_availability_gate() { + // Why the parked rows get a table of their own. Nothing has judged a + // parked sidecar's inclusion proof, its KZG batch or its proposer + // signature, so a peer that could get one counted as custodied would + // be able to release a held block with a column it invented. + let mut server = beacon_server(beacon_store_at_slot_10()); + let sidecar = sidecar_at(10, H256::repeat_byte(9)); + let block_root = sidecar.signed_block_header.message.hash_tree_root(); + + server.park_data_columns(vec![DataColumnSidecar::Fulu(sidecar)]); + + assert_eq!( + server + .store + .data_column_indices_for(10, &block_root) + .expect("DB read should succeed"), + Vec::::new(), + "an unverified sidecar must be invisible to data_column_indices_for" + ); + } + + #[test] + fn a_sidecar_parked_twice_takes_one_slot_in_the_queue() { + // The by-root and by-range fetch paths skip gossip's seen cache + // entirely, so nothing between them and this actor dedups a + // re-delivery while the parent is still stateless; it is ordinary. A + // second entry would leave a key with no row behind it once the + // first replay took it. + let mut server = beacon_server(beacon_store_at_slot_10()); + let parent_root = H256::repeat_byte(9); + + server.park_data_columns(vec![DataColumnSidecar::Fulu(sidecar_at(10, parent_root))]); + server.park_data_columns(vec![DataColumnSidecar::Fulu(sidecar_at(10, parent_root))]); + + assert_eq!( + server + .sidecars_awaiting_parent + .get(&parent_root) + .map(HashSet::len), + Some(1) + ); + } + + #[test] + fn parked_sidecars_are_dropped_once_finality_passes_their_slot() { + // Populated directly rather than through `park_data_columns`: the + // chain checks refuse a sidecar at or below the finalized slot before + // it could ever be parked, so the only way to observe the sweep is to + // park one behind their back. The finalized slot is fixed at init, so + // the store carries it rather than the test moving it. + let mut server = beacon_server(beacon_store(GENESIS_TIME, 10)); + let superseded = H256::repeat_byte(1); + let still_wanted = H256::repeat_byte(2); + let parked_at = |slot: u64| ParkedColumn { + slot, + block_root: H256::repeat_byte(9), + index: 0, + }; + server + .sidecars_awaiting_parent + .insert(superseded, HashSet::from([parked_at(10)])); + server + .sidecars_awaiting_parent + .insert(still_wanted, HashSet::from([parked_at(20)])); + + server.evict_sidecars_awaiting_parent_at_or_below_finality(); + + // A parent root that never arrives would otherwise pin its children's + // sidecars for this node's whole uptime; one still above finality is + // a parent that may yet show up. + assert!(!server.sidecars_awaiting_parent.contains_key(&superseded)); + assert!(server.sidecars_awaiting_parent.contains_key(&still_wanted)); + } +} diff --git a/crates/blockchain/src/beacon_engine.rs b/crates/blockchain/src/beacon_engine.rs index 510edbb9a..38f34c39e 100644 --- a/crates/blockchain/src/beacon_engine.rs +++ b/crates/blockchain/src/beacon_engine.rs @@ -7,7 +7,7 @@ //! pull in `reqwest`. This crate already depends on both, so the assembly that //! needs a helper from each lives here. //! -//! # Everything a request needs comes from the block alone +//! # Everything a pre-gloas request needs comes from the block alone //! //! `NewPayloadRequest` in the specification reads `parent_beacon_block_root` //! off `state.latest_block_header.parent_root`, which after @@ -15,14 +15,19 @@ //! fields are body fields. So the whole question is answerable before the state //! transition runs, which is what lets the engine round trip happen outside the //! state transition and the state transition stay synchronous. +//! +//! Gloas is the exception: its payload is revealed in an envelope, so its +//! request ([`gloas_new_payload_request`]) needs the block and the envelope. use ethlambda_engine::types::PayloadStatusValue; use ethlambda_engine::{EngineClient, EngineError, PayloadStatusV1}; use ethlambda_state_transition::beacon::containers::{self, SignedBeaconBlock}; use ethlambda_state_transition::beacon::fork_choice::PayloadValidity; +use ethlambda_state_transition::beacon::primitives::HashTreeRoot as _; use ethlambda_state_transition::beacon::primitives::{Bytes32, Root}; use ethlambda_state_transition::beacon::stf::deneb::kzg_commitment_to_versioned_hash; use ethlambda_state_transition::beacon::stf::electra::get_execution_requests_list; +use ethlambda_state_transition::beacon::stf::gloas; /// Everything `engine_newPayloadV4` takes, derived from one block. pub struct NewPayloadRequest<'a> { @@ -32,6 +37,76 @@ pub struct NewPayloadRequest<'a> { pub execution_requests: Vec>, } +/// Everything `engine_newPayloadV5` takes, derived from a gloas block and the +/// envelope that reveals its payload. +pub struct GloasNewPayloadRequest<'a> { + pub execution_payload: &'a containers::gloas::ExecutionPayload, + pub versioned_hashes: Vec, + pub parent_beacon_block_root: Root, + pub execution_requests: Vec>, +} + +/// The question to ask about `block`'s revealed payload. +/// +/// `Err(EnvelopeMismatch)` when `block` is not a gloas block, when the +/// envelope's `beacon_block_root` is not `block`'s root, or when the bid's +/// `block_hash` is not the payload's. Unlike [`new_payload_request`]'s `None`, +/// which means "nothing to ask, import it", none of these has an importable +/// reading, so a caller cannot mistake one for a pass. The checks are cheap +/// and stop an envelope being validated against a block it does not belong to. +/// +/// `NewPayloadRequest` as `verify_execution_payload_envelope` builds it +/// (gloas `beacon-chain.md`): the payload and the requests come from the +/// envelope, the parent root is the envelope's own `parent_beacon_block_root`, +/// and the versioned hashes come from the BLOCK, off its bid's +/// `blob_kzg_commitments`, since the envelope does not carry commitments. +/// Requests go through gloas's own list builder, which also encodes the +/// builder deposit and exit request types. +/// +/// The fork is chosen by an explicit match: a fork before gloas carries its +/// payload in the block and is asked about with [`new_payload_request`] and +/// `engine_newPayloadV4`, and no fork after gloas exists yet to need a V6. +pub fn gloas_new_payload_request<'a>( + block: &SignedBeaconBlock, + envelope: &'a containers::gloas::SignedExecutionPayloadEnvelope, +) -> Result, EngineError> { + let mismatch = |reason: &str| Err(EngineError::EnvelopeMismatch(reason.to_string())); + let inner = match block { + SignedBeaconBlock::Gloas(inner) => inner, + SignedBeaconBlock::Phase0(_) + | SignedBeaconBlock::Altair(_) + | SignedBeaconBlock::Bellatrix(_) + | SignedBeaconBlock::Capella(_) + | SignedBeaconBlock::Deneb(_) + | SignedBeaconBlock::Electra(_) + | SignedBeaconBlock::Fulu(_) + | SignedBeaconBlock::Lean(_) => return mismatch("the block is not a gloas block"), + }; + + if envelope.message.beacon_block_root != inner.message.hash_tree_root() { + return mismatch("the envelope's beacon_block_root is not the block's root"); + } + let bid = &inner.message.body.signed_execution_payload_bid.message; + if bid.block_hash != envelope.message.payload.block_hash { + return mismatch("the bid's block_hash is not the payload's block_hash"); + } + + let versioned_hashes = bid + .blob_kzg_commitments + .iter() + .map(kzg_commitment_to_versioned_hash) + .collect(); + + Ok(GloasNewPayloadRequest { + execution_payload: &envelope.message.payload, + versioned_hashes, + parent_beacon_block_root: envelope.message.parent_beacon_block_root, + execution_requests: gloas::get_execution_requests_list( + &envelope.message.execution_requests, + ), + }) +} + /// The question to ask about `block`, or `None` if there is nothing to ask. /// /// `None` for phase0 and altair, which predate the merge and carry no payload, @@ -47,8 +122,7 @@ pub struct NewPayloadRequest<'a> { /// (see `containers::gloas::SignedExecutionPayloadEnvelope`), so a gloas /// block carries nothing this function could ask about at all. The engine /// question gloas does ask, whether a payload is valid, belongs to that -/// envelope rather than to the block, and this follower does not receive -/// envelopes yet, so nothing here asks it. +/// envelope rather than to the block: see [`gloas_new_payload_request`]. pub fn new_payload_request(block: &SignedBeaconBlock) -> Option> { let inner = match block { SignedBeaconBlock::Electra(inner) | SignedBeaconBlock::Fulu(inner) => inner, @@ -129,6 +203,43 @@ pub async fn ask( Ok(Some(verdict(&status))) } +/// Asks the execution client about the payload `envelope` reveals for `block`, +/// with `engine_newPayloadV5`. +/// +/// Never `Ok(None)`: unlike [`ask`] there is no "nothing to ask" reading, and a +/// block/envelope pair that does not match is an +/// [`EngineError::EnvelopeMismatch`], which the caller must treat as a refusal +/// to import. +pub async fn ask_envelope( + client: &EngineClient, + block: &SignedBeaconBlock, + envelope: &containers::gloas::SignedExecutionPayloadEnvelope, +) -> Result { + ask_envelope_status(client, block, envelope) + .await + .map(|status| verdict(&status)) +} + +/// [`ask_envelope`] with the engine's raw answer, for a caller that must tell +/// `INVALID_BLOCK_HASH` (the envelope's contents do not hash to its claimed +/// block hash, so only that envelope is bad) from `INVALID` (the payload the +/// hash names is bad). +pub async fn ask_envelope_status( + client: &EngineClient, + block: &SignedBeaconBlock, + envelope: &containers::gloas::SignedExecutionPayloadEnvelope, +) -> Result { + let request = gloas_new_payload_request(block, envelope)?; + client + .new_payload_v5( + request.execution_payload, + &request.versioned_hashes, + request.parent_beacon_block_root, + &request.execution_requests, + ) + .await +} + #[cfg(test)] mod tests { use ethlambda_state_transition::beacon::preset; @@ -254,6 +365,124 @@ mod tests { ); } + /// A gloas envelope whose payload is otherwise zero. + fn gloas_envelope( + beacon_block_root: Root, + parent_beacon_block_root: Root, + ) -> containers::gloas::SignedExecutionPayloadEnvelope { + let payload = containers::gloas::ExecutionPayload { + parent_hash: ExecutionBlockHash::ZERO, + fee_recipient: ExecutionAddress::ZERO, + state_root: Bytes32::ZERO, + receipts_root: Bytes32::ZERO, + logs_bloom: containers::bellatrix::LogsBloom::try_from(vec![ + 0u8; + preset::BYTES_PER_LOGS_BLOOM + ]) + .expect("built at exactly BYTES_PER_LOGS_BLOOM"), + prev_randao: Bytes32::ZERO, + block_number: 0, + gas_limit: 0, + gas_used: 0, + timestamp: 0, + extra_data: Default::default(), + base_fee_per_gas: Uint256::ZERO, + block_hash: ExecutionBlockHash::ZERO, + transactions: Default::default(), + withdrawals: Default::default(), + blob_gas_used: 0, + excess_blob_gas: 0, + block_access_list: Default::default(), + slot_number: 5, + }; + containers::gloas::SignedExecutionPayloadEnvelope { + message: containers::gloas::ExecutionPayloadEnvelope { + payload, + execution_requests: Default::default(), + builder_index: 0, + beacon_block_root, + parent_beacon_block_root, + }, + signature: BlsSignature::default(), + } + } + + #[test] + fn a_gloas_request_reads_each_input_from_the_place_the_spec_names() { + let mut block = containers::gloas::SignedBeaconBlock::default(); + // The block's own parent root must NOT be what the request carries. + block.message.parent_root = Root::repeat_byte(4); + block + .message + .body + .signed_execution_payload_bid + .message + .blob_kzg_commitments + .push(KzgCommitment::default()); + let block_root = block.message.hash_tree_root(); + let block = SignedBeaconBlock::Gloas(block); + + let mut envelope = gloas_envelope(block_root, Root::repeat_byte(9)); + envelope + .message + .execution_requests + .builder_exits + .push(Default::default()); + + let request = gloas_new_payload_request(&block, &envelope) + .unwrap_or_else(|err| panic!("a matching pair has a request: {err}")); + + assert_eq!(request.parent_beacon_block_root, Root::repeat_byte(9)); + assert_eq!(request.execution_payload.slot_number, 5); + assert_eq!(request.versioned_hashes.len(), 1); + assert_eq!( + request.versioned_hashes[0].0[0], + ethlambda_state_transition::beacon::constants::VERSIONED_HASH_VERSION_KZG + ); + // The builder exit type is only in gloas's list builder. + assert_eq!(request.execution_requests.len(), 1); + assert_eq!( + request.execution_requests[0][0], + ethlambda_state_transition::beacon::constants::BUILDER_EXIT_REQUEST_TYPE + ); + } + + #[test] + fn a_pre_gloas_block_has_no_envelope_question() { + let block = SignedBeaconBlock::Fulu(electra_block(Root::ZERO)); + let envelope = gloas_envelope(Root::ZERO, Root::ZERO); + + assert!(matches!( + gloas_new_payload_request(&block, &envelope), + Err(EngineError::EnvelopeMismatch(_)) + )); + } + + #[test] + fn an_envelope_for_another_block_is_refused() { + let block = SignedBeaconBlock::Gloas(containers::gloas::SignedBeaconBlock::default()); + let envelope = gloas_envelope(Root::repeat_byte(0xee), Root::ZERO); + + assert!(matches!( + gloas_new_payload_request(&block, &envelope), + Err(EngineError::EnvelopeMismatch(_)) + )); + } + + #[test] + fn a_payload_other_than_the_bids_is_refused() { + let inner = containers::gloas::SignedBeaconBlock::default(); + let block_root = inner.message.hash_tree_root(); + let block = SignedBeaconBlock::Gloas(inner); + let mut envelope = gloas_envelope(block_root, Root::ZERO); + envelope.message.payload.block_hash = ExecutionBlockHash::repeat_byte(0xab); + + assert!(matches!( + gloas_new_payload_request(&block, &envelope), + Err(EngineError::EnvelopeMismatch(_)) + )); + } + #[test] fn an_execution_status_becomes_the_matching_consensus_verdict() { let syncing = PayloadStatusV1 { diff --git a/crates/blockchain/src/beacon_envelope.rs b/crates/blockchain/src/beacon_envelope.rs new file mode 100644 index 000000000..ca2046014 --- /dev/null +++ b/crates/blockchain/src/beacon_envelope.rs @@ -0,0 +1,3090 @@ +//! The chain actor's import path for gloas execution payload envelopes. +//! +//! A gloas block carries only the builder's bid; the payload arrives apart from +//! it, in a `SignedExecutionPayloadEnvelope`, and fork choice weighs a FULL and +//! an EMPTY node per block. Three things can make the order of arrival wrong, +//! and each has a bounded queue here: +//! +//! ```text +//! envelope ─► block imported (has post-state)? ─no─► block stored? ─yes (bid-checked)─► awaiting_import[root] +//! │yes │no +//! │ └─► awaiting_block[root] +//! ▼ +//! bid has commitments and a sampled column missing? ─yes─► awaiting_columns[root] +//! │no +//! ▼ +//! advance clock; on_execution_payload_envelope ─► record payload status +//! ─► redrive blocks_awaiting_parent_payload[root] ─► recompute head +//! +//! gloas block ─► parent FULL and its payload not verified ─yes─► blocks_awaiting_parent_payload[parent] +//! ``` +//! +//! Data availability moves here from the block: a gloas block's commitments are +//! in its bid, and the columns they commit to gate the envelope that reveals the +//! payload, not the block's own import. +//! +//! # Driving the queues +//! +//! Nothing here awaits inside the import path. A block import, a stored column +//! or a tick only pushes the root whose envelope may now be ready onto +//! [`EnvelopeQueues::work`]; the handler that ran it then calls +//! [`BlockChainServer::settle_envelopes`], which drains that list. That keeps +//! the cycle envelope, block import, envelope from being a recursion between +//! async functions. +//! +//! # What bounds this +//! +//! An envelope for a block this node has never seen cannot be judged, so +//! nobody has vouched for it: at most [`MAX_ENVELOPES_AWAITING_BLOCK_PER_SLOT`] +//! are kept for the slot they arrived in, and a few per root. An envelope for a +//! block that is stored but not imported (pending on its parent, or held for +//! columns or a parent's payload) is checked against that block's bid instead, +//! and held in `awaiting_import` outside that cap, bounded by the stored blocks +//! it names. Range catch-up depends on it: a batch's envelopes arrive when all +//! but its first block are in that state. Blocks held for a parent's payload +//! are capped per parent by [`MAX_BLOCKS_HELD_PER_PARENT_PAYLOAD`]. Every queue +//! is swept at finality by +//! [`BlockChainServer::evict_envelope_queues_at_or_below_finality`], and +//! `awaiting_import` also when its block is discarded. + +use std::collections::{HashMap, VecDeque}; + +use ethlambda_engine::EngineError; +use ethlambda_engine::types::PayloadStatusValue; +use ethlambda_network_api::FetchRequest; +use ethlambda_state_transition::beacon::fork_choice::{ + self, PayloadStatus, PayloadStatusEnum, PayloadValidity, +}; +use ethlambda_state_transition::beacon::stf::ExecutionEngine; +use ethlambda_state_transition::beacon::stf::gloas as stf_gloas; +use ethlambda_types::ShortRoot; +use ethlambda_types::beacon::containers::{DataColumnSidecar, SignedBeaconBlock, gloas}; +use ethlambda_types::primitives::{H256, HashTreeRoot as _}; +use tracing::{debug, error, info, trace, warn}; + +use crate::beacon_engine; +use crate::import_timing::ImportTimings; +use crate::{BlockChainServer, custody_columns_present, metrics}; + +/// How many envelopes for blocks this node has not imported may be held for one +/// slot. +/// +/// A slot has one canonical block, and a short fork or an equivocating +/// proposer adds a few more. The envelope names its block by an unverified +/// root, so past this cap the extra roots at a slot are ones nobody can vouch +/// for. +pub(crate) const MAX_ENVELOPES_AWAITING_BLOCK_PER_SLOT: usize = 4; + +/// How many distinct envelopes may be held for one block root this node has not +/// imported. +/// +/// Nobody has checked these against a bid yet, so a forged envelope for a real +/// root must not be able to evict the honest one: the root keeps a few, and +/// when the block imports each is tried until one verifies. +pub(crate) const MAX_ENVELOPES_PER_UNMATCHED_ROOT: usize = 3; + +/// How many envelopes one by-range envelope answer can carry, mirroring p2p's +/// `MAX_REQUEST_PAYLOADS` (this crate does not depend on p2p). +const SYNC_BATCH_ENVELOPES: usize = 128; + +/// How many sync batches of envelopes `awaiting_import` may hold. +const SYNC_BATCHES_AWAITING_IMPORT: usize = 4; + +/// The most envelopes held for stored, unimported blocks, across all roots. +/// +/// The per-root cap bounds one block, and finality bounds the stored blocks, +/// but p2p's fetch lead over import is unbounded across sessions: with the +/// actor stalled (a missing envelope, the engine down) range sync keeps +/// delivering while nothing finalizes to sweep the queue. A few batches is the +/// working set catch-up needs, since the envelopes of one batch arrive before +/// the next is released. Past it the envelopes nearest the import frontier +/// (lowest slots) are kept and the highest dropped: those are the furthest +/// from import, and are asked for again by root once their child is held. +pub(crate) const MAX_ENVELOPES_AWAITING_IMPORT: usize = + SYNC_BATCH_ENVELOPES * SYNC_BATCHES_AWAITING_IMPORT; + +/// How many slots an envelope for a block this node has not imported is kept. +/// +/// A builder reveals its payload after the block it bid in, so the block +/// should reach this node within the slot or the next; an envelope still +/// unmatched after that is for a block that is not coming, or one the fetch +/// path (`FetchRequest::needs_envelope`) recovers later. Bounds the queue by +/// time where the per-slot and per-root caps bound it by count, so the total +/// stays finite while the chain does not finalize. +pub(crate) const ENVELOPE_AWAITING_BLOCK_TTL_SLOTS: u64 = 2; + +/// How many blocks may wait on one parent's payload. +/// +/// The children of a block that all name it as a FULL parent: one per slot the +/// chain skipped past it, plus equivocations. +pub(crate) const MAX_BLOCKS_HELD_PER_PARENT_PAYLOAD: usize = 4; + +/// How many envelope roots the execution client refused for their own +/// contents are remembered. +/// +/// Enough to cover the copies one slot can bring; an older one asked about +/// again costs one engine call and is refused again. +pub(crate) const MAX_REJECTED_ENVELOPES: usize = 64; + +/// An envelope the actor is holding until it can be verified. +pub(crate) struct HeldEnvelope { + envelope: gloas::SignedExecutionPayloadEnvelope, + /// The envelope's own root, which tells a re-delivery from a different + /// envelope for the same block. + envelope_root: H256, + /// The slot the hold is evicted by: the slot the envelope arrived in while + /// its block is unknown (the envelope names no slot), the block's own once + /// it is. + slot: u64, + /// Unix milliseconds the envelope reached this node, which the store clock + /// is advanced to before verifying so the timeliness deadlines are judged + /// at arrival rather than at redrive. + event_ms: u64, +} + +impl HeldEnvelope { + pub(crate) fn new( + envelope: gloas::SignedExecutionPayloadEnvelope, + slot: u64, + event_ms: u64, + ) -> Self { + Self { + envelope_root: envelope.hash_tree_root(), + envelope, + slot, + event_ms, + } + } +} + +/// The three queues, and the list of roots whose envelope may have become +/// ready. +#[derive(Default)] +pub(crate) struct EnvelopeQueues { + /// Envelopes whose block has no post-state yet, by the block's root, up to + /// [`MAX_ENVELOPES_PER_UNMATCHED_ROOT`] distinct ones each. + pub(crate) awaiting_block: HashMap>, + /// Envelopes for blocks that are stored but not imported (pending on a + /// missing parent, held for columns or for their parent's payload), by the + /// block's root. Each was checked against the stored block's bid, which + /// commits to the whole payload, so they sit outside the per-arrival-slot + /// cap and are bounded by the stored blocks instead. Up to + /// [`MAX_ENVELOPES_PER_UNMATCHED_ROOT`] each, since the signature is not + /// checked until the block imports and a forged copy must not shut the + /// honest one out. + pub(crate) awaiting_import: HashMap>, + /// Envelopes whose bid has commitments and whose sampled columns are not + /// all stored, by the block's root. + pub(crate) awaiting_columns: HashMap, + /// Consensus-valid envelopes the execution client gave no answer for, by + /// the block's root, asked again once a slot. + pub(crate) awaiting_engine: HashMap, + /// Roots of envelopes the execution client refused for their own contents + /// (`INVALID_BLOCK_HASH`), newest last and bounded by + /// [`MAX_REJECTED_ENVELOPES`], so a copy that arrives again is not asked + /// about again. + rejected: VecDeque, + /// Set when an engine verdict changed what fork choice sees without a + /// payload being verified, so the head is recomputed. + pub(crate) head_dirty: bool, + /// Blocks whose parent is FULL with a payload not yet verified, by the + /// parent's root, each with its own slot. The block's bytes are in + /// `Table::BlockHeaders`/`BlockBodies` as for any other held block. + pub(crate) blocks_awaiting_parent_payload: HashMap>, + /// Roots to look at in [`BlockChainServer::settle_envelopes`]. + pub(crate) work: VecDeque, + /// Roots whose envelope was verified since the last settle, whose waiting + /// children [`BlockChainServer::settle_envelopes`] then releases. + pub(crate) verified: VecDeque, +} + +impl EnvelopeQueues { + /// Whether `block_root` is a block held for its parent's payload. + pub(crate) fn is_block_held(&self, block_root: &H256) -> bool { + self.blocks_awaiting_parent_payload + .values() + .any(|children| children.contains_key(block_root)) + } + + /// Whether the execution client already refused the envelope with this + /// root. + fn is_rejected(&self, envelope_root: &H256) -> bool { + self.rejected.contains(envelope_root) + } + + fn reject(&mut self, envelope_root: H256) { + if self.rejected.len() >= MAX_REJECTED_ENVELOPES { + self.rejected.pop_front(); + } + self.rejected.push_back(envelope_root); + } + + /// Drop envelopes from `awaiting_import`, highest slot first, until it + /// holds at most [`MAX_ENVELOPES_AWAITING_IMPORT`]. + fn trim_awaiting_import(&mut self) { + let mut total: usize = self.awaiting_import.values().map(Vec::len).sum(); + while total > MAX_ENVELOPES_AWAITING_IMPORT { + let Some(furthest) = self + .awaiting_import + .iter() + .filter_map(|(root, held)| held.iter().map(|h| h.slot).max().map(|s| (s, *root))) + .max() + .map(|(_, root)| root) + else { + return; + }; + if let Some(held) = self.awaiting_import.get_mut(&furthest) { + held.pop(); + total -= 1; + if held.is_empty() { + self.awaiting_import.remove(&furthest); + } + } + } + } + + fn is_empty(&self) -> bool { + self.awaiting_block.is_empty() + && self.awaiting_import.is_empty() + && self.awaiting_columns.is_empty() + && self.awaiting_engine.is_empty() + && self.blocks_awaiting_parent_payload.is_empty() + } + + fn held_blocks(&self) -> usize { + self.blocks_awaiting_parent_payload + .values() + .map(HashMap::len) + .sum() + } +} + +/// The sampled columns an envelope still needs, or `None` if nothing is +/// outstanding. +/// +/// A bid without commitments has no columns to wait for. With commitments, +/// every column this node samples must be stored: the spec's +/// `is_data_available` is vacuous over an empty list, so a partial set must +/// never reach fork choice as evidence. +pub(crate) fn missing_columns_for_envelope( + commitment_count: usize, + custody_columns: &[u64], + present: &[u64], +) -> Option> { + if commitment_count == 0 { + return None; + } + let missing: Vec = custody_columns + .iter() + .copied() + .filter(|index| !present.contains(index)) + .collect(); + (!missing.is_empty()).then_some(missing) +} + +/// The checks of `verify_execution_payload_envelope` that need only the block's +/// bid, not its post-state: the envelope is the one the bid promised. +fn envelope_matches_bid( + envelope: &gloas::ExecutionPayloadEnvelope, + bid: &gloas::ExecutionPayloadBid, + block_slot: u64, +) -> bool { + envelope.builder_index == bid.builder_index + && envelope.payload.block_hash == bid.block_hash + && envelope.payload.prev_randao == bid.prev_randao + && envelope.payload.gas_limit == bid.gas_limit + && envelope.execution_requests.hash_tree_root() == bid.execution_requests_root + && envelope.payload.slot_number == block_slot +} + +impl BlockChainServer { + /// Take an envelope: verify it now, or hold it until what it waits on + /// arrives. Callers may hand over envelopes nothing has checked (a + /// checkpoint source's, for one); the actor verifies them itself. The + /// caller then settles the queues. + pub(crate) async fn receive_envelope( + &mut self, + envelope: gloas::SignedExecutionPayloadEnvelope, + event_ms: u64, + ) { + let config = self.store.config(); + let arrival_slot = fork_choice::get_current_slot(&self.store, &config); + self.apply_or_hold_envelope(HeldEnvelope::new(envelope, arrival_slot, event_ms)) + .await; + } + + /// Verify `held` against its block, or put it back in the queue for what it + /// is waiting on. `true` when the envelope was verified and recorded. + async fn apply_or_hold_envelope(&mut self, held: HeldEnvelope) -> bool { + let root = held.envelope.message.beacon_block_root; + if self + .store + .beacon_block_payload_status(root) + .is_invalidated() + { + trace!( + block_root = %ShortRoot(&root.0), + "Dropping an envelope whose payload the execution client found invalid" + ); + return false; + } + if self.store.has_verified_payload(&root) { + trace!( + block_root = %ShortRoot(&root.0), + "Dropping an envelope whose payload is already verified" + ); + return false; + } + // Before any work: a copy the execution client already refused costs + // nothing. + if self.envelopes.is_rejected(&held.envelope_root) { + trace!( + block_root = %ShortRoot(&root.0), + "Dropping an envelope the execution client already rejected" + ); + return false; + } + if self + .envelopes + .awaiting_engine + .get(&root) + .is_some_and(|other| other.envelope_root == held.envelope_root) + { + return false; + } + + let has_state = self.store.has_state(&root).expect("DB read should succeed"); + let block = if has_state { + self.store.get_signed_block(&root).ok().flatten() + } else { + None + }; + let Some(block) = block else { + if let Some(held) = self.hold_envelope_for_stored_block(root, held) { + self.hold_envelope_awaiting_block(root, held); + } + return false; + }; + + let slot = block.slot(); + let finalized = self + .store + .latest_finalized() + .expect("finalized checkpoint exists"); + // Strictly below, and the finalized block exempted by name: its + // children may still name it as their FULL parent, and the checkpoint + // slot is an epoch boundary the finalized block can sit before. + if slot < finalized.slot && root != finalized.root { + trace!(%slot, "Dropping an envelope for a block below the finalized slot"); + return false; + } + + let Some(bid) = crate::beacon_payloads::bid_of(&block) else { + warn!( + %slot, + block_root = %ShortRoot(&root.0), + "Dropping an envelope: its block is not a gloas block" + ); + return false; + }; + let commitment_count = bid.blob_kzg_commitments.len(); + + // The cheap half of verification, before anything is held: only an + // envelope that matches the bid its block committed to may occupy a + // hold or cost a fetch. The signature is checked later, where an + // envelope is parked (see `Self::hold_envelope_awaiting_columns`), or + // by `on_execution_payload_envelope` when it is applied: gossip + // validation does not reach it for an envelope it queued. + if !envelope_matches_bid(&held.envelope.message, bid, slot) { + warn!( + %slot, + block_root = %ShortRoot(&root.0), + "Dropping an envelope that does not match its block's bid" + ); + self.refetch_envelope_if_children_wait(root); + return false; + } + + let present = self + .store + .data_column_indices_for(slot, &root) + .expect("DB read should succeed"); + if let Some(missing) = + missing_columns_for_envelope(commitment_count, &self.custody_columns, &present) + { + self.hold_envelope_awaiting_columns(root, slot, held, missing); + return false; + } + + let sidecars = if commitment_count == 0 { + Vec::new() + } else { + match self.stored_gloas_columns(slot, &root) { + Some(sidecars) => sidecars, + None => { + // Every sampled column was listed as stored a moment ago, + // so failing to read one back is a storage or decode + // fault, not a wait. Holding would name nothing missing + // and be redriven every slot for as long as the fault + // lasts, so the envelope is dropped (and asked for again + // if children wait on it). + warn!( + %slot, + block_root = %ShortRoot(&root.0), + "Dropping an envelope: a stored sampled column could not be read back" + ); + self.refetch_envelope_if_children_wait(root); + return false; + } + } + }; + + // The consensus checks come first, as `verify_execution_payload_envelope` + // orders them, so the execution client is only ever asked about an + // envelope this node would accept: signed by the builder, matching + // the bid and the state, and with its data available. + self.advance_beacon_clock_to(held.event_ms); + let config = self.store.config(); + if let Err(err) = fork_choice::check_execution_payload_envelope( + &self.store, + &held.envelope, + &config, + &sidecars, + &ExecutionEngine::valid(), + ) { + // p2p judged the envelope before it got here, so no peer is + // penalized from this side. + warn!( + %slot, + block_root = %ShortRoot(&root.0), + ?err, + "Dropping an execution payload envelope that failed verification" + ); + self.refetch_envelope_if_children_wait(root); + return false; + } + + // Then the execution client, whose answer has three outcomes where the + // specification's engine has two (see `docs/spec_deviations.md`). + // Without a client the follower trusts the payload, as it already does + // for every pre-gloas block it imports (`NotRequired`). + let status = match self.engine.clone() { + None => PayloadStatusEnum::Valid, + Some(client) => { + let asked = + beacon_engine::ask_envelope_status(&client, &block, &held.envelope).await; + // The client answered, so the envelopes waiting on it can be + // asked about in this same pass. + if asked.is_ok() { + self.release_envelopes_awaiting_engine(); + } + match asked { + // The payload does not hash to the block hash it claims: + // this envelope is bad, the payload the bid names may not + // be. Only this envelope is refused, so the builder's real + // one can still arrive. + // Some execution clients have no `INVALID_BLOCK_HASH` and + // answer a hash mismatch with `INVALID` and a null + // `latestValidHash` (ethrex does), whereas a payload that + // executed and failed names its last valid ancestor. So a + // null one is read the same way. + Ok(raw) + if raw.status == PayloadStatusValue::InvalidBlockHash + || (raw.status == PayloadStatusValue::Invalid + && raw.latest_valid_hash.is_none()) => + { + warn!( + %slot, + block_root = %ShortRoot(&root.0), + "Execution client found an envelope's contents do not match its \ + block hash; rejecting this envelope" + ); + self.envelopes.reject(held.envelope_root); + self.refetch_envelope_if_children_wait(root); + return false; + } + Ok(raw) => match beacon_engine::verdict(&raw) { + PayloadValidity::Validated | PayloadValidity::NotRequired => { + PayloadStatusEnum::Valid + } + PayloadValidity::Optimistic => PayloadStatusEnum::Syncing, + PayloadValidity::Invalidated { latest_valid_hash } => { + warn!( + %slot, + block_root = %ShortRoot(&root.0), + "Execution client found an envelope's payload invalid; \ + not applying it" + ); + self.condemn_payload_chain(root, latest_valid_hash); + // Children held for this payload are gone, and the + // head may have named a branch the verdict removed. + self.envelopes.head_dirty = true; + return false; + } + }, + Err(EngineError::EnvelopeMismatch(reason)) => { + warn!( + %slot, + block_root = %ShortRoot(&root.0), + reason, + "Dropping an envelope that does not match its block" + ); + self.refetch_envelope_if_children_wait(root); + return false; + } + // No answer after the whole retry ladder. The envelope is + // consensus-valid, so it is kept, and the per-slot redrive + // asks again; fetching it would only repeat the ladder + // for every copy that arrives. + Err(err) => { + warn!( + %slot, + block_root = %ShortRoot(&root.0), + %err, + "No verdict from the execution client; holding the envelope" + ); + metrics::inc_engine_no_verdict(); + self.hold_envelope_awaiting_engine(root, slot, held); + return false; + } + } + } + }; + + // `VALID` and `SYNCING` both let the payload in: the consensus layer + // has verified it, the execution layer may not have finished. + fork_choice::accept_execution_payload_envelope(&mut self.store, &held.envelope); + info!( + %slot, + block_root = %ShortRoot(&root.0), + ?status, + "Execution payload envelope verified" + ); + self.store + .insert_beacon_block_payload_status(root, slot, status); + // Lets `latestValidHash` find this FULL node. + self.store.insert_beacon_el_block_hash( + root, + slot, + held.envelope.message.payload.block_hash, + ); + self.envelopes.verified.push_back(root); + true + } + + /// Keep a consensus-valid envelope until the execution client answers. + fn hold_envelope_awaiting_engine(&mut self, root: H256, slot: u64, mut held: HeldEnvelope) { + held.slot = slot; + self.envelopes.awaiting_engine.entry(root).or_insert(held); + self.publish_envelope_queues(); + } + + /// Once a slot: ask the execution client again about the oldest envelope + /// it has not answered for. + /// + /// One, not all: with the client down each ask spends the whole retry + /// ladder, and the held envelopes grow with the finality lag. The rest are + /// released by [`Self::release_envelopes_awaiting_engine`] once an ask gets + /// any answer. + pub(crate) fn redrive_envelopes_awaiting_engine(&mut self) { + let oldest = self + .envelopes + .awaiting_engine + .iter() + .min_by_key(|(root, held)| (held.slot, held.event_ms, **root)) + .map(|(root, _)| *root); + self.envelopes.work.extend(oldest); + } + + /// Queue every envelope held for the execution client. + fn release_envelopes_awaiting_engine(&mut self) { + let roots: Vec = self.envelopes.awaiting_engine.keys().copied().collect(); + self.envelopes.work.extend(roots); + } + + /// Every sampled column stored for `root`, decoded as gloas sidecars, or + /// `None` if any is missing or not one. + fn stored_gloas_columns( + &self, + slot: u64, + root: &H256, + ) -> Option> { + self.custody_columns + .iter() + .map( + |&index| match self.store.get_data_column(slot, root, index) { + Ok(Some(DataColumnSidecar::Gloas(sidecar))) => Some(sidecar), + Ok(Some(DataColumnSidecar::Fulu(_))) | Ok(None) => None, + Err(err) => { + error!(%err, "Failed to read back a stored data column sidecar"); + None + } + }, + ) + .collect() + } + + /// Hold `held` for a block that is in the store but has no post-state yet, + /// after checking it against that block's bid. + /// + /// Range catch-up delivers a batch's envelopes right behind its blocks, + /// when all but the first block are still pending or held for a parent's + /// payload. Those envelopes are not strangers: the stored block names what + /// payload it committed to, so one that matches is held outside the + /// per-arrival-slot cap, which exists for roots nobody has vouched for. + /// `Err(held)` hands the envelope back when the block is not stored at all, + /// for the unverified path. + fn hold_envelope_for_stored_block( + &mut self, + root: H256, + mut held: HeldEnvelope, + ) -> Option { + // Repeat copies cost nothing: the per-root bound and the duplicate + // check come before the block is decoded. + if let Some(held_for_root) = self.envelopes.awaiting_import.get(&root) + && (held_for_root.len() >= MAX_ENVELOPES_PER_UNMATCHED_ROOT + || held_for_root + .iter() + .any(|other| other.envelope_root == held.envelope_root)) + { + return None; + } + if !self.store.has_block(&root) { + return Some(held); + } + let Ok(Some(block)) = self.store.get_signed_block(&root) else { + return Some(held); + }; + let slot = block.slot(); + let Some(bid) = crate::beacon_payloads::bid_of(&block) else { + warn!( + %slot, + block_root = %ShortRoot(&root.0), + "Dropping an envelope: its block is not a gloas block" + ); + return None; + }; + if !envelope_matches_bid(&held.envelope.message, bid, slot) { + warn!( + %slot, + block_root = %ShortRoot(&root.0), + "Dropping an envelope that does not match its stored block's bid" + ); + return None; + } + let finalized = self + .store + .latest_finalized() + .expect("finalized checkpoint exists"); + if slot < finalized.slot && root != finalized.root { + return None; + } + held.slot = slot; + debug!( + %slot, + block_root = %ShortRoot(&root.0), + "Holding an envelope until its stored block is imported" + ); + self.envelopes + .awaiting_import + .entry(root) + .or_default() + .push(held); + self.envelopes.trim_awaiting_import(); + self.publish_envelope_queues(); + None + } + + fn hold_envelope_awaiting_block(&mut self, root: H256, held: HeldEnvelope) { + let queues = &mut self.envelopes; + if let Some(held_for_root) = queues.awaiting_block.get_mut(&root) { + if held_for_root.len() >= MAX_ENVELOPES_PER_UNMATCHED_ROOT + || held_for_root + .iter() + .any(|other| other.envelope_root == held.envelope_root) + { + return; + } + held_for_root.push(held); + self.publish_envelope_queues(); + return; + } + let roots_at_slot = queues + .awaiting_block + .values() + .filter(|others| others.iter().any(|other| other.slot == held.slot)) + .count(); + if roots_at_slot >= MAX_ENVELOPES_AWAITING_BLOCK_PER_SLOT { + trace!( + slot = held.slot, + block_root = %ShortRoot(&root.0), + "Dropping an envelope: its slot already holds the most unmatched roots allowed" + ); + return; + } + debug!( + block_root = %ShortRoot(&root.0), + "Holding an envelope until its block is imported" + ); + queues.awaiting_block.insert(root, vec![held]); + self.publish_envelope_queues(); + } + + /// Whether `envelope`'s builder signature holds against the post-state of + /// the block `root`, which must be imported. A state that cannot be read, + /// or a builder the state does not know, reads as an invalid signature. + fn envelope_signature_holds( + &self, + root: H256, + envelope: &gloas::SignedExecutionPayloadEnvelope, + ) -> bool { + let Some(state) = self.store.get_state(&root).ok().flatten() else { + return false; + }; + stf_gloas::verify_execution_payload_envelope_signature(&state, envelope).unwrap_or(false) + } + + fn hold_envelope_awaiting_columns( + &mut self, + root: H256, + slot: u64, + mut held: HeldEnvelope, + missing: Vec, + ) { + held.slot = slot; + // One envelope per root; a re-delivery replaces nothing. + if self.envelopes.awaiting_columns.contains_key(&root) { + return; + } + // Gossip validation answered `Queue` for an envelope whose block it + // had not seen, before it reached the signature, so nothing has + // checked this one yet. A forged copy carrying the honest payload and + // a bad signature would otherwise take the root's one slot here and + // shut the honest envelope out. One BLS verification per root, against + // the state of the block it names. + if !self.envelope_signature_holds(root, &held.envelope) { + warn!( + %slot, + block_root = %ShortRoot(&root.0), + "Dropping an envelope with an invalid signature" + ); + self.refetch_envelope_if_children_wait(root); + return; + } + info!( + %slot, + block_root = %ShortRoot(&root.0), + ?missing, + "Holding an envelope: its sampled columns have not all arrived yet" + ); + self.envelopes.awaiting_columns.insert(root, held); + if !missing.is_empty() { + self.request_missing_columns(root, missing); + } + self.publish_envelope_queues(); + } + + /// Queue `root`'s envelope for a look in [`Self::settle_envelopes`], if one + /// is waiting for the block to import. + pub(crate) fn note_block_imported_for_envelope(&mut self, root: H256) { + if self.envelopes.awaiting_block.contains_key(&root) + || self.envelopes.awaiting_import.contains_key(&root) + { + self.envelopes.work.push_back(root); + } + } + + /// Queue `root`'s envelope for a look, if one is waiting for columns. + pub(crate) fn note_column_stored_for_envelope(&mut self, root: H256) { + if self.envelopes.awaiting_columns.contains_key(&root) { + self.envelopes.work.push_back(root); + } + } + + /// Once a slot: ask for the envelope of every parent that has blocks held + /// for its payload and no envelope held or verified, so one failed fetch + /// does not stall the child for good. + pub(crate) fn redrive_missing_envelopes(&self) { + for &parent_root in self.envelopes.blocks_awaiting_parent_payload.keys() { + if self.parent_envelope_is_unaccounted_for(&parent_root) { + self.request_missing_envelope(parent_root); + } + } + } + + /// Whether `parent_root`'s envelope is neither held in any queue nor + /// verified (nor its payload found invalid), so asking for it can help. + fn parent_envelope_is_unaccounted_for(&self, parent_root: &H256) -> bool { + !self.envelopes.awaiting_block.contains_key(parent_root) + && !self.envelopes.awaiting_import.contains_key(parent_root) + && !self.envelopes.awaiting_columns.contains_key(parent_root) + && !self.envelopes.awaiting_engine.contains_key(parent_root) + && !self.store.has_verified_payload(parent_root) + && !self + .store + .beacon_block_payload_status(*parent_root) + .is_invalidated() + } + + /// Once a slot: look at every envelope waiting for columns, and ask peers + /// for whatever is still missing. + pub(crate) fn redrive_envelopes_awaiting_columns(&mut self) { + let held: Vec<(H256, u64)> = self + .envelopes + .awaiting_columns + .iter() + .map(|(&root, held)| (root, held.slot)) + .collect(); + for (root, slot) in held { + let present = self + .store + .data_column_indices_for(slot, &root) + .expect("DB read should succeed"); + let missing: Vec = self + .custody_columns + .iter() + .copied() + .filter(|index| !present.contains(index)) + .collect(); + if missing.is_empty() { + self.envelopes.work.push_back(root); + } else { + self.request_missing_columns(root, missing); + } + } + } + + /// The held envelopes for `root` if what they waited on is now there. + fn take_ready_envelopes(&mut self, root: H256) -> Vec { + if (self.envelopes.awaiting_block.contains_key(&root) + || self.envelopes.awaiting_import.contains_key(&root)) + && self.store.has_state(&root).expect("DB read should succeed") + { + let mut ready = self + .envelopes + .awaiting_import + .remove(&root) + .unwrap_or_default(); + ready.extend( + self.envelopes + .awaiting_block + .remove(&root) + .unwrap_or_default(), + ); + return ready; + } + if let Some(held) = self.envelopes.awaiting_engine.remove(&root) { + return vec![held]; + } + let Some(slot) = self + .envelopes + .awaiting_columns + .get(&root) + .map(|held| held.slot) + else { + return Vec::new(); + }; + if custody_columns_present(&self.store, slot, &root, &self.custody_columns) { + return self + .envelopes + .awaiting_columns + .remove(&root) + .into_iter() + .collect(); + } + Vec::new() + } + + /// Drain the roots pushed since the last call: verify each ready envelope, + /// import the blocks that were waiting on each verified payload, and + /// recompute the head once if anything changed. + pub(crate) async fn settle_envelopes(&mut self) { + // Nothing to do on most ticks, and none at all on lean: leave the + // gauges alone rather than rewriting them every interval. + if self.envelopes.work.is_empty() + && self.envelopes.verified.is_empty() + && !self.envelopes.head_dirty + { + return; + } + let mut changed = std::mem::take(&mut self.envelopes.head_dirty); + loop { + if let Some(root) = self.envelopes.work.pop_front() { + // The first that verifies wins; the rest are forgeries or + // duplicates of it. One parked for columns is enough too: it + // matched the bid, whose block hash commits to the whole + // payload, so a second bid-consistent candidate differs only + // in a signature, which a parked one has had checked. + for held in self.take_ready_envelopes(root) { + if self.apply_or_hold_envelope(held).await { + break; + } + } + } else if let Some(root) = self.envelopes.verified.pop_front() { + changed = true; + self.release_blocks_awaiting_parent_payload(root).await; + } else { + break; + } + changed |= std::mem::take(&mut self.envelopes.head_dirty); + } + self.publish_envelope_queues(); + if changed { + self.recompute_beacon_head().await; + } + } + + /// Whether `block` is a gloas block whose parent is FULL while that + /// parent's payload is not yet verified: `on_block` would reject it with + /// `is_payload_verified(store, block.parent_root)`. + pub(crate) fn parent_payload_unverified(&self, block: &SignedBeaconBlock) -> bool { + if !block.fork_name().has_payload_envelopes() { + return false; + } + matches!( + fork_choice::get_parent_payload_status(&self.store, block), + Ok(PayloadStatus::Full) + ) && !fork_choice::is_payload_verified(&self.store, block.parent_root()) + } + + /// Keep `block` until its parent's envelope is verified. + /// + /// Written the way a block held for a missing parent is: no `LiveChain` + /// entry, so it stays invisible to fork choice, and readable back by root. + pub(crate) fn hold_block_for_parent_payload( + &mut self, + block: SignedBeaconBlock, + timings: ImportTimings, + ) { + let slot = block.slot(); + let block_root = block.message_hash_tree_root(); + let parent_root = block.parent_root(); + + // The parent's payload was found invalid, so this block builds on a + // FULL node that will never exist, and neither will its descendants. + if self + .store + .beacon_block_payload_status(parent_root) + .is_invalidated() + { + warn!( + %slot, + block_root = %ShortRoot(&block_root.0), + parent_root = %ShortRoot(&parent_root.0), + "Dropping a block: it builds on a payload the execution client found invalid" + ); + self.discard_pending_subtree(block_root); + return; + } + + let held_for_parent = self + .envelopes + .blocks_awaiting_parent_payload + .get(&parent_root); + if held_for_parent.is_some_and(|held| held.contains_key(&block_root)) { + return; + } + if held_for_parent.is_some_and(|held| held.len() >= MAX_BLOCKS_HELD_PER_PARENT_PAYLOAD) { + warn!( + %slot, + block_root = %ShortRoot(&block_root.0), + parent_root = %ShortRoot(&parent_root.0), + "Dropping a block: its parent already has the most children held for its payload" + ); + // Its own pending children would otherwise wait on a root that + // never imports, and their descendants re-queue it. + self.discard_pending_subtree(block_root); + return; + } + + info!( + %slot, + block_root = %ShortRoot(&block_root.0), + parent_root = %ShortRoot(&parent_root.0), + "Holding block: its parent's execution payload is not verified yet" + ); + self.store + .insert_pending_block(block_root, block) + .expect("DB insert should succeed"); + self.envelopes + .blocks_awaiting_parent_payload + .entry(parent_root) + .or_default() + .insert(block_root, slot); + self.held_timings.insert(block_root, timings); + self.publish_envelope_queues(); + // Asked at once whatever the block's source. A synced block is not + // covered by a range batch's envelopes: `BlockSource::Sync` also marks + // by-root answers and pending children a cascade released, and a range + // answer that fails or comes back empty falls back to this same ask. + // Waiting for the once-a-slot redrive instead is what held a catching-up + // follower to one block per slot. p2p dedups in-flight roots, so a + // range batch whose envelopes are on their way costs at most one extra + // request per block. + // Unless a queue already holds it: a cascade of held children whose + // parents' envelopes arrived with their range batch would otherwise + // cost one by-root request and signature check per block. + if self.parent_envelope_is_unaccounted_for(&parent_root) { + self.request_missing_envelope(parent_root); + } + } + + /// Ask p2p for `root`'s envelope by root. The request carries nothing but + /// the flag, since an envelope names its block and nothing else. + pub(crate) fn request_missing_envelope(&self, root: H256) { + if let Some(ref p2p) = self.p2p { + let _ = p2p + .fetch_block(FetchRequest { + block_root: root, + needs_block: false, + needs_envelope: true, + columns: Vec::new(), + }) + .inspect_err(|err| { + error!(block_root = %ShortRoot(&root.0), %err, "Failed to request a missing envelope") + }); + } + } + + /// After an envelope was dropped: ask again if blocks wait on exactly it. + fn refetch_envelope_if_children_wait(&self, root: H256) { + if self + .envelopes + .blocks_awaiting_parent_payload + .contains_key(&root) + { + self.request_missing_envelope(root); + } + } + + /// Re-import every block that waited on `parent_root`'s payload, now that + /// it is verified. + async fn release_blocks_awaiting_parent_payload(&mut self, parent_root: H256) { + let Some(children) = self + .envelopes + .blocks_awaiting_parent_payload + .remove(&parent_root) + else { + return; + }; + self.publish_envelope_queues(); + for (block_root, slot) in children { + let timings = self + .held_timings + .remove(&block_root) + .unwrap_or_else(ImportTimings::starting_now); + let Ok(Some(block)) = self.store.get_signed_block(&block_root) else { + error!( + %slot, + block_root = %ShortRoot(&block_root.0), + "A block held for its parent's payload vanished from the store" + ); + continue; + }; + debug!( + %slot, + block_root = %ShortRoot(&block_root.0), + "Parent payload verified; re-importing a held block" + ); + self.on_block(block, timings).await; + } + } + + /// Forget `block_root` as a block held for a parent's payload. Called from + /// the funnel every eviction goes through, so `held_timings` and the queue + /// cannot outlive the block they describe. + pub(crate) fn forget_block_held_for_parent_payload(&mut self, block_root: H256) { + if !self.envelopes.is_block_held(&block_root) { + return; + } + self.envelopes + .blocks_awaiting_parent_payload + .retain(|_, children| { + children.remove(&block_root); + !children.is_empty() + }); + self.publish_envelope_queues(); + } + + /// Drop the envelopes held for `block_root`'s import, which is not coming. + /// Called from the funnel every block eviction goes through. + pub(crate) fn forget_envelopes_awaiting_import(&mut self, block_root: H256) { + if self.envelopes.awaiting_import.remove(&block_root).is_some() { + self.publish_envelope_queues(); + } + } + + /// Drop every queued envelope and held block that finality has superseded. + /// + /// The counterpart of [`Self::evict_held_blocks_at_or_below_finality`], run + /// beside it: an envelope whose block never arrives, or a parent whose + /// payload never does, would otherwise hold its entry for the node's whole + /// uptime. Envelopes are kept at the finalized slot itself, since its + /// children may still name it as their FULL parent. + pub(crate) fn evict_envelope_queues_at_or_below_finality(&mut self) { + if self.envelopes.is_empty() { + return; + } + let finalized = self + .store + .latest_finalized() + .expect("finalized checkpoint exists"); + let finalized_slot = finalized.slot; + + // The finalized block's own envelope is exempt by name, as its + // execution hash is from `prune_beacon_el_block_hashes`: the + // checkpoint slot is an epoch boundary, and the block can sit before + // a skipped one. + let config = self.store.config(); + let current_slot = fork_choice::get_current_slot(&self.store, &config); + self.envelopes.awaiting_block.retain(|root, held| { + held.retain(|other| { + let fresh = + other.slot.saturating_add(ENVELOPE_AWAITING_BLOCK_TTL_SLOTS) >= current_slot; + fresh && (other.slot >= finalized_slot || *root == finalized.root) + }); + !held.is_empty() + }); + self.envelopes.awaiting_import.retain(|root, held| { + held.retain(|other| other.slot >= finalized_slot || *root == finalized.root); + !held.is_empty() + }); + self.envelopes + .awaiting_columns + .retain(|root, held| held.slot >= finalized_slot || *root == finalized.root); + self.envelopes + .awaiting_engine + .retain(|root, held| held.slot >= finalized_slot || *root == finalized.root); + + let stale: Vec = self + .envelopes + .blocks_awaiting_parent_payload + .values() + .flat_map(|children| children.iter()) + .filter(|&(_, &slot)| slot <= finalized_slot) + .map(|(&root, _)| root) + .collect(); + for root in stale { + self.discard_pending_subtree(root); + } + self.publish_envelope_queues(); + } + + pub(crate) fn publish_envelope_queues(&self) { + let unmatched: usize = self + .envelopes + .awaiting_block + .values() + .chain(self.envelopes.awaiting_import.values()) + .map(Vec::len) + .sum(); + metrics::set_envelopes_awaiting_block(unmatched as u64); + metrics::set_envelopes_awaiting_columns(self.envelopes.awaiting_columns.len() as u64); + metrics::set_blocks_awaiting_parent_payload(self.envelopes.held_blocks() as u64); + } +} + +#[cfg(test)] +mod tests { + use std::path::PathBuf; + use std::sync::Arc; + + use ethlambda_state_transition::beacon::config::Config; + use ethlambda_storage::backend::InMemoryBackend; + use ethlambda_types::beacon::containers::BeaconState; + use libssz::SszDecode as _; + + use super::*; + use crate::tests::{GENESIS_TIME, beacon_server, beacon_store}; + use ethlambda_network_api::BlockSource; + use ethlambda_state_transition::beacon::fork::ForkName; + + /// The mainnet preset with every fork active from genesis, as the fork + /// choice fixtures that carry no `config.yaml` of their own are run. + fn fixture_config() -> Config { + ForkName::ALL + .into_iter() + .fold(Config::mainnet(), |config, fork| { + config.with_fork_epoch(fork, 0) + }) + } + + /// One fork choice fixture case, read the way the state transition crate's + /// spec runner reads it. + struct Case(PathBuf); + + impl Case { + fn new(handler: &str, case: &str) -> Self { + Self( + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../../consensus-spec-tests/tests/mainnet/gloas/fork_choice") + .join(handler) + .join("pyspec_tests") + .join(case), + ) + } + + fn bytes(&self, name: &str) -> Vec { + let path = self.0.join(format!("{name}.ssz_snappy")); + let compressed = std::fs::read(&path) + .unwrap_or_else(|err| panic!("reading {}: {err}", path.display())); + snap::raw::Decoder::new() + .decompress_vec(&compressed) + .unwrap_or_else(|err| panic!("decompressing {}: {err}", path.display())) + } + + fn block(&self, root_hex: &str) -> SignedBeaconBlock { + let bytes = self.bytes(&format!("block_0x{root_hex}")); + SignedBeaconBlock::from_ssz(ForkName::Gloas, &bytes).expect("block decodes") + } + + fn envelope(&self, root_hex: &str) -> gloas::SignedExecutionPayloadEnvelope { + let bytes = self.bytes(&format!("execution_payload_envelope_0x{root_hex}")); + gloas::SignedExecutionPayloadEnvelope::from_ssz_bytes(&bytes).expect("envelope decodes") + } + + /// A server over the case's anchor, its clock at `time` seconds. + fn server(&self, time: u64) -> BlockChainServer { + let state = BeaconState::from_ssz(ForkName::Gloas, &self.bytes("anchor_state")) + .expect("anchor state decodes"); + let anchor = SignedBeaconBlock::Gloas(gloas::SignedBeaconBlock { + message: libssz::SszDecode::from_ssz_bytes(&self.bytes("anchor_block")) + .expect("anchor block decodes"), + signature: Default::default(), + }); + let config = fixture_config(); + let backend = Arc::new(InMemoryBackend::new()); + let store = fork_choice::get_forkchoice_store(backend, state, anchor, &config) + .expect("anchor store"); + let mut server = beacon_server(store); + fork_choice::on_tick(&mut server.store, time, &config); + server + } + } + + const TIEBREAK: &str = "get_head_full_payload_tiebreak"; + const PARENT: &str = "780218dcb4c1afc8ba72f325eb4f9ad43523356242345752d0e7a52391c15582"; + const CHILD: &str = "14f9d9ee25f47e0d8dd8ea7d01b206737e2aede3a7732d953546348511f58128"; + const PARENT_ENVELOPE: &str = + "bba7bbe6c4fb93af68a15e1048436396821dfaf7af6e2a5049dbef906df6801e"; + + fn root(hex_root: &str) -> H256 { + let bytes: [u8; 32] = hex::decode(hex_root).unwrap().try_into().unwrap(); + H256(bytes) + } + + fn head(server: &BlockChainServer) -> (H256, PayloadStatus) { + let node = fork_choice::get_head_node(&server.store, &fixture_config()).expect("head"); + (node.root, node.payload_status) + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_gloas_block_imports_without_columns_and_is_not_held() { + let case = Case::new("get_head", TIEBREAK); + let mut server = case.server(12); + server.custody_columns = (0..8).collect(); + + server + .on_block(case.block(PARENT), ImportTimings::default()) + .await; + + assert!(server.store.has_state(&root(PARENT)).unwrap()); + assert!(server.blocks_awaiting_columns.is_empty()); + assert!(server.envelopes.blocks_awaiting_parent_payload.is_empty()); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_envelope_ahead_of_its_block_is_held_then_applied_when_the_block_imports() { + let case = Case::new("get_head", TIEBREAK); + let mut server = case.server(12); + + server + .receive_envelope(case.envelope(PARENT_ENVELOPE), 0) + .await; + server.settle_envelopes().await; + assert!(server.envelopes.awaiting_block.contains_key(&root(PARENT))); + assert!(!server.store.has_verified_payload(&root(PARENT))); + + server + .on_block(case.block(PARENT), ImportTimings::default()) + .await; + // The import only queues the look; settling is the handler's step. + assert!(!server.store.has_verified_payload(&root(PARENT))); + server.settle_envelopes().await; + + assert!(server.envelopes.awaiting_block.is_empty()); + assert!(server.store.has_verified_payload(&root(PARENT))); + assert_eq!( + server.store.beacon_block_payload_status(root(PARENT)), + PayloadStatusEnum::Valid + ); + assert_eq!(head(&server), (root(PARENT), PayloadStatus::Full)); + } + + /// The `on_execution_payload_envelope` cases share one anchor and one + /// first block, so the valid case's envelope fits the invalid-full-child + /// case's chain: the one place a block built on a FULL parent exists. + const FULL_CHILD_CASE: &str = "on_execution_payload_envelope_invalid_full_child"; + const FULL_PARENT: &str = "76bf14e70fc96a5a3442a46dff3bfb897d5568cdb4dc7e94419f625df2fcd9e5"; + const FULL_CHILD: &str = "7a25db19479cdcc24bd59bb8f5feaeee2e9b24bd25f5eddd54dd809c567b5e5b"; + + fn valid_envelope_for_full_parent() -> gloas::SignedExecutionPayloadEnvelope { + Case::new( + "on_execution_payload_envelope", + "on_execution_payload_envelope_valid", + ) + .envelope("240ec2c17454f3e653fd3b43961e3c9a16770d336b0e882535c01f99b9223b41") + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_block_on_a_full_parent_waits_for_the_parents_envelope_then_imports() { + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let mut server = case.server(12); + server + .on_block(case.block(FULL_PARENT), ImportTimings::default()) + .await; + fork_choice::on_tick(&mut server.store, 24, &fixture_config()); + + server + .on_block(case.block(FULL_CHILD), ImportTimings::default()) + .await; + + // Held, not failed, and invisible to fork choice. + assert!(!server.store.has_state(&root(FULL_CHILD)).unwrap()); + assert!(server.envelopes.is_block_held(&root(FULL_CHILD))); + assert!(server.held_timings.contains_key(&root(FULL_CHILD))); + assert_eq!(head(&server).0, root(FULL_PARENT)); + + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + server.settle_envelopes().await; + + assert!(server.store.has_state(&root(FULL_CHILD)).unwrap()); + assert!(server.envelopes.blocks_awaiting_parent_payload.is_empty()); + assert!(!server.held_timings.contains_key(&root(FULL_CHILD))); + assert_eq!(head(&server).0, root(FULL_CHILD)); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_envelope_that_fails_verification_is_dropped_and_its_child_stays_held() { + let case = Case::new( + "on_execution_payload_envelope", + "on_execution_payload_envelope_invalid_full_child", + ); + let parent = "76bf14e70fc96a5a3442a46dff3bfb897d5568cdb4dc7e94419f625df2fcd9e5"; + let child = "7a25db19479cdcc24bd59bb8f5feaeee2e9b24bd25f5eddd54dd809c567b5e5b"; + let bad_envelope = "73c5948d041ccb171b9331c0eac4a3bb8cdcf01994916fee4c7493e63e890559"; + let mut server = case.server(12); + server + .on_block(case.block(parent), ImportTimings::default()) + .await; + fork_choice::on_tick(&mut server.store, 24, &fixture_config()); + server + .on_block(case.block(child), ImportTimings::default()) + .await; + assert!(server.envelopes.is_block_held(&root(child))); + + server + .receive_envelope(case.envelope(bad_envelope), 0) + .await; + server.settle_envelopes().await; + + assert!(!server.store.has_verified_payload(&root(parent))); + assert!(server.envelopes.awaiting_block.is_empty()); + assert!(server.envelopes.awaiting_columns.is_empty()); + assert!(server.envelopes.is_block_held(&root(child))); + assert!(!server.store.has_state(&root(child)).unwrap()); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_envelope_for_an_already_verified_payload_is_ignored() { + let case = Case::new("get_head", TIEBREAK); + let mut server = case.server(12); + server + .on_block(case.block(PARENT), ImportTimings::default()) + .await; + server + .receive_envelope(case.envelope(PARENT_ENVELOPE), 0) + .await; + assert!(server.store.has_verified_payload(&root(PARENT))); + + server + .receive_envelope(case.envelope(PARENT_ENVELOPE), 0) + .await; + + assert!(server.envelopes.awaiting_block.is_empty()); + assert!(server.envelopes.awaiting_columns.is_empty()); + } + + #[test] + fn an_envelope_waits_for_every_sampled_column_when_the_bid_has_commitments() { + let custody = [2, 5, 9]; + // Nothing outstanding without commitments, whatever is stored. + assert_eq!(missing_columns_for_envelope(0, &custody, &[]), None); + // Some stored, some not: the rest are named, in custody order. + assert_eq!( + missing_columns_for_envelope(3, &custody, &[5]), + Some(vec![2, 9]) + ); + // A column outside the sample does not count. + assert_eq!( + missing_columns_for_envelope(1, &custody, &[1, 2, 5]), + Some(vec![9]) + ); + assert_eq!(missing_columns_for_envelope(1, &custody, &[9, 5, 2]), None); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_stored_column_queues_only_an_envelope_that_waits_for_columns() { + let case = Case::new("get_head", TIEBREAK); + let mut server = case.server(12); + let envelope = case.envelope(PARENT_ENVELOPE); + + server.note_column_stored_for_envelope(root(PARENT)); + assert!(server.envelopes.work.is_empty()); + + server + .envelopes + .awaiting_columns + .insert(root(PARENT), HeldEnvelope::new(envelope, 1, 0)); + server.note_column_stored_for_envelope(root(PARENT)); + assert_eq!(server.envelopes.work, VecDeque::from([root(PARENT)])); + + // Columns still missing: the look finds nothing ready, and the + // envelope stays where it is. + server.custody_columns = vec![3]; + server.settle_envelopes().await; + assert!( + server + .envelopes + .awaiting_columns + .contains_key(&root(PARENT)) + ); + } + + /// Gossip validation queues an envelope for an unseen block before it + /// reaches the signature, so the actor checks it where it parks one: a + /// copy with the honest payload and a bad signature, arriving first, must + /// not take the root's one slot from the honest envelope. + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_bad_signature_copy_does_not_take_the_parking_slot() { + let case = Case::new("get_head", TIEBREAK); + let mut server = case.server(12); + server + .on_block(case.block(PARENT), ImportTimings::default()) + .await; + let honest = case.envelope(PARENT_ENVELOPE); + let mut forged = honest.clone(); + forged.signature = Default::default(); + assert_ne!(forged.signature, honest.signature); + + server.hold_envelope_awaiting_columns( + root(PARENT), + 1, + HeldEnvelope::new(forged, 1, 0), + vec![], + ); + assert!(server.envelopes.awaiting_columns.is_empty()); + + server.hold_envelope_awaiting_columns( + root(PARENT), + 1, + HeldEnvelope::new(honest.clone(), 1, 0), + vec![], + ); + assert_eq!( + server.envelopes.awaiting_columns[&root(PARENT)].envelope, + honest + ); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn unmatched_envelopes_are_capped_per_slot_and_per_root() { + let case = Case::new("get_head", TIEBREAK); + let mut server = case.server(12); + let envelope = case.envelope(PARENT_ENVELOPE); + let with_root = |byte: u8| { + let mut envelope = envelope.clone(); + envelope.message.beacon_block_root = H256::repeat_byte(byte); + envelope + }; + + for byte in 1..=MAX_ENVELOPES_AWAITING_BLOCK_PER_SLOT as u8 + 2 { + server.receive_envelope(with_root(byte), 0).await; + } + // A second delivery of a held root changes nothing. + server.receive_envelope(with_root(1), 0).await; + + assert_eq!( + server.envelopes.awaiting_block.len(), + MAX_ENVELOPES_AWAITING_BLOCK_PER_SLOT + ); + assert!( + server + .envelopes + .awaiting_block + .contains_key(&H256::repeat_byte(1)) + ); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn blocks_held_for_one_parents_payload_are_capped() { + let case = Case::new("get_head", TIEBREAK); + let mut server = case.server(12); + let child = case.block(CHILD); + for extra in 0..MAX_BLOCKS_HELD_PER_PARENT_PAYLOAD as u64 + 2 { + let mut block = child.clone(); + if let SignedBeaconBlock::Gloas(inner) = &mut block { + inner.message.slot += extra; + } + server.hold_block_for_parent_payload(block, ImportTimings::default()); + } + + let held = &server.envelopes.blocks_awaiting_parent_payload; + assert_eq!(held.len(), 1); + assert_eq!( + held[&child.parent_root()].len(), + MAX_BLOCKS_HELD_PER_PARENT_PAYLOAD + ); + assert_eq!( + server.held_timings.len(), + MAX_BLOCKS_HELD_PER_PARENT_PAYLOAD + ); + } + + #[test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + fn finality_evicts_all_three_queues() { + // The finalized slot is fixed at init, so the queues are filled + // directly around it. + let mut server = beacon_server(beacon_store(GENESIS_TIME, 10)); + let envelope = Case::new("get_head", TIEBREAK).envelope(PARENT_ENVELOPE); + let held = |slot: u64| HeldEnvelope::new(envelope.clone(), slot, 0); + let queues = &mut server.envelopes; + queues + .awaiting_block + .insert(H256::repeat_byte(1), vec![held(9)]); + queues + .awaiting_block + .insert(H256::repeat_byte(2), vec![held(11)]); + queues + .awaiting_columns + .insert(H256::repeat_byte(3), held(9)); + queues + .awaiting_columns + .insert(H256::repeat_byte(4), held(10)); + let parent = H256::repeat_byte(5); + queues.blocks_awaiting_parent_payload.insert( + parent, + HashMap::from([(H256::repeat_byte(6), 10), (H256::repeat_byte(7), 11)]), + ); + server + .held_timings + .insert(H256::repeat_byte(6), ImportTimings::default()); + + server.evict_envelope_queues_at_or_below_finality(); + + let queues = &server.envelopes; + assert_eq!( + queues.awaiting_block.keys().collect::>(), + vec![&H256::repeat_byte(2)] + ); + // The finalized slot itself stays: its children may name it as FULL. + assert_eq!( + queues.awaiting_columns.keys().collect::>(), + vec![&H256::repeat_byte(4)] + ); + assert_eq!( + queues.blocks_awaiting_parent_payload[&parent], + HashMap::from([(H256::repeat_byte(7), 11)]) + ); + assert!(!server.held_timings.contains_key(&H256::repeat_byte(6))); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_forged_envelope_for_a_root_does_not_evict_the_honest_one() { + let case = Case::new("get_head", TIEBREAK); + let mut server = case.server(12); + let honest = case.envelope(PARENT_ENVELOPE); + let forged = |gas_limit: u64| { + let mut envelope = honest.clone(); + envelope.message.payload.gas_limit = gas_limit; + envelope + }; + + // The forgeries arrive first and fill what the root may hold, bar one. + for gas_limit in 1..MAX_ENVELOPES_PER_UNMATCHED_ROOT as u64 { + server.receive_envelope(forged(gas_limit), 0).await; + } + server.receive_envelope(honest.clone(), 0).await; + // A re-delivery of the honest one and a surplus forgery add nothing. + server.receive_envelope(honest.clone(), 0).await; + server.receive_envelope(forged(99), 0).await; + assert_eq!( + server.envelopes.awaiting_block[&root(PARENT)].len(), + MAX_ENVELOPES_PER_UNMATCHED_ROOT + ); + + server + .on_block(case.block(PARENT), ImportTimings::default()) + .await; + server.settle_envelopes().await; + + assert!(server.store.has_verified_payload(&root(PARENT))); + assert!(server.envelopes.awaiting_block.is_empty()); + } + + #[test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + fn the_finalized_blocks_own_envelope_survives_eviction() { + // Finalized at slot 10, so the finalized root is the zero anchor of + // `beacon_store`. + let mut server = beacon_server(beacon_store(GENESIS_TIME, 10)); + let envelope = Case::new("get_head", TIEBREAK).envelope(PARENT_ENVELOPE); + let finalized_root = server.store.latest_finalized().unwrap().root; + server.envelopes.awaiting_block.insert( + finalized_root, + vec![HeldEnvelope::new(envelope.clone(), 3, 0)], + ); + server + .envelopes + .awaiting_columns + .insert(finalized_root, HeldEnvelope::new(envelope, 3, 0)); + + server.evict_envelope_queues_at_or_below_finality(); + + assert!( + server + .envelopes + .awaiting_block + .contains_key(&finalized_root) + ); + assert!( + server + .envelopes + .awaiting_columns + .contains_key(&finalized_root) + ); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_block_held_for_its_parents_payload_asks_p2p_for_the_envelope() { + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let state_server = case.server(12); + let (mut server, p2p) = crate::tests::beacon_server_recording(state_server.store.clone()); + server + .on_block(case.block(FULL_PARENT), ImportTimings::default()) + .await; + fork_choice::on_tick(&mut server.store, 24, &fixture_config()); + + server + .on_block(case.block(FULL_CHILD), ImportTimings::default()) + .await; + + let fetches = p2p.fetches.lock().unwrap(); + assert!(fetches.iter().any(|request| request.needs_envelope + && !request.needs_block + && request.block_root == root(FULL_PARENT))); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_dropped_block_takes_its_pending_children_with_it() { + let case = Case::new("get_head", TIEBREAK); + let mut server = case.server(12); + let child = case.block(CHILD); + let held_roots: Vec = (0..MAX_BLOCKS_HELD_PER_PARENT_PAYLOAD as u64) + .map(|extra| { + let mut block = child.clone(); + if let SignedBeaconBlock::Gloas(inner) = &mut block { + inner.message.slot += extra; + } + let block_root = block.message_hash_tree_root(); + server.hold_block_for_parent_payload(block, ImportTimings::default()); + block_root + }) + .collect(); + assert_eq!(held_roots.len(), MAX_BLOCKS_HELD_PER_PARENT_PAYLOAD); + + // One more, with a child pending on it: the cap drops the block and + // the child goes too. + let mut surplus = child.clone(); + if let SignedBeaconBlock::Gloas(inner) = &mut surplus { + inner.message.slot += 50; + } + let surplus_root = surplus.message_hash_tree_root(); + let pending_child = H256::repeat_byte(0xcc); + server.pending_blocks.insert( + surplus_root, + std::collections::HashSet::from([pending_child]), + ); + server + .pending_block_parents + .insert(pending_child, surplus_root); + + server.hold_block_for_parent_payload(surplus, ImportTimings::default()); + + assert!(server.pending_blocks.is_empty()); + assert!(server.pending_block_parents.is_empty()); + } + + /// A held child asks for its parent's envelope at once whatever its + /// source: `BlockSource::Sync` also marks by-root answers and cascade + /// releases, which no range batch is bringing an envelope for. + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_held_child_asks_for_the_envelope_at_once_whatever_its_source() { + for (source, asks) in [ + (Some(BlockSource::Sync), 1), + (Some(BlockSource::Gossip), 1), + (None, 1), + ] { + let case = Case::new("get_head", TIEBREAK); + let mut server = case.server(12); + let p2p = std::sync::Arc::new(crate::tests::RecordingP2P::default()); + server.p2p = Some(p2p.clone()); + let child = case.block(CHILD); + let parent = child.parent_root(); + + let timings = ImportTimings { + source, + ..ImportTimings::default() + }; + server.hold_block_for_parent_payload(child, timings); + + let fetches = p2p.fetches.lock().unwrap(); + assert_eq!(fetches.len(), asks, "source {source:?}"); + assert!(fetches.iter().all(|request| request.needs_envelope + && !request.needs_block + && request.block_root == parent)); + } + } + + /// A stored, unimported gloas block at `slot_offset` past the fixture + /// child's slot, whose bid names the valid envelope's payload, and that + /// envelope rewritten to belong to it. The stand-in for one block of a + /// range batch. + fn stored_block_with_matching_envelope( + server: &mut BlockChainServer, + slot_offset: u64, + ) -> (H256, gloas::SignedExecutionPayloadEnvelope) { + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let mut envelope = valid_envelope_for_full_parent(); + let mut block = case.block(FULL_CHILD); + let SignedBeaconBlock::Gloas(inner) = &mut block else { + unreachable!("the fixture block is a gloas block"); + }; + inner.message.slot += slot_offset; + let slot = inner.message.slot; + let bid = &mut inner.message.body.signed_execution_payload_bid.message; + bid.builder_index = envelope.message.builder_index; + bid.block_hash = envelope.message.payload.block_hash; + bid.prev_randao = envelope.message.payload.prev_randao; + bid.gas_limit = envelope.message.payload.gas_limit; + bid.execution_requests_root = envelope.message.execution_requests.hash_tree_root(); + bid.blob_kzg_commitments = Default::default(); + envelope.message.payload.slot_number = slot; + let block_root = block.message_hash_tree_root(); + envelope.message.beacon_block_root = block_root; + server + .store + .insert_pending_block(block_root, block) + .expect("pending block stores"); + (block_root, envelope) + } + + /// Range catch-up: a batch's later blocks are stored but unimported when + /// their envelopes arrive, all in one arrival slot. The per-arrival-slot + /// cap is for roots nobody has vouched for, so none of these may count + /// against it. + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn envelopes_for_stored_blocks_are_kept_outside_the_per_slot_cap() { + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let mut server = case.server(12); + let count = MAX_ENVELOPES_AWAITING_BLOCK_PER_SLOT as u64 * 3; + let mut roots = Vec::new(); + let mut envelopes = Vec::new(); + for offset in 0..count { + let (block_root, envelope) = stored_block_with_matching_envelope(&mut server, offset); + roots.push(block_root); + envelopes.push(envelope); + } + + for envelope in envelopes { + server.receive_envelope(envelope, 0).await; + server.settle_envelopes().await; + } + + for block_root in &roots { + assert!( + server.envelopes.awaiting_import.contains_key(block_root), + "every stored block keeps its envelope" + ); + } + assert!(server.envelopes.awaiting_block.is_empty()); + + // The cap still binds roots nothing is stored for. + let stranger = valid_envelope_for_full_parent(); + for byte in 1..=MAX_ENVELOPES_AWAITING_BLOCK_PER_SLOT as u8 + 2 { + let mut envelope = stranger.clone(); + envelope.message.beacon_block_root = H256::repeat_byte(byte); + server.receive_envelope(envelope, 0).await; + } + assert_eq!( + server.envelopes.awaiting_block.len(), + MAX_ENVELOPES_AWAITING_BLOCK_PER_SLOT + ); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_envelope_that_contradicts_the_stored_bid_is_dropped() { + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let mut server = case.server(12); + let (block_root, mut envelope) = stored_block_with_matching_envelope(&mut server, 0); + envelope.message.payload.gas_limit += 1; + + server.receive_envelope(envelope, 0).await; + + assert!(server.envelopes.awaiting_import.is_empty()); + assert!(!server.envelopes.awaiting_block.contains_key(&block_root)); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_envelope_held_for_a_stored_block_goes_with_the_block() { + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let mut server = case.server(12); + let (block_root, envelope) = stored_block_with_matching_envelope(&mut server, 0); + server.receive_envelope(envelope, 0).await; + assert!(server.envelopes.awaiting_import.contains_key(&block_root)); + + server.discard_pending_subtree(block_root); + + assert!(server.envelopes.awaiting_import.is_empty()); + } + + /// The whole catch-up chain imports on deliveries alone: the child arrives + /// ahead of its parent, then the parent, then the parent's envelope, and + /// the clock is not ticked in between. + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn catch_up_imports_the_chain_without_a_tick() { + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let mut server = case.server(24); + let p2p = std::sync::Arc::new(crate::tests::RecordingP2P::default()); + server.p2p = Some(p2p.clone()); + let time_before = server.store.ms_since_genesis(); + + let timings = || ImportTimings { + source: Some(BlockSource::Sync), + ..ImportTimings::default() + }; + server.on_block(case.block(FULL_CHILD), timings()).await; + server.on_block(case.block(FULL_PARENT), timings()).await; + assert!(server.envelopes.is_block_held(&root(FULL_CHILD))); + // Asked for at once, though the child came from sync. + assert!( + p2p.fetches + .lock() + .unwrap() + .iter() + .any(|request| request.needs_envelope && request.block_root == root(FULL_PARENT)) + ); + + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + server.settle_envelopes().await; + + assert!(server.store.has_state(&root(FULL_CHILD)).unwrap()); + assert!(server.envelopes.blocks_awaiting_parent_payload.is_empty()); + assert_eq!(server.store.ms_since_genesis(), time_before); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_held_envelope_for_the_parent_suppresses_the_immediate_ask() { + let case = Case::new("get_head", TIEBREAK); + let mut server = case.server(12); + let p2p = std::sync::Arc::new(crate::tests::RecordingP2P::default()); + server.p2p = Some(p2p.clone()); + let child = case.block(CHILD); + let envelope = case.envelope(PARENT_ENVELOPE); + server + .envelopes + .awaiting_import + .insert(child.parent_root(), vec![HeldEnvelope::new(envelope, 0, 0)]); + + server.hold_block_for_parent_payload(child, ImportTimings::default()); + + assert!(p2p.fetches.lock().unwrap().is_empty()); + } + + #[test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + fn awaiting_import_keeps_the_lowest_slots_when_over_its_cap() { + let mut queues = EnvelopeQueues::default(); + let envelope = Case::new("get_head", TIEBREAK).envelope(PARENT_ENVELOPE); + let total = MAX_ENVELOPES_AWAITING_IMPORT as u64 + 5; + for slot in 0..total { + let mut root_bytes = [0u8; 32]; + root_bytes[..8].copy_from_slice(&slot.to_be_bytes()); + queues.awaiting_import.insert( + H256(root_bytes), + vec![HeldEnvelope::new(envelope.clone(), slot, 0)], + ); + } + + queues.trim_awaiting_import(); + + let mut kept: Vec = queues + .awaiting_import + .values() + .flatten() + .map(|held| held.slot) + .collect(); + kept.sort(); + assert_eq!( + kept, + (0..MAX_ENVELOPES_AWAITING_IMPORT as u64).collect::>() + ); + } + + #[test] + fn each_slot_re_asks_for_the_envelope_of_a_parent_with_held_children() { + let (mut server, p2p) = + crate::tests::beacon_server_recording(beacon_store(GENESIS_TIME, 0)); + let parent = H256::repeat_byte(5); + server + .envelopes + .blocks_awaiting_parent_payload + .insert(parent, HashMap::from([(H256::repeat_byte(6), 3)])); + + server.redrive_missing_envelopes(); + server.redrive_missing_envelopes(); + + let fetches = p2p.fetches.lock().unwrap(); + assert_eq!(fetches.len(), 2, "one ask per call, so one per slot"); + assert!(fetches.iter().all(|request| request.needs_envelope + && !request.needs_block + && request.block_root == parent)); + } + + #[test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + fn a_parent_whose_envelope_is_already_held_is_not_asked_for() { + let (mut server, p2p) = + crate::tests::beacon_server_recording(beacon_store(GENESIS_TIME, 0)); + let envelope = Case::new("get_head", TIEBREAK).envelope(PARENT_ENVELOPE); + let parent = H256::repeat_byte(5); + server + .envelopes + .blocks_awaiting_parent_payload + .insert(parent, HashMap::from([(H256::repeat_byte(6), 3)])); + server + .envelopes + .awaiting_block + .insert(parent, vec![HeldEnvelope::new(envelope, 0, 0)]); + + server.redrive_missing_envelopes(); + + assert!(p2p.fetches.lock().unwrap().is_empty()); + } + + #[test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + fn an_unmatched_envelope_ages_out_after_the_ttl() { + // The store clock reads slot 10. + let mut server = beacon_server(crate::tests::beacon_store_at_slot_10()); + let envelope = Case::new("get_head", TIEBREAK).envelope(PARENT_ENVELOPE); + let fresh = H256::repeat_byte(1); + let stale = H256::repeat_byte(2); + let fresh_slot = 10 - ENVELOPE_AWAITING_BLOCK_TTL_SLOTS; + server.envelopes.awaiting_block.insert( + fresh, + vec![HeldEnvelope::new(envelope.clone(), fresh_slot, 0)], + ); + server + .envelopes + .awaiting_block + .insert(stale, vec![HeldEnvelope::new(envelope, fresh_slot - 1, 0)]); + + server.evict_envelope_queues_at_or_below_finality(); + + assert!(server.envelopes.awaiting_block.contains_key(&fresh)); + assert!(!server.envelopes.awaiting_block.contains_key(&stale)); + } + + #[test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + fn the_unmatched_queue_is_bounded_across_slots_by_the_ttl() { + // Fill every slot the TTL keeps to the per-slot cap and per-root cap, + // then read off the most the queue can hold. + let mut server = beacon_server(crate::tests::beacon_store_at_slot_10()); + let envelope = Case::new("get_head", TIEBREAK).envelope(PARENT_ENVELOPE); + for slot in 0..=10u64 { + for byte in 0..MAX_ENVELOPES_AWAITING_BLOCK_PER_SLOT as u8 { + let mut root_bytes = [0u8; 32]; + root_bytes[0] = slot as u8; + root_bytes[1] = byte; + server.envelopes.awaiting_block.insert( + H256(root_bytes), + vec![HeldEnvelope::new(envelope.clone(), slot, 0)], + ); + } + } + + server.evict_envelope_queues_at_or_below_finality(); + + let kept: usize = server.envelopes.awaiting_block.values().map(Vec::len).sum(); + assert_eq!( + kept, + (ENVELOPE_AWAITING_BLOCK_TTL_SLOTS as usize + 1) + * MAX_ENVELOPES_AWAITING_BLOCK_PER_SLOT + ); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_gossip_payload_attestation_message_is_applied_to_its_blocks_votes() { + let case = Case::new( + "on_payload_attestation_message", + "on_payload_attestation_message_valid", + ); + let block = "76bf14e70fc96a5a3442a46dff3bfb897d5568cdb4dc7e94419f625df2fcd9e5"; + let message_name = "payload_attestation_message_0xeaae966a9a1931c94efc5ca0280b35bac86166ebf8d47a936156bbeb1e7179d1"; + let mut server = case.server(12); + server + .on_block(case.block(block), ImportTimings::default()) + .await; + let message = gloas::PayloadAttestationMessage::from_ssz_bytes(&case.bytes(message_name)) + .expect("message decodes"); + let before = server.store.time_ms().unwrap(); + assert!( + server + .store + .payload_timeliness_vote(&root(block)) + .expect("votes exist") + .iter() + .all(Option::is_none) + ); + + server.apply_payload_attestation_message( + &message, + ðlambda_network_api::BlockArrival::now(), + ); + + assert!(server.store.time_ms().unwrap() >= before); + assert!( + server + .store + .payload_timeliness_vote(&root(block)) + .expect("votes exist") + .contains(&Some(true)) + ); + } + + // --------------------------------------------------------------------- + // The execution client's part: newPayloadV5 on envelope import and + // forkchoiceUpdated for a gloas head. + // --------------------------------------------------------------------- + + use ethlambda_engine::{EngineClient, JwtSecret}; + use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; + use tokio::net::TcpListener; + + /// What the mock execution client answers and has been asked. + #[derive(Default)] + struct MockState { + /// The `payloadStatus` object `engine_newPayloadV5` answers with. + new_payload: String, + /// The `payloadStatus` object `engine_forkchoiceUpdated*` answers with. + forkchoice: String, + /// Request bodies, in arrival order. + requests: Vec, + } + + fn status_json(status: &str, latest_valid_hash: Option) -> String { + let hash = latest_valid_hash.map_or("null".to_string(), |hash| format!("\"{hash:?}\"")); + format!(r#"{{"status":"{status}","latestValidHash":{hash},"validationError":null}}"#) + } + + /// A hand-rolled JSON-RPC server answering by method name, like + /// `ethlambda-engine`'s wire smoke tests but for many requests. + async fn mock_engine( + new_payload: String, + forkchoice: String, + ) -> (EngineClient, Arc>) { + let state = Arc::new(std::sync::Mutex::new(MockState { + new_payload, + forkchoice, + requests: Vec::new(), + })); + let listener = TcpListener::bind("127.0.0.1:0") + .await + .expect("a loopback port"); + let addr = listener.local_addr().expect("bound address"); + let shared = Arc::clone(&state); + tokio::spawn(async move { + loop { + let Ok((mut socket, _)) = listener.accept().await else { + return; + }; + let shared = Arc::clone(&shared); + tokio::spawn(async move { + let mut pending: Vec = Vec::new(); + loop { + // One request: headers up to the blank line, then + // `Content-Length` bytes of body. + let (head_end, body_len) = loop { + if let Some(end) = + pending.windows(4).position(|window| window == b"\r\n\r\n") + { + let head = String::from_utf8_lossy(&pending[..end]).to_lowercase(); + let len = head + .lines() + .find_map(|line| line.strip_prefix("content-length:")) + .and_then(|value| value.trim().parse::().ok()) + .unwrap_or(0); + break (end + 4, len); + } + let mut chunk = [0u8; 16 * 1024]; + match socket.read(&mut chunk).await { + Ok(0) | Err(_) => return, + Ok(read) => pending.extend_from_slice(&chunk[..read]), + } + }; + while pending.len() < head_end + body_len { + let mut chunk = [0u8; 16 * 1024]; + match socket.read(&mut chunk).await { + Ok(0) | Err(_) => return, + Ok(read) => pending.extend_from_slice(&chunk[..read]), + } + } + let body = String::from_utf8_lossy(&pending[head_end..head_end + body_len]) + .to_string(); + pending.drain(..head_end + body_len); + + let result = { + let mut state = shared.lock().unwrap(); + state.requests.push(body.clone()); + if body.contains("engine_newPayloadV5") { + state.new_payload.clone() + } else { + state.forkchoice.clone() + } + }; + let result = if body.contains("engine_newPayloadV5") { + result + } else { + format!(r#"{{"payloadStatus":{result},"payloadId":null}}"#) + }; + let payload = if result == "ERROR" { + r#"{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"mock failure"}}"#.to_string() + } else { + format!(r#"{{"jsonrpc":"2.0","id":1,"result":{result}}}"#) + }; + let response = format!( + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n{payload}", + payload.len() + ); + if socket.write_all(response.as_bytes()).await.is_err() { + return; + } + } + }); + } + }); + let client = EngineClient::new(format!("http://{addr}"), JwtSecret::new([0x0f; 32])) + .expect("a client"); + (client, state) + } + + fn methods(state: &Arc>) -> Vec { + state + .lock() + .unwrap() + .requests + .iter() + .map(|body| { + let rest = body.split("\"method\":\"").nth(1).expect("a method"); + rest.split('"').next().expect("closing quote").to_string() + }) + .collect() + } + + /// The `on_execution_payload_envelope` chain: a parent (FULL once its + /// envelope is applied) and a child that builds on the FULL parent, held + /// until the envelope verifies. + async fn server_with_held_child(client: EngineClient) -> BlockChainServer { + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let mut server = case.server(12); + server.engine = Some(client); + server + .on_block(case.block(FULL_PARENT), ImportTimings::default()) + .await; + fork_choice::on_tick(&mut server.store, 24, &fixture_config()); + server + .on_block(case.block(FULL_CHILD), ImportTimings::default()) + .await; + assert!(server.envelopes.is_block_held(&root(FULL_CHILD))); + server + } + + fn payload_hash(envelope: &gloas::SignedExecutionPayloadEnvelope) -> H256 { + envelope.message.payload.block_hash + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_valid_verdict_applies_the_envelope_and_records_valid() { + let (client, mock) = + mock_engine(status_json("VALID", None), status_json("SYNCING", None)).await; + let mut server = server_with_held_child(client).await; + let envelope = valid_envelope_for_full_parent(); + + server.receive_envelope(envelope.clone(), 0).await; + server.settle_envelopes().await; + + assert!(server.store.has_verified_payload(&root(FULL_PARENT))); + assert_eq!( + server.store.beacon_block_payload_status(root(FULL_PARENT)), + PayloadStatusEnum::Valid + ); + assert_eq!( + server.store.beacon_el_block_hash(root(FULL_PARENT)), + Some(payload_hash(&envelope)) + ); + assert!(server.store.has_state(&root(FULL_CHILD)).unwrap()); + let methods = methods(&mock); + assert_eq!( + methods + .iter() + .filter(|m| *m == "engine_newPayloadV5") + .count(), + 1 + ); + // The head is gloas, so forkchoiceUpdated is V4. + assert!(methods.iter().any(|m| m == "engine_forkchoiceUpdatedV4")); + assert!(!methods.iter().any(|m| m == "engine_forkchoiceUpdatedV3")); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_syncing_verdict_applies_the_envelope_as_not_validated() { + let (client, _mock) = + mock_engine(status_json("SYNCING", None), status_json("SYNCING", None)).await; + let mut server = server_with_held_child(client).await; + + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + server.settle_envelopes().await; + + assert!(server.store.has_verified_payload(&root(FULL_PARENT))); + let status = server.store.beacon_block_payload_status(root(FULL_PARENT)); + assert_eq!(status, PayloadStatusEnum::Syncing); + assert!(status.is_not_validated()); + assert!(server.store.has_state(&root(FULL_CHILD)).unwrap()); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_invalid_verdict_leaves_no_full_node_and_drops_the_full_child() { + // A non-null `latestValidHash`: the payload executed and failed. + let (client, mock) = mock_engine( + status_json("INVALID", Some(H256::repeat_byte(0x11))), + status_json("SYNCING", None), + ) + .await; + let mut server = server_with_held_child(client).await; + + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + server.settle_envelopes().await; + + assert!(!server.store.has_verified_payload(&root(FULL_PARENT))); + assert_eq!( + server.store.beacon_block_payload_status(root(FULL_PARENT)), + PayloadStatusEnum::Invalid + ); + // The child built on the FULL node, so it is gone; the parent block and + // its EMPTY branch stay. + assert!(!server.envelopes.is_block_held(&root(FULL_CHILD))); + assert!(!server.store.has_state(&root(FULL_CHILD)).unwrap()); + assert!(server.store.block_index().contains_key(&root(FULL_PARENT))); + assert_eq!(head(&server), (root(FULL_PARENT), PayloadStatus::Empty)); + + // A second delivery is not asked about again, and a later block that + // builds on the FULL node is dropped instead of held. + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + server + .on_block(case.block(FULL_CHILD), ImportTimings::default()) + .await; + assert!(!server.envelopes.is_block_held(&root(FULL_CHILD))); + let asked = methods(&mock) + .iter() + .filter(|m| *m == "engine_newPayloadV5") + .count(); + assert_eq!(asked, 1); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn no_verdict_does_not_apply_the_envelope() { + // Port 1 has nothing listening, so every attempt is refused at once. + let client = EngineClient::new("http://127.0.0.1:1".to_string(), JwtSecret::new([0u8; 32])) + .expect("client builds"); + let mut server = server_with_held_child(client).await; + + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + + assert!(!server.store.has_verified_payload(&root(FULL_PARENT))); + assert!(server.envelopes.is_block_held(&root(FULL_CHILD))); + } + + /// The fixture payloads all hash to zero, which `forkchoiceUpdated` rightly + /// treats as "nothing to say"; give `root`'s bid and cached payload a real + /// hash so the call can be observed. + fn give_payload_a_hash(server: &mut BlockChainServer, root: H256, hash: H256) { + let mut block = server + .store + .get_signed_block(&root) + .unwrap() + .expect("block stored"); + let slot = block.slot(); + match &mut block { + SignedBeaconBlock::Gloas(inner) => { + inner + .message + .body + .signed_execution_payload_bid + .message + .block_hash = hash; + } + _ => panic!("a gloas block"), + } + server + .store + .insert_pending_block(root, block) + .expect("overwrite"); + server.store.insert_beacon_el_block_hash(root, slot, hash); + } + + /// A server whose only block is `FULL_PARENT`, its envelope applied with + /// `new_payload`'s verdict, then given a nonzero payload hash and made the + /// FULL head. + async fn full_head_server( + new_payload: &str, + forkchoice: &str, + ) -> (BlockChainServer, Arc>, H256) { + let (client, mock) = mock_engine( + status_json(new_payload, None), + status_json(forkchoice, None), + ) + .await; + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let mut server = case.server(12); + server.engine = Some(client); + server + .on_block(case.block(FULL_PARENT), ImportTimings::default()) + .await; + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + server.settle_envelopes().await; + assert!(server.store.has_verified_payload(&root(FULL_PARENT))); + + let hash = H256::repeat_byte(0x42); + give_payload_a_hash(&mut server, root(FULL_PARENT), hash); + fork_choice::get_head(&mut server.store, &fixture_config()).expect("head"); + server + .store + .set_head_payload_status(root(FULL_PARENT), PayloadStatus::Full); + (server, mock, hash) + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn the_forkchoice_hashes_follow_the_head_node_and_the_checkpoints_bids() { + let (server, _mock, hash) = full_head_server("VALID", "SYNCING").await; + let parent = server + .store + .get_signed_block(&root(FULL_PARENT)) + .unwrap() + .unwrap(); + let bid = crate::beacon_payloads::bid_of(&parent).expect("gloas block"); + let anchor_root = server.store.beacon_finalized_checkpoint().root; + let anchor = server + .store + .get_signed_block(&anchor_root) + .unwrap() + .unwrap(); + let anchor_parent_hash = crate::beacon_payloads::bid_of(&anchor) + .expect("gloas anchor") + .parent_block_hash; + + // A FULL head names the revealed payload; finalized and safe are the + // checkpoint blocks' bids' parent hashes. + let plan = crate::beacon_payloads::forkchoice_plan(&server.store, root(FULL_PARENT)) + .expect("a plan"); + assert!(plan.gloas); + assert_eq!(plan.state.head_block_hash, hash); + assert_eq!(plan.state.finalized_block_hash, anchor_parent_hash); + assert_eq!(plan.state.safe_block_hash, anchor_parent_hash); + + // An EMPTY head names the payload before it. + server + .store + .set_head_payload_status(root(FULL_PARENT), PayloadStatus::Empty); + let plan = crate::beacon_payloads::forkchoice_plan(&server.store, root(FULL_PARENT)) + .expect("a plan"); + assert_eq!(plan.state.head_block_hash, bid.parent_block_hash); + + // No status for the head yet: nothing to say. + server + .store + .set_head_payload_status(H256::repeat_byte(1), PayloadStatus::Full); + assert!( + crate::beacon_payloads::forkchoice_plan(&server.store, root(FULL_PARENT)).is_none() + ); + } + + #[test] + fn a_pre_gloas_head_keeps_its_own_hash_and_uses_forkchoice_v3() { + let mut store = beacon_store(GENESIS_TIME, 0); + let block = crate::tests::fulu_block(H256::ZERO, 1, 0); + let block_root = block.message_hash_tree_root(); + store + .insert_pending_block(block_root, block) + .expect("insert"); + store.insert_beacon_el_block_hash(block_root, 1, H256::repeat_byte(7)); + + let plan = crate::beacon_payloads::forkchoice_plan(&store, block_root).expect("a plan"); + + assert!(!plan.gloas); + assert_eq!(plan.state.head_block_hash, H256::repeat_byte(7)); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_gloas_head_sends_forkchoice_v4_with_the_custody_columns() { + let (mut server, mock, hash) = full_head_server("VALID", "SYNCING").await; + server.custody_columns = vec![0, 9]; + + server.notify_forkchoice_updated().await; + + let last = mock + .lock() + .unwrap() + .requests + .last() + .cloned() + .expect("a request"); + assert!(last.contains("engine_forkchoiceUpdatedV4")); + assert!(last.contains(&format!("{hash:?}"))); + // Columns 0 and 9 as a little-endian bitvector. + assert!(last.contains("0x01020000000000000000000000000000")); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_valid_forkchoice_response_promotes_optimistic_payloads() { + let (mut server, mock, _hash) = full_head_server("SYNCING", "SYNCING").await; + assert_eq!( + server.store.beacon_block_payload_status(root(FULL_PARENT)), + PayloadStatusEnum::Syncing + ); + + // Still syncing: nothing changes. + server.notify_forkchoice_updated().await; + assert_eq!( + server.store.beacon_block_payload_status(root(FULL_PARENT)), + PayloadStatusEnum::Syncing + ); + + mock.lock().unwrap().forkchoice = status_json("VALID", None); + server.notify_forkchoice_updated().await; + + assert_eq!( + server.store.beacon_block_payload_status(root(FULL_PARENT)), + PayloadStatusEnum::Valid + ); + assert!(!server.store.has_unvalidated_block_payloads()); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_invalid_forkchoice_response_unverifies_the_payload_and_keeps_the_block() { + let (mut server, _mock, _hash) = full_head_server("SYNCING", "INVALID").await; + + server.notify_forkchoice_updated().await; + + assert!(!server.store.has_verified_payload(&root(FULL_PARENT))); + assert_eq!( + server.store.beacon_block_payload_status(root(FULL_PARENT)), + PayloadStatusEnum::Invalid + ); + assert!(server.store.block_index().contains_key(&root(FULL_PARENT))); + assert_eq!(head(&server), (root(FULL_PARENT), PayloadStatus::Empty)); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_invalid_forkchoice_response_removes_a_child_built_on_the_full_payload() { + let (mut server, _mock, hash) = full_head_server("SYNCING", "INVALID").await; + // A child whose bid names the parent's payload as its own parent: the + // fixture child, retargeted at the hash this test gave the payload. + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let mut child = case.block(FULL_CHILD); + let child_root = child.message_hash_tree_root(); + match &mut child { + SignedBeaconBlock::Gloas(inner) => { + inner + .message + .body + .signed_execution_payload_bid + .message + .parent_block_hash = hash; + } + _ => panic!("a gloas block"), + } + server + .store + .insert_pending_block(child_root, child) + .expect("insert"); + let child_slot = server + .store + .get_signed_block(&child_root) + .unwrap() + .unwrap() + .slot(); + server + .store + .insert_live_chain_entry(child_slot, child_root, root(FULL_PARENT)); + assert!(server.store.block_index().contains_key(&child_root)); + + server.notify_forkchoice_updated().await; + + assert!(!server.store.block_index().contains_key(&child_root)); + } + + fn count(mock: &Arc>, method: &str) -> usize { + methods(mock).iter().filter(|m| *m == method).count() + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn the_engine_is_not_asked_about_an_envelope_that_fails_consensus_checks() { + let (client, mock) = + mock_engine(status_json("VALID", None), status_json("SYNCING", None)).await; + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let bad_envelope = "73c5948d041ccb171b9331c0eac4a3bb8cdcf01994916fee4c7493e63e890559"; + let mut server = case.server(12); + server.engine = Some(client); + server + .on_block(case.block(FULL_PARENT), ImportTimings::default()) + .await; + + server + .receive_envelope(case.envelope(bad_envelope), 0) + .await; + + assert_eq!(count(&mock, "engine_newPayloadV5"), 0); + assert!(!server.store.has_verified_payload(&root(FULL_PARENT))); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_invalid_block_hash_rejects_only_that_envelope() { + let (client, mock) = mock_engine( + status_json("INVALID_BLOCK_HASH", None), + status_json("SYNCING", None), + ) + .await; + let mut server = server_with_held_child(client).await; + + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + server.settle_envelopes().await; + + // The root's payload is not condemned: another envelope may still + // reveal it, and the child keeps waiting for one. + assert!(!server.store.has_verified_payload(&root(FULL_PARENT))); + assert!( + !server + .store + .beacon_block_payload_status(root(FULL_PARENT)) + .is_invalidated() + ); + assert!(server.envelopes.is_block_held(&root(FULL_CHILD))); + + // The same envelope is not asked about again. + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + assert_eq!(count(&mock, "engine_newPayloadV5"), 1); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_engine_error_holds_the_envelope_and_asks_again_once_a_slot() { + let (client, mock) = mock_engine("ERROR".to_string(), status_json("SYNCING", None)).await; + let mut server = server_with_held_child(client).await; + let p2p = Arc::new(crate::tests::RecordingP2P::default()); + server.p2p = Some(p2p.clone()); + let fetches_before = p2p.fetches.lock().unwrap().len(); + + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + assert!( + server + .envelopes + .awaiting_engine + .contains_key(&root(FULL_PARENT)) + ); + assert_eq!(count(&mock, "engine_newPayloadV5"), 1); + + // The same copy again, and the slot's other redrives, add no asks. + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + server.redrive_missing_envelopes(); + assert_eq!(count(&mock, "engine_newPayloadV5"), 1); + + // One ask per slot while it keeps failing, and no fetch at all. + for asks in 2..=3 { + server.redrive_envelopes_awaiting_engine(); + server.settle_envelopes().await; + assert_eq!(count(&mock, "engine_newPayloadV5"), asks); + } + assert_eq!(p2p.fetches.lock().unwrap().len(), fetches_before); + assert!(!server.store.has_verified_payload(&root(FULL_PARENT))); + + // The client recovers: the next redrive applies it. + mock.lock().unwrap().new_payload = status_json("VALID", None); + server.redrive_envelopes_awaiting_engine(); + server.settle_envelopes().await; + assert!(server.store.has_verified_payload(&root(FULL_PARENT))); + assert!(server.envelopes.awaiting_engine.is_empty()); + assert!(server.store.has_state(&root(FULL_CHILD)).unwrap()); + } + + /// A chain `q <- p <- r` of gloas blocks above the anchor, each with a + /// verified payload and the cached hash, where `r` built on `p`'s EMPTY + /// node: its bid's parent hash is `q`'s payload, not `p`'s. + /// + /// Returns `(server, [q, p, r], [hq, hp, hr])`. Blocks are the fixture + /// parent with its slot, parent and bid hashes rewritten, stored under + /// made-up roots, which is all the walks read. + fn skipped_payload_chain() -> (BlockChainServer, [H256; 3], [H256; 3]) { + let case = Case::new("on_execution_payload_envelope", FULL_CHILD_CASE); + let mut server = case.server(12); + let anchor = server.store.latest_finalized().unwrap(); + let roots = [ + H256::repeat_byte(0xa1), + H256::repeat_byte(0xa2), + H256::repeat_byte(0xa3), + ]; + let hashes = [ + H256::repeat_byte(0xb1), + H256::repeat_byte(0xb2), + H256::repeat_byte(0xb3), + ]; + // (parent root, bid.parent_block_hash) per block. + let links = [ + (anchor.root, H256::repeat_byte(0xb0)), + (roots[0], hashes[0]), + (roots[1], hashes[0]), + ]; + for i in 0..3 { + let slot = anchor.slot + 1 + i as u64; + let mut block = case.block(FULL_PARENT); + match &mut block { + SignedBeaconBlock::Gloas(inner) => { + inner.message.slot = slot; + inner.message.parent_root = links[i].0; + let bid = &mut inner.message.body.signed_execution_payload_bid.message; + bid.block_hash = hashes[i]; + bid.parent_block_hash = links[i].1; + } + _ => panic!("a gloas block"), + } + server + .store + .insert_pending_block(roots[i], block) + .expect("insert"); + server + .store + .insert_live_chain_entry(slot, roots[i], links[i].0); + let mut envelope = valid_envelope_for_full_parent(); + envelope.message.beacon_block_root = roots[i]; + server.store.insert_verified_payload(slot, &envelope); + server + .store + .insert_beacon_el_block_hash(roots[i], slot, hashes[i]); + server.store.insert_beacon_block_payload_status( + roots[i], + slot, + PayloadStatusEnum::Syncing, + ); + } + (server, roots, hashes) + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_latest_valid_hash_does_not_reach_a_payload_the_block_skipped() { + let (mut server, [q, p, r], [hq, _hp, _hr]) = skipped_payload_chain(); + + // r's EL parent is q: p was skipped (r built on p's EMPTY node). + assert_eq!( + crate::beacon_payloads::resolve_invalid_payload(&server.store, r, Some(hq)), + r + ); + assert_eq!( + crate::beacon_payloads::resolve_invalid_payload(&server.store, r, None), + r + ); + // A hash that is not on the chain reads as null. + assert_eq!( + crate::beacon_payloads::resolve_invalid_payload( + &server.store, + r, + Some(H256::repeat_byte(0xee)) + ), + r + ); + // All zeroes: the earliest block on the chain, which is q, not p. + assert_eq!( + crate::beacon_payloads::resolve_invalid_payload(&server.store, r, Some(H256::ZERO)), + q + ); + + server.condemn_payload_chain(r, Some(hq)); + + assert!(!server.store.has_verified_payload(&r)); + assert!( + server.store.has_verified_payload(&p), + "p's payload survives" + ); + assert!(server.store.has_verified_payload(&q)); + assert_eq!( + server.store.beacon_block_payload_status(r), + PayloadStatusEnum::Invalid + ); + assert_eq!( + server.store.beacon_block_payload_status(p), + PayloadStatusEnum::Syncing + ); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn validation_does_not_promote_a_payload_the_block_skipped() { + let (mut server, [q, p, r], _hashes) = skipped_payload_chain(); + + crate::beacon_payloads::mark_payloads_validated(&mut server.store, r); + + assert_eq!( + server.store.beacon_block_payload_status(r), + PayloadStatusEnum::Valid + ); + assert_eq!( + server.store.beacon_block_payload_status(q), + PayloadStatusEnum::Valid + ); + assert_eq!( + server.store.beacon_block_payload_status(p), + PayloadStatusEnum::Syncing, + "r never built on p's payload" + ); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_invalid_payload_takes_its_full_child_and_spares_the_empty_one() { + let (mut server, [q, p, r], [_hq, hp, _hr]) = skipped_payload_chain(); + // Two children of p: `full` names p's payload as its parent's, `empty` + // names the payload before it. Condemning p's payload takes only `full`. + let slot = server.store.get_signed_block(&p).unwrap().unwrap().slot() + 1; + let full = H256::repeat_byte(0xc1); + let empty = H256::repeat_byte(0xc2); + for (root, parent_hash) in [(full, hp), (empty, H256::repeat_byte(0xb1))] { + let mut block = server.store.get_signed_block(&r).unwrap().unwrap(); + match &mut block { + SignedBeaconBlock::Gloas(inner) => { + inner.message.slot = slot; + inner.message.parent_root = p; + inner + .message + .body + .signed_execution_payload_bid + .message + .parent_block_hash = parent_hash; + } + _ => panic!("a gloas block"), + } + server + .store + .insert_pending_block(root, block) + .expect("insert"); + server.store.insert_live_chain_entry(slot, root, p); + } + + server.condemn_payload_chain(p, None); + + let index = server.store.block_index(); + assert!(!server.store.has_verified_payload(&p)); + assert!(index.contains_key(&p), "the block stays"); + assert!(!index.contains_key(&full), "the FULL child goes"); + assert!(index.contains_key(&empty), "the EMPTY child stays"); + assert!(server.store.has_verified_payload(&q)); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_restored_payload_is_seeded_with_a_verdict_a_valid_answer_can_promote() { + let (mut server, [q, p, r], _hashes) = skipped_payload_chain(); + // A restart empties the status map; the verified set is on disk. + for root in [q, p, r] { + server + .store + .insert_beacon_block_payload_status(root, 0, PayloadStatusEnum::Valid); + } + let mut restarted = server; + + crate::beacon_payloads::seed_resumed_payload_statuses(&mut restarted.store, true); + assert_eq!( + restarted.store.beacon_block_payload_status(r), + PayloadStatusEnum::Syncing + ); + assert!(restarted.store.has_unvalidated_block_payloads()); + crate::beacon_payloads::mark_payloads_validated(&mut restarted.store, r); + assert_eq!( + restarted.store.beacon_block_payload_status(r), + PayloadStatusEnum::Valid + ); + + crate::beacon_payloads::seed_resumed_payload_statuses(&mut restarted.store, false); + assert_eq!( + restarted.store.beacon_block_payload_status(p), + PayloadStatusEnum::Valid + ); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn an_invalid_verdict_with_a_null_latest_valid_hash_rejects_only_that_envelope() { + // How ethrex reports a payload whose contents do not hash to its block + // hash: INVALID, `latestValidHash` null. + let (client, mock) = + mock_engine(status_json("INVALID", None), status_json("SYNCING", None)).await; + let mut server = server_with_held_child(client).await; + + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + server.settle_envelopes().await; + + assert!(!server.store.has_verified_payload(&root(FULL_PARENT))); + assert!( + !server + .store + .beacon_block_payload_status(root(FULL_PARENT)) + .is_invalidated() + ); + assert!(server.envelopes.is_block_held(&root(FULL_CHILD))); + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + assert_eq!(count(&mock, "engine_newPayloadV5"), 1); + } + + #[tokio::test] + #[cfg_attr( + not(feature = "beacon-spec-tests"), + ignore = "needs the fork choice fixtures; run `make consensus-spec-tests`" + )] + async fn a_down_engine_is_probed_once_per_redrive_however_many_envelopes_wait() { + let (client, mock) = mock_engine("ERROR".to_string(), status_json("SYNCING", None)).await; + let mut server = server_with_held_child(client).await; + server + .receive_envelope(valid_envelope_for_full_parent(), 0) + .await; + // More envelopes waiting, for blocks later than the real one. + for byte in 1..=4u8 { + let mut envelope = valid_envelope_for_full_parent(); + envelope.message.beacon_block_root = H256::repeat_byte(byte); + server.envelopes.awaiting_engine.insert( + H256::repeat_byte(byte), + HeldEnvelope::new(envelope, 1_000 + u64::from(byte), 0), + ); + } + assert_eq!(server.envelopes.awaiting_engine.len(), 5); + assert_eq!(count(&mock, "engine_newPayloadV5"), 1); + + for asks in 2..=3 { + server.redrive_envelopes_awaiting_engine(); + server.settle_envelopes().await; + assert_eq!(count(&mock, "engine_newPayloadV5"), asks); + assert_eq!(server.envelopes.awaiting_engine.len(), 5); + } + + // The engine answers: the oldest is applied and the rest are released + // in the same redrive. + mock.lock().unwrap().new_payload = status_json("VALID", None); + server.redrive_envelopes_awaiting_engine(); + server.settle_envelopes().await; + assert!(server.store.has_verified_payload(&root(FULL_PARENT))); + assert!(server.envelopes.awaiting_engine.is_empty()); + } +} diff --git a/crates/blockchain/src/beacon_payloads.rs b/crates/blockchain/src/beacon_payloads.rs new file mode 100644 index 000000000..a935fb240 --- /dev/null +++ b/crates/blockchain/src/beacon_payloads.rs @@ -0,0 +1,426 @@ +//! Execution-layer bookkeeping for gloas payloads. +//! +//! Before gloas a block carries its payload, so the execution client's verdict +//! on it is the block's and the optimistic-sync machinery in +//! `ethlambda_state_transition::beacon::fork_choice` works on blocks. In gloas +//! the payload arrives later, in an envelope, and fork choice weighs a FULL and +//! an EMPTY node per block. A payload the execution client rejects must take +//! only its FULL node (and whatever builds on it) with it; the block and its +//! EMPTY branch stay viable. This module is that second granularity: +//! +//! ```text +//! envelope ──► newPayloadV5 ──► VALID apply, status Valid +//! ├─► SYNCING apply, status Syncing (NOT_VALIDATED) +//! └─► INVALID do not apply, status Invalid, +//! drop children built on the FULL node +//! +//! forkchoiceUpdated(V4) ──► VALID Syncing payloads on the head's ancestry -> Valid +//! └─► INVALID first invalid payload after latestValidHash: +//! unverify it, drop its FULL children +//! ``` +//! +//! # Which hash names what +//! +//! A gloas block's EL hash depends on the node: its FULL node is the payload +//! the envelope revealed (the bid's `block_hash`); its EMPTY node is the last +//! payload before it (the bid's `parent_block_hash`). `SignedBeaconBlock:: +//! execution_block_hash` is `None` for gloas on purpose, so the helpers here +//! read the bid. + +use std::collections::HashMap; + +use ethlambda_engine::ForkchoiceStateV1; +use ethlambda_state_transition::beacon::fork_choice::{ + self, PayloadStatus, PayloadStatusEnum, PayloadValidity, +}; +use ethlambda_storage::Store; +use ethlambda_types::ShortRoot; +use ethlambda_types::beacon::containers::{SignedBeaconBlock, gloas}; +use ethlambda_types::beacon::primitives::ExecutionBlockHash; +use ethlambda_types::beacon::signing::compute_start_slot_at_epoch; +use ethlambda_types::primitives::H256; +use tracing::{debug, error, warn}; + +use crate::BlockChainServer; + +/// The bid a gloas block carries, or `None` for any other fork. +pub(crate) fn bid_of(block: &SignedBeaconBlock) -> Option<&gloas::ExecutionPayloadBid> { + match block { + SignedBeaconBlock::Gloas(inner) => { + Some(&inner.message.body.signed_execution_payload_bid.message) + } + SignedBeaconBlock::Phase0(_) + | SignedBeaconBlock::Altair(_) + | SignedBeaconBlock::Bellatrix(_) + | SignedBeaconBlock::Capella(_) + | SignedBeaconBlock::Deneb(_) + | SignedBeaconBlock::Electra(_) + | SignedBeaconBlock::Fulu(_) + | SignedBeaconBlock::Lean(_) => None, + } +} + +/// The execution hash of the payload `root` carries on the chain, or `None` if +/// it carries none: a pre-gloas block's own payload (cached at import), or a +/// gloas block's verified payload (cached when the envelope is applied, and +/// read back off the bid for a payload verified before a restart emptied the +/// cache). +pub(crate) fn payload_hash_of(store: &Store, root: H256) -> Option { + if let Some(hash) = store.beacon_el_block_hash(root) { + return Some(hash); + } + if !store.has_verified_payload(&root) { + return None; + } + let block = store.get_signed_block(&root).ok().flatten()?; + bid_of(&block).map(|bid| bid.block_hash) +} + +/// The execution hash a checkpoint root stands for in `forkchoiceUpdated`. +/// +/// A gloas block is `bid.parent_block_hash` (gloas `fork-choice.md`, "Modified +/// `notify_forkchoice_updated`": the payload a checkpoint block builds on is +/// the one a node can know to be final, since the block's own payload may +/// never be revealed). A pre-gloas block keeps its own payload's hash. +pub fn checkpoint_hash(store: &Store, root: H256) -> ExecutionBlockHash { + if let Some(block) = store.get_signed_block(&root).ok().flatten() + && let Some(bid) = bid_of(&block) + { + return bid.parent_block_hash; + } + store.beacon_el_block_hash(root).unwrap_or(H256::ZERO) +} + +/// What `forkchoiceUpdated` should carry for the current head, or `None` when +/// there is nothing to say yet. +pub(crate) struct ForkchoicePlan { + pub(crate) state: ForkchoiceStateV1, + /// Whether the head block is gloas or later, which selects + /// `engine_forkchoiceUpdatedV4` (with custody columns) over V3. + pub(crate) gloas: bool, +} + +/// Builds the `forkchoiceUpdated` payload for `head_root`. +/// +/// Head: a gloas head's FULL node is the payload its envelope revealed +/// (`bid.block_hash`), its EMPTY node the payload before it +/// (`bid.parent_block_hash`). `None` while the head's payload status has not +/// been computed (a gloas head is ambiguous without it), and while the head +/// hash is zero, as before. +/// +/// Finalized: the finalized block's `bid.parent_block_hash` (spec). Safe: the +/// spec's `get_safe_execution_block_hash` is fast-confirmation based, which +/// this follower does not run, so the justified checkpoint block's +/// `bid.parent_block_hash` stands in (see `docs/spec_deviations.md`). A +/// pre-gloas checkpoint block, including one under a gloas head, uses its own +/// payload hash. +pub(crate) fn forkchoice_plan(store: &Store, head_root: H256) -> Option { + let head_block = store.get_signed_block(&head_root).ok().flatten(); + let head_bid = head_block.as_ref().and_then(bid_of); + let head_block_hash = match head_bid { + Some(bid) => match store.head_payload_status()? { + PayloadStatus::Full => bid.block_hash, + PayloadStatus::Empty | PayloadStatus::Pending => bid.parent_block_hash, + }, + None => store.beacon_el_block_hash(head_root).unwrap_or(H256::ZERO), + }; + if head_block_hash.is_zero() { + return None; + } + let state = ForkchoiceStateV1 { + head_block_hash, + safe_block_hash: checkpoint_hash(store, store.beacon_justified_checkpoint().root), + finalized_block_hash: checkpoint_hash(store, store.beacon_finalized_checkpoint().root), + }; + Some(ForkchoicePlan { + state, + gloas: head_bid.is_some(), + }) +} + +/// The slot at or below which a block is final: the EL chain walks stop there. +fn finality_floor(store: &Store) -> (u64, H256) { + let finalized = store.beacon_finalized_checkpoint(); + (compute_start_slot_at_epoch(finalized.epoch), finalized.root) +} + +/// The block whose payload `root`'s payload builds on in the execution layer, +/// or `None` when it is below what this store knows (an anchor's parent) or +/// null. +/// +/// A gloas block names it in its bid: the ancestor whose payload hash is +/// `bid.parent_block_hash`, skipping every block between that built on an +/// EMPTY node (their payloads are not in this chain). A pre-gloas block +/// builds on its parent's payload. The one rule both +/// [`resolve_invalid_payload`] and [`mark_payloads_validated`] walk by, since +/// treating every verified ancestor as an EL ancestor would condemn or +/// validate payloads the block never built on. +fn el_parent(store: &Store, root: H256) -> Option { + let block = store.get_signed_block(&root).ok().flatten()?; + let (_slot, parent) = store.block_entry(&root)?; + let Some(bid) = bid_of(&block) else { + return payload_hash_of(store, parent).map(|_| parent); + }; + if bid.parent_block_hash.is_zero() { + return None; + } + let (floor_slot, floor_root) = finality_floor(store); + let mut cursor = parent; + loop { + if payload_hash_of(store, cursor) == Some(bid.parent_block_hash) { + return Some(cursor); + } + // Nothing below finality is searched: the walk would only scan the + // whole window for an answer no verdict can act on. + let (slot, next) = store.block_entry(&cursor)?; + if cursor == floor_root || slot <= floor_slot { + return None; + } + cursor = next; + } +} + +/// `start` and then each EL parent in turn, ending at the first block at or +/// below finality (included) or the first with no EL parent. +fn el_chain(store: &Store, start: H256) -> Vec { + let (floor_slot, floor_root) = finality_floor(store); + let mut chain = vec![start]; + let mut cursor = start; + while let Some(parent) = el_parent(store, cursor) { + chain.push(parent); + let at_floor = store + .block_entry(&parent) + .is_some_and(|(slot, _)| slot <= floor_slot); + if parent == floor_root || at_floor { + break; + } + cursor = parent; + } + chain +} + +/// The nearest ancestor-or-self of `from` whose payload hash is `hash`: the +/// block a `forkchoiceUpdated` verdict about `hash` is about. +pub(crate) fn payload_holder(store: &Store, from: H256, hash: ExecutionBlockHash) -> Option { + let (floor_slot, floor_root) = finality_floor(store); + let mut cursor = from; + loop { + if payload_hash_of(store, cursor) == Some(hash) { + return Some(cursor); + } + let (slot, next) = store.block_entry(&cursor)?; + if cursor == floor_root || slot <= floor_slot { + return None; + } + cursor = next; + } +} + +/// The block whose payload is the first invalid one on `holder`'s EL chain, +/// per `optimistic-sync.md`'s `latestValidHash` table, reading payloads rather +/// than blocks. +/// +/// [`fork_choice::resolve_invalid_block`] walks parents while each carries a +/// cached payload hash, which stops at the first gloas block whose payload was +/// never revealed. Here the walk follows [`el_parent`], so a payload applied +/// to an ancestor that `holder` did not build on (it built on that ancestor's +/// EMPTY node) is never reached, and an `INVALID` verdict about `holder` +/// cannot delete it. +/// +/// | `latest_valid_hash` | result | +/// |---|---| +/// | a payload hash found on this chain | the next block after it on the chain | +/// | all zeroes | the earliest block on the chain | +/// | `None`, or a hash not on this chain | `holder` itself | +pub(crate) fn resolve_invalid_payload( + store: &Store, + holder: H256, + latest_valid_hash: Option, +) -> H256 { + let Some(latest_valid_hash) = latest_valid_hash else { + return holder; + }; + let chain = el_chain(store, holder); + for pair in chain.windows(2) { + if payload_hash_of(store, pair[1]) == Some(latest_valid_hash) { + return pair[0]; + } + } + if latest_valid_hash.is_zero() { + *chain.last().expect("the chain holds its start") + } else { + // Not on this chain: `optimistic-sync.md` says to treat it as `null`. + holder + } +} + +/// Moves `root` and every optimistic block on its EL chain to validated: +/// pre-gloas blocks leave `optimistic_roots`, gloas payloads recorded +/// `NOT_VALIDATED` become `VALID`. `optimistic-sync.md`: a block's ancestors +/// transition with it. Only EL ancestors do: a payload applied to a block the +/// chain skipped past was not part of what the execution client validated. +/// +/// Skipped entirely when nothing is optimistic, and bounded at finality +/// otherwise. +pub(crate) fn mark_payloads_validated(store: &mut Store, root: H256) { + if !store.has_unvalidated_block_payloads() && !store.has_beacon_optimistic_roots() { + return; + } + for block in el_chain(store, root) { + store.remove_beacon_optimistic_root(block); + if store.has_verified_payload(&block) + && store.beacon_block_payload_status(block).is_not_validated() + && let Some((slot, _parent)) = store.block_entry(&block) + { + store.insert_beacon_block_payload_status(block, slot, PayloadStatusEnum::Valid); + } + } +} + +/// Gives a payload restored from disk a verdict: the status map is scratch, so +/// without this a verified payload from before a restart has no entry, reads +/// as `NOT_VALIDATED`, and is never promoted. With an engine configured they +/// start `SYNCING`, so the first `forkchoiceUpdated` `VALID` for the head +/// validates them; without one the follower trusts them, as it trusts every +/// envelope it applies. +pub(crate) fn seed_resumed_payload_statuses(store: &mut Store, engine_configured: bool) { + let status = if engine_configured { + PayloadStatusEnum::Syncing + } else { + PayloadStatusEnum::Valid + }; + for root in store.verified_payload_roots() { + if let Some((slot, _parent)) = store.block_entry(&root) { + store.insert_beacon_block_payload_status(root, slot, status); + } + } +} + +impl BlockChainServer { + /// [`Self::apply_forkchoice_verdict`] for a gloas head, whose `head_hash` + /// names a payload. + /// + /// `VALID` promotes every optimistic payload on the holder's ancestry; + /// `INVALID` condemns from the first payload after `latestValidHash`. + /// `SYNCING` and `ACCEPTED` change nothing. + pub(crate) fn apply_gloas_forkchoice_verdict( + &mut self, + head_root: H256, + head_hash: ExecutionBlockHash, + status: ðlambda_engine::PayloadStatusV1, + ) { + let verdict = crate::beacon_engine::verdict(status); + let (PayloadValidity::Validated | PayloadValidity::Invalidated { .. }) = verdict else { + return; + }; + let Some(holder) = payload_holder(&self.store, head_root, head_hash) else { + // Expected for a head whose parent payload is below the index (an + // anchor): the specification treats it as known-null. + debug!( + head_root = %ShortRoot(&head_root.0), + "No block on the head's chain carries the payload forkchoiceUpdated named" + ); + return; + }; + match verdict { + PayloadValidity::Validated => mark_payloads_validated(&mut self.store, holder), + PayloadValidity::Invalidated { latest_valid_hash } => { + if self.condemn_payload_chain(holder, latest_valid_hash) { + // Not `recompute_beacon_head`: this runs inside + // `forkchoiceUpdated`'s own response handling. + self.update_head_from_fork_choice(); + } + } + PayloadValidity::Optimistic | PayloadValidity::NotRequired => {} + } + } + + /// Applies an `INVALID` verdict about the payload of `holder` (an imported + /// block, or a gloas block whose envelope was just refused). + /// + /// Resolves how far the verdict reaches ([`resolve_invalid_payload`]) and + /// condemns that block's payload: a pre-gloas block is removed with its + /// subtree, as `forkchoiceUpdated` always did; a gloas payload loses its + /// FULL node and every child built on it, and the block itself stays with + /// its EMPTY branch. Returns whether anything left fork choice. + pub(crate) fn condemn_payload_chain( + &mut self, + holder: H256, + latest_valid_hash: Option, + ) -> bool { + let condemned = resolve_invalid_payload(&self.store, holder, latest_valid_hash); + let index = self.store.block_index(); + let mut removed = self.condemn_gloas_payload(condemned, &index); + // The holder's own payload is invalid whichever ancestor the verdict + // reached; when it is a gloas payload it must not stay FULL. + if condemned != holder { + let index = self.store.block_index(); + removed |= self.condemn_gloas_payload(holder, &index); + } + removed + } + + /// Records `root`'s payload as `INVALID`, removes its FULL node, and drops + /// every child built on it, held or imported. + /// + /// A pre-gloas `root` takes the block-level path instead: its payload is + /// the block's, so the whole subtree goes. Refuses a block at or below + /// finality, like [`fork_choice::invalidate_subtree`]. + fn condemn_gloas_payload(&mut self, root: H256, index: &HashMap) -> bool { + let Some(block) = self.store.get_signed_block(&root).ok().flatten() else { + return false; + }; + let Some(bid) = bid_of(&block) else { + return fork_choice::invalidate_subtree(&mut self.store, root) > 0; + }; + let hash = bid.block_hash; + let slot = block.slot(); + let finalized = self.store.beacon_finalized_checkpoint(); + if root == finalized.root || slot <= compute_start_slot_at_epoch(finalized.epoch) { + error!( + condemned = %ShortRoot(&root.0), + "The execution client condemned a finalized payload; refusing to invalidate. \ + The execution and consensus layers disagree about finalized history and \ + this node needs operator attention" + ); + return false; + } + + warn!( + %slot, + block_root = %ShortRoot(&root.0), + payload_hash = %ShortRoot(&hash.0), + "Execution client found a payload invalid; dropping its FULL node" + ); + self.store + .insert_beacon_block_payload_status(root, slot, PayloadStatusEnum::Invalid); + let mut removed = self.store.remove_verified_payload(&root); + + // Children held for this payload build on it by construction. + if let Some(held) = self.envelopes.blocks_awaiting_parent_payload.remove(&root) { + for (child_root, _slot) in held { + self.discard_pending_subtree(child_root); + } + self.publish_envelope_queues(); + } + + // Imported children whose bid names this payload as their parent's. + let full_children: Vec = index + .iter() + .filter(|(_, (_, parent))| *parent == root) + .map(|(child, _)| *child) + .filter(|child| { + self.store + .get_signed_block(child) + .ok() + .flatten() + .and_then(|block| bid_of(&block).map(|bid| bid.parent_block_hash == hash)) + .unwrap_or(false) + }) + .collect(); + for child in full_children { + removed |= fork_choice::invalidate_subtree(&mut self.store, child) > 0; + } + removed + } +} diff --git a/crates/blockchain/src/events.rs b/crates/blockchain/src/events.rs index b948ddb9c..1e82f89e8 100644 --- a/crates/blockchain/src/events.rs +++ b/crates/blockchain/src/events.rs @@ -22,7 +22,8 @@ use ethlambda_storage::{Chain, Store}; use ethlambda_types::ShortRoot; use ethlambda_types::attestation::{AttestationData, SignedAttestation}; use ethlambda_types::beacon::containers::{ - BeaconState, SignedAggregateAndProof, SignedBeaconBlock, electra, fulu, phase0, + BeaconState, DataColumnSidecar, SignedAggregateAndProof, SignedBeaconBlock, electra, gloas, + phase0, }; use ethlambda_types::beacon::serde_helpers::quoted_or_bare; use ethlambda_types::beacon::signing::{compute_epoch_at_slot, compute_start_slot_at_epoch}; @@ -403,6 +404,7 @@ pub struct BeaconChainReorgEvent { pub enum BeaconAttestationEvent { Phase0(phase0::Attestation), Electra(electra::Attestation), + Gloas(gloas::Attestation), } impl From<&SignedAggregateAndProof> for BeaconAttestationEvent { @@ -414,6 +416,7 @@ impl From<&SignedAggregateAndProof> for BeaconAttestationEvent { SignedAggregateAndProof::Electra(signed) => { Self::Electra(signed.message.aggregate.clone()) } + SignedAggregateAndProof::Gloas(signed) => Self::Gloas(signed.message.aggregate.clone()), } } } @@ -671,12 +674,12 @@ impl ChainEvents { /// Beacon `data_column_sidecar`: `sidecar`, of the block under /// `block_root`, was stored. - pub(crate) fn publish_data_column(&self, block_root: H256, sidecar: &fulu::DataColumnSidecar) { + pub(crate) fn publish_data_column(&self, block_root: H256, sidecar: &DataColumnSidecar) { self.bus .emit(ChainEvent::DataColumnSidecar(DataColumnSidecarEvent { block_root, - index: sidecar.index, - slot: sidecar.signed_block_header.message.slot, + index: sidecar.index(), + slot: sidecar.slot(), })); } } diff --git a/crates/blockchain/src/lib.rs b/crates/blockchain/src/lib.rs index 7d2960b9a..7692bc2bd 100644 --- a/crates/blockchain/src/lib.rs +++ b/crates/blockchain/src/lib.rs @@ -1,4 +1,4 @@ -use ethlambda_engine::{EngineClient, ForkchoiceStateV1, PayloadStatusV1 as EnginePayloadStatus}; +use ethlambda_engine::{CustodyColumns, EngineClient, PayloadStatusV1 as EnginePayloadStatus}; use ethlambda_network_api::{ AggregateArrival, BlockAnnouncement, BlockArrival, BlockChainToP2PRef, BlockSource, DeferredFrom, FetchRequest, InitP2P, @@ -15,7 +15,7 @@ use ethlambda_types::{ beacon::{ config::Config, constants, - containers::{SignedAggregateAndProof, SignedBeaconBlock, fulu}, + containers::{DataColumnSidecar, SignedAggregateAndProof, SignedBeaconBlock, gloas}, preset, primitives::ValidatorIndex, signing::compute_epoch_at_slot, @@ -25,7 +25,6 @@ use ethlambda_types::{ primitives::{H256, HashTreeRoot as _}, time::unix_now_ms, }; -use libssz::{SszDecode as _, SszEncode as _}; use std::collections::{HashMap, HashSet, VecDeque}; use std::time::{Duration, Instant, SystemTime}; @@ -45,6 +44,8 @@ use spawned_concurrency::tasks::{ use tokio_util::sync::CancellationToken; use tracing::{debug, error, info, trace, warn}; +use crate::beacon_columns::{ColumnParking, ParkedColumn}; +use crate::beacon_envelope::EnvelopeQueues; use crate::block_builder::ProposerConfig; use crate::events::ChainEvents; use crate::import_timing::{BlockImportReport, CascadeTimings, HeadTimings, ImportTimings}; @@ -54,7 +55,11 @@ pub use events::{ChainEvent, EventBus, Topic, UnknownTopic}; pub mod aggregation; mod beacon_aggregates; +mod beacon_columns; pub mod beacon_engine; +mod beacon_envelope; +mod beacon_payloads; +pub use beacon_payloads::checkpoint_hash; pub mod block_builder; pub(crate) mod coverage; pub mod events; @@ -135,18 +140,6 @@ pub const GOSSIP_DISPARITY_INTERVALS: u64 = 1; pub const MAXIMUM_GOSSIP_CLOCK_DISPARITY: Duration = Duration::from_millis(ethlambda_types::beacon::constants::MAXIMUM_GOSSIP_CLOCK_DISPARITY); -/// Where a parked data column sidecar was put. -/// -/// The three fields are exactly `Table::PendingDataColumns`'s key, so reading -/// the sidecar back needs nothing else. `slot` is also what the finality sweep -/// compares against. -#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)] -struct ParkedColumn { - slot: u64, - block_root: H256, - index: u64, -} - #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub(crate) enum SlotInterval { BlockPublication, @@ -387,7 +380,7 @@ impl BlockChain { /// `safe_slots_to_import_optimistically`, which nothing on that arm reads. #[allow(clippy::too_many_arguments)] fn start_actor( - store: Store, + mut store: Store, sync_status: SyncStatusTracker, sync_status_controller: SyncStatusController, events: EventBus, @@ -398,6 +391,12 @@ impl BlockChain { ) -> BlockChain { let genesis_time = store.config().genesis_time; + // Payloads verified before a restart have no recorded verdict, since + // the map is scratch; see `seed_resumed_payload_statuses`. + if matches!(duties, ChainDuties::Beacon) { + beacon_payloads::seed_resumed_payload_statuses(&mut store, engine.is_some()); + } + // `sidecars_awaiting_parent` starts empty below, and it is the only // index into `Table::PendingDataColumns`. Anything a previous run // parked there is unreachable from here on, so it goes now rather than @@ -415,6 +414,8 @@ impl BlockChain { blocks_awaiting_columns: HashMap::new(), held_timings: HashMap::new(), sidecars_awaiting_parent: HashMap::new(), + column_parking: ColumnParking::default(), + envelopes: EnvelopeQueues::default(), beacon_aggregates: Default::default(), custody_columns, engine, @@ -509,11 +510,22 @@ pub struct BlockChainServer { /// never have can therefore grow it until finality reclaims the slots; /// see [`Self::queue_sidecar_awaiting_parent`]. Always empty on lean. /// - /// A set per parent, so "the same column is never parked twice" is the + /// Keyed by the root whose post-state the sidecar is judged against: its + /// parent for a fulu sidecar, its own block for a gloas one, whose + /// commitments are in that block's bid ([`awaited_root`]). + /// + /// A set per root, so "the same column is never parked twice" is the /// container's own rule rather than a scan every arrival pays for. Order /// is not one: a replay checks and stores each sidecar on its own, and a /// held block is released by its last column arriving, whichever that is. sidecars_awaiting_parent: HashMap>, + /// What bounds [`Self::sidecars_awaiting_parent`]: its running size, and + /// which gloas roots are parked at each slot. + column_parking: ColumnParking, + + /// Gloas execution payload envelopes held for their block or columns, and + /// blocks held for their parent's envelope. See `beacon_envelope`. + envelopes: EnvelopeQueues, /// The columns this node samples, computed once at startup from its node /// id (see `das::custody_columns`). Empty on lean. @@ -829,12 +841,18 @@ fn data_availability_for( let sidecars = custody_columns .iter() .map(|&index| { - let encoded = store - .get_data_column_sidecar(slot, &block_root, index) - .expect("DB read should succeed") - .expect("presence just confirmed above"); - fulu::DataColumnSidecar::from_ssz_bytes(&encoded) + let sidecar = store + .get_data_column(slot, &block_root, index) .expect("a sidecar this node verified before storing decodes") + .expect("presence just confirmed above"); + match sidecar { + DataColumnSidecar::Fulu(sidecar) => sidecar, + // The row decodes by its slot's fork, and this arm is + // the fulu block at that slot. + DataColumnSidecar::Gloas(_) => { + unreachable!("a fulu block's slot decodes its rows as fulu") + } + } }) .collect(); @@ -845,17 +863,12 @@ fn data_availability_for( | SignedBeaconBlock::Bellatrix(_) | SignedBeaconBlock::Capella(_) => Some(fork_choice::DataAvailability::NotRequired), // ePBS (EIP-7732) moves the availability question onto the payload - // envelope's data columns, not this block's own (nonexistent) - // `blob_kzg_commitments`, so this gate has nothing gloas-shaped to - // check yet. Unreachable from network input today: - // `process_or_pend_block` refuses every gloas block before this - // function is ever called, because this node cannot yet deliver the - // payload envelopes and payload attestations a gloas chain needs. - // `on_block` itself accepts a gloas block and reads no availability - // evidence for it. `None` here regardless, defensively: were that - // gate ever bypassed, `None` keeps a block this node cannot follow - // out of `on_block` entirely. - SignedBeaconBlock::Gloas(_) => None, + // envelope: the commitments are in the block's bid, and the columns + // they commit to gate the envelope that reveals the payload, not the + // block's own import (see `beacon_envelope`). So a gloas block has + // nothing to wait for here, and `on_block` reads no availability + // evidence for it. + SignedBeaconBlock::Gloas(_) => Some(fork_choice::DataAvailability::NotRequired), // `process_block` dispatches a lean block to `store::on_block` before // this function is ever reached, so this arm is never observed for // one; named on its own rather than folded into the group above so @@ -1046,7 +1059,12 @@ impl BlockChainServer { // why nothing else evicts a held block nobody redelivers. self.evict_held_blocks_at_or_below_finality(); self.evict_sidecars_awaiting_parent_at_or_below_finality(); + self.evict_envelope_queues_at_or_below_finality(); self.redrive_held_blocks().await; + self.redrive_envelopes_awaiting_columns(); + self.redrive_envelopes_awaiting_engine(); + self.redrive_missing_envelopes(); + self.settle_envelopes().await; // Per-interval duties for this tick. Lean-only, so this is where a // beacon follower's tick ends: it has no validator duties (see @@ -1913,6 +1931,7 @@ impl BlockChainServer { // held block nobody redelivers; see the method's own documentation. self.evict_held_blocks_at_or_below_finality(); self.evict_sidecars_awaiting_parent_at_or_below_finality(); + self.evict_envelope_queues_at_or_below_finality(); self.refresh_chain_metrics(); @@ -1963,6 +1982,8 @@ impl BlockChainServer { blocks_awaiting_columns: HashMap::new(), held_timings: HashMap::new(), sidecars_awaiting_parent: HashMap::new(), + column_parking: ColumnParking::default(), + envelopes: EnvelopeQueues::default(), custody_columns: Vec::new(), engine, safe_slots_to_import_optimistically, @@ -2436,7 +2457,7 @@ impl BlockChainServer { /// A failure here means fork choice could not find a head (for instance /// every known block is unjustifiable), which is a condition to log and /// wait out, not a reason to crash a follower. - fn update_head_from_fork_choice(&mut self) -> HeadTimings { + pub(crate) fn update_head_from_fork_choice(&mut self) -> HeadTimings { let _timing = metrics::time_beacon_head_compute(); let mut timings = HeadTimings { head_start: Some(Instant::now()), @@ -2500,35 +2521,14 @@ impl BlockChainServer { let client = self.engine.clone()?; let (_head_slot, head_root) = self.store.beacon_head()?; - let justified_root = self.store.beacon_justified_checkpoint().root; - let finalized_root = self.store.beacon_finalized_checkpoint().root; - - // `H256::ZERO` explicitly rather than `unwrap_or_default()`: the zero - // hash is a meaningful value here, not an absence. EIP-3675 requires - // `finalized_block_hash` to be zero before a post-transition block is - // finalized, and the specification's own - // `get_safe_execution_block_hash` returns zero when no payload is - // justified yet. - let state = ForkchoiceStateV1 { - head_block_hash: self - .store - .beacon_el_block_hash(head_root) - .unwrap_or(H256::ZERO), - safe_block_hash: self - .store - .beacon_el_block_hash(justified_root) - .unwrap_or(H256::ZERO), - finalized_block_hash: self - .store - .beacon_el_block_hash(finalized_root) - .unwrap_or(H256::ZERO), - }; - - // Nothing to say yet: a follower whose head has no cached payload hash - // is still on its checkpoint anchor. - if state.head_block_hash.is_zero() { - return None; - } + // A gloas head is ambiguous without its payload status (FULL names the + // revealed payload, EMPTY the one before), and a head whose status has + // not been computed yet is skipped. The plan also carries the hashes + // for a pre-gloas head, which keeps its own cached payload hash; + // `None` there means a follower still on its checkpoint anchor. + let plan = beacon_payloads::forkchoice_plan(&self.store, head_root)?; + let state = plan.state; + let gloas = plan.gloas; // `head_root` stays valid across the await: this is a single-threaded // actor, so no other message is handled until this one returns. An @@ -2539,8 +2539,18 @@ impl BlockChainServer { // `apply_forkchoice_verdict` recomputes the head itself rather than // leaving it to the next tick. let start = Instant::now(); - match client.forkchoice_updated(&state).await { - Ok(status) => self.apply_forkchoice_verdict(head_root, &status), + // V4 from gloas on: it carries the custody columns the execution + // client must keep. The follower sends `null` attributes either way. + let response = if gloas { + let custody = CustodyColumns::from_indices(self.custody_columns.iter().copied()); + client.forkchoice_updated_v4(&state, custody).await + } else { + client.forkchoice_updated(&state).await + }; + match response { + Ok(status) => { + self.apply_forkchoice_verdict(head_root, state.head_block_hash, gloas, &status) + } Err(err) => warn!(%err, "forkchoiceUpdated failed"), } Some((start, Instant::now())) @@ -2551,7 +2561,23 @@ impl BlockChainServer { /// `VALID` clears the head and every optimistic ancestor; `INVALID` cuts the /// condemned branch out of fork choice. `SYNCING` and `ACCEPTED` say the /// execution client is still working and change nothing. - fn apply_forkchoice_verdict(&mut self, head_root: H256, status: &EnginePayloadStatus) { + /// + /// For a gloas head the verdict is about `head_hash`, which names a payload + /// rather than a block: the head's own (FULL) or an ancestor's (EMPTY). It + /// is resolved to the block carrying that payload first, and an `INVALID` + /// verdict then takes that payload's FULL node and what builds on it, not + /// the block (see `beacon_payloads`). + fn apply_forkchoice_verdict( + &mut self, + head_root: H256, + head_hash: H256, + gloas: bool, + status: &EnginePayloadStatus, + ) { + if gloas { + self.apply_gloas_forkchoice_verdict(head_root, head_hash, status); + return; + } match beacon_engine::verdict(status) { fork_choice::PayloadValidity::Validated => { fork_choice::mark_validated(&mut self.store, head_root); @@ -2624,38 +2650,6 @@ impl BlockChainServer { let proposer = signed_block.proposer_index(); timings.guards_start = Some(Instant::now()); - // Refused before anything else, including the columns/finalized-slot - // checks below: this node cannot follow a gloas chain yet, since - // nothing delivers the payload envelopes a block's full branch needs - // (`fork_choice::on_execution_payload_envelope`) or the payload - // attestations that vote on them, so persisting a gloas block as - // pending is pure waste. Worse than waste were this not here - // first: a gloas block whose parent is missing would otherwise be - // `insert_pending_block`-ed and tracked in `pending_block_parents`, - // and every later block naming it as an ancestor would walk back to - // it, fetch it out of storage, and requeue it, paying that same - // round trip again on every single delivery, forever, since nothing - // ever marks it imported. `discard_pending_subtree` clears any - // children already queued under it before this landed; logged once, - // here, rather than once per retry, since after this there is no - // retry. - // - // Decided by chain first: a lean block is not a point on the beacon - // fork schedule, and `ForkName::is_followed` refuses to answer for it. - let refused = match self.store.chain() { - Chain::Lean => false, - Chain::Beacon => !signed_block.fork_name().is_followed(), - }; - if refused { - warn!( - %slot, - block_root = %ShortRoot(&block_root.0), - "Refusing a gloas block: this build cannot follow a gloas chain yet" - ); - self.discard_pending_subtree(block_root); - return None; - } - // Asked before the parent check, so that an absent `columns_wait` row // can be read two ways rather than one: the columns were never // missing, or they landed while this block was held for its parent. @@ -2795,7 +2789,9 @@ impl BlockChainServer { // a checkpoint sync, the 68 blocks behind the first one // re-held it 68 times over 54 s while its columns were // already on their way. - if self.blocks_awaiting_columns.contains_key(&missing_root) { + if self.blocks_awaiting_columns.contains_key(&missing_root) + || self.envelopes.is_block_held(&missing_root) + { return None; } // Parent state available — enqueue for processing, cascade @@ -2833,6 +2829,17 @@ impl BlockChainServer { return None; } + // A gloas block built on its parent's FULL node cannot be imported + // until that parent's envelope is verified (`on_block` asserts + // `is_payload_verified`). Held until the envelope arrives rather than + // failed, since it is a block the network accepted. `None`: the + // payload queue is now responsible for it. + if self.parent_payload_unverified(&signed_block) { + timings.guards_end = Some(Instant::now()); + self.hold_block_for_parent_payload(signed_block, timings); + return None; + } + // Parent exists, proceed with processing. Clone the block so we // can run post-import reaggregation against its merged proof — // `process_block` consumes the original for the storage layer. @@ -2891,6 +2898,9 @@ impl BlockChainServer { // sidecar parked under it was waiting for. self.drain_sidecars_awaiting_parent(block_root); + // An envelope that arrived ahead of this block can be judged now. + self.note_block_imported_for_envelope(block_root); + Some(ImportOutcome::Imported) } // A hold writes no post-state, so nothing pending on this root is @@ -2973,6 +2983,7 @@ impl BlockChainServer { .fetch_block(FetchRequest { block_root, needs_block: true, + needs_envelope: false, // Nothing to name: a block this node has never seen has // told it nothing about what it committed to. columns: Vec::new(), @@ -2995,6 +3006,7 @@ impl BlockChainServer { let request = FetchRequest { block_root, needs_block: false, + needs_envelope: false, columns: missing, }; let _ = p2p.fetch_block(request).inspect_err( @@ -3128,7 +3140,7 @@ impl BlockChainServer { /// /// Used when a block is rejected (e.g., at/below finalized slot) to clean up /// children that would otherwise remain stuck in the pending maps indefinitely. - fn discard_pending_subtree(&mut self, block_root: H256) { + pub(crate) fn discard_pending_subtree(&mut self, block_root: H256) { // A block held for its custody columns already had a known parent // when it was held (see `hold_block_for_columns`), so it carries no // entry of its own in `pending_blocks` and would not be reached by @@ -3142,6 +3154,10 @@ impl BlockChainServer { metrics::set_blocks_held_for_columns(self.blocks_awaiting_columns.len() as u64); } + self.forget_block_held_for_parent_payload(block_root); + // An envelope held for this block has nothing left to wait for. + self.forget_envelopes_awaiting_import(block_root); + // Both eviction paths reach this function, so removing here is what // keeps `held_timings` from outliving the blocks it describes. self.held_timings.remove(&block_root); @@ -3224,6 +3240,8 @@ impl BlockChainServer { // that follows the import cascade runs. self.store .prune_beacon_optimistic_roots(finalized_slot, finalized.root); + self.store + .prune_beacon_block_payload_statuses(finalized_slot); if self.blocks_awaiting_columns.is_empty() { return; @@ -3299,15 +3317,15 @@ impl BlockChainServer { /// /// Beacon-only: a lean node subscribes to no column subnet, so nothing /// ever delivers this message there. - async fn on_checked_data_columns(&mut self, sidecars: Vec) { + async fn on_checked_data_columns(&mut self, sidecars: Vec) { for sidecar in sidecars { #[cfg(debug_assertions)] { use ethlambda_state_transition::beacon::gossip::{ Outcome, - column::{ChainVerdict, chain_checks}, + column::{ChainVerdict, chain_checks_for}, }; - let verdict = chain_checks(&self.store, &sidecar, unix_now_ms()); + let verdict = chain_checks_for(&self.store, &sidecar, unix_now_ms()); assert!( matches!( verdict, @@ -3321,10 +3339,10 @@ impl BlockChainServer { } /// Store a checked sidecar and release the held block it may complete. - async fn keep_data_column(&mut self, sidecar: fulu::DataColumnSidecar) { - let header = &sidecar.signed_block_header.message; - let slot = header.slot; - let block_root = header.hash_tree_root(); + async fn keep_data_column(&mut self, sidecar: DataColumnSidecar) { + let slot = sidecar.slot(); + let block_root = sidecar.block_root(); + let index = sidecar.index(); // Two copies of one column can pass the checks at once (from gossip // and from a fetch, say). The second write would store the same row @@ -3333,14 +3351,11 @@ impl BlockChainServer { .store .data_column_indices_for(slot, &block_root) .expect("DB read should succeed"); - if stored.contains(&sidecar.index) { + if stored.contains(&index) { return; } - if let Err(err) = - self.store - .put_data_column_sidecar(slot, &block_root, sidecar.index, sidecar.to_ssz()) - { + if let Err(err) = self.store.put_data_column(&sidecar) { error!(%err, "Failed to store a data column sidecar"); return; } @@ -3349,246 +3364,8 @@ impl BlockChainServer { // A held block may now be complete. self.release_block_if_columns_complete(block_root).await; - } - - /// Park sidecars the chain checks found no parent post-state for, or send - /// them straight back to be checked if the parent has one by now. - /// - /// The second case is a race this actor has to close, because the checks - /// run elsewhere: the p2p layer looked for the parent's post-state, found - /// none and sent these, and if the parent imported in between, its - /// [`Self::drain_sidecars_awaiting_parent`] has already run and will not - /// run again, so a sidecar parked now would wait for nothing until - /// finality evicts it. Asked with the same `get_state` the checks use, so - /// a sidecar sent back is one they will find a parent state for. - fn park_data_columns(&mut self, sidecars: Vec) { - let mut ready = Vec::new(); - for sidecar in sidecars { - let parent_root = sidecar.signed_block_header.message.parent_root; - if matches!(self.store.get_state(&parent_root), Ok(Some(_))) { - ready.push(sidecar); - continue; - } - let block_root = sidecar.signed_block_header.message.hash_tree_root(); - self.queue_sidecar_awaiting_parent(block_root, sidecar); - } - self.send_data_columns_for_checks(ready); - } - - /// Hand sidecars to the p2p layer's chain checks, which send back the - /// ones that pass through `new_data_column_sidecars`. - fn send_data_columns_for_checks(&self, sidecars: Vec) { - if sidecars.is_empty() { - return; - } - let Some(ref p2p) = self.p2p else { - return; - }; - let _ = p2p.check_data_column_sidecars(sidecars).inspect_err( - |err| error!(%err, "Failed to send data column sidecars to the p2p layer for checks"), - ); - } - - /// Park `sidecar` against the parent root it could not be checked against. - /// - /// Counted as `queued_for_parent` rather than as a rejection: nothing about - /// the sidecar has been judged yet, and conflating the two is what made the - /// deadlock invisible in the metrics (every column read as - /// `rejected{reason="unknown_parent"}` while the real fault was upstream). - /// - /// Nothing is refused here. The queue used to hold whole sidecars and so - /// carried a count cap, which a follower behind the tip hit constantly: - /// it receives gossip for the tip continuously, so the queue filled with - /// sidecars for blocks it would not reach for minutes and then refused - /// the ones for the block it was about to import. Measured on the eth-4 - /// follower a hundred slots behind, the queue sat pinned at its cap and - /// dropped 2,561 sidecars in ten minutes while the chain ground through - /// by-root lookups for slots whose columns gossip had already delivered - /// and this function had thrown away. Evicting the furthest-ahead entry - /// instead of the newest fixed which sidecar was lost, not that one was. - /// - /// The cap is gone now that the sidecars live in - /// `Table::PendingDataColumns` and only their keys are held here, so what - /// grows is disk rather than this actor's memory. - /// [`Self::evict_sidecars_awaiting_parent_at_or_below_finality`] is what - /// bounds it, which bounds how *long* an entry lives but not how fast - /// they arrive: the chain checks do not require `parent_root` to name a - /// block this node knows. Every sidecar reaching here, gossiped or - /// fetched, has had its header's signature checked against the head state - /// by those checks (`queue_unless_forged`, in - /// `ethlambda_state_transition::beacon::gossip::column`), but only when a - /// head state is already cached *and* the header's `proposer_index` names - /// a validator in it: with no cached head state, or a proposer index that - /// names none (`u64::MAX`, say), that check is skipped and a made-up - /// header still reaches here and parks a row. A peer exploiting either - /// gap can still park rows as fast as it can invent a slot, proposer and - /// index, until finality catches up. - fn queue_sidecar_awaiting_parent( - &mut self, - block_root: H256, - sidecar: fulu::DataColumnSidecar, - ) { - let header = &sidecar.signed_block_header.message; - let parent_root = header.parent_root; - let parked = ParkedColumn { - slot: header.slot, - block_root, - index: sidecar.index, - }; - - // A re-delivery of something already parked. The by-root and by-range - // fetch paths skip gossip's seen cache entirely, so they never touch - // the p2p actor's `SeenColumns` (which in any case only records an - // Accept, never a park); a re-delivery reaching here is ordinary, and - // without this check the same column would take a second slot in the - // queue and leave a stale key behind after the first replay took its - // row. Asked before the write rather than left to the set below, - // because the write is what costs. - if self - .sidecars_awaiting_parent - .get(&parent_root) - .is_some_and(|parked_columns| parked_columns.contains(&parked)) - { - return; - } - - // The bytes go to disk before the key goes in the map, so a failed - // write leaves no key pointing at a row that is not there. - if let Err(err) = self.store.put_pending_data_column_sidecar( - parked.slot, - &parked.block_root, - parked.index, - sidecar.to_ssz(), - ) { - error!(%err, "Failed to park a data column sidecar"); - return; - } - - trace!( - slot = parked.slot, - column = parked.index, - parent_root = %ShortRoot(&parent_root.0), - "Queueing a data column sidecar until its parent has a post-state" - ); - self.sidecars_awaiting_parent - .entry(parent_root) - .or_default() - .insert(parked); - self.publish_sidecars_awaiting_parent(); - } - - /// Republish how many sidecars are parked, from the map that decides it. - fn publish_sidecars_awaiting_parent(&self) { - let total: usize = self - .sidecars_awaiting_parent - .values() - .map(HashSet::len) - .sum(); - metrics::set_sidecars_awaiting_parent(total as u64); - } - - /// Send every sidecar parked against `block_root` back to the p2p layer's - /// chain checks, now that it has a post-state to be checked against. - /// - /// Called from the one arm that means "this root now has a post-state". - /// The ones that pass come back through `new_data_column_sidecars` as a - /// new message, so nothing here re-enters the import path. - fn drain_sidecars_awaiting_parent(&mut self, block_root: H256) { - let Some(parked_columns) = self.sidecars_awaiting_parent.remove(&block_root) else { - return; - }; - debug!( - parent_root = %ShortRoot(&block_root.0), - count = parked_columns.len(), - "Replaying data column sidecars whose parent just imported" - ); - self.publish_sidecars_awaiting_parent(); - - let mut sidecars = Vec::with_capacity(parked_columns.len()); - for parked in parked_columns { - // Taken, not read: the row has served its purpose either way. A - // replay that passes is written to `DataColumns`, and one that - // fails a check has been judged, so neither leaves anything worth - // keeping here. - let encoded = match self.store.take_pending_data_column_sidecar( - parked.slot, - &parked.block_root, - parked.index, - ) { - Ok(Some(encoded)) => encoded, - Ok(None) => { - error!( - slot = parked.slot, - column = parked.index, - block_root = %ShortRoot(&parked.block_root.0), - "A parked data column sidecar has no row to replay from" - ); - continue; - } - Err(err) => { - error!(%err, "Failed to read back a parked data column sidecar"); - continue; - } - }; - let Ok(sidecar) = fulu::DataColumnSidecar::from_ssz_bytes(&encoded) else { - error!( - slot = parked.slot, - column = parked.index, - "A parked data column sidecar did not decode" - ); - continue; - }; - sidecars.push(sidecar); - } - self.send_data_columns_for_checks(sidecars); - } - - /// Drop parked sidecars whose block finality has superseded. - /// - /// The counterpart of [`Self::evict_held_blocks_at_or_below_finality`] and - /// run beside it, for the same reason: a parent root that never arrives - /// would otherwise pin its children's sidecars for this node's whole - /// uptime. A sidecar at or below the finalized slot can never be needed - /// again, since the chain checks would drop it outright now. - fn evict_sidecars_awaiting_parent_at_or_below_finality(&mut self) { - if self.sidecars_awaiting_parent.is_empty() { - return; - } - let finalized_slot = self - .store - .latest_finalized() - .expect("finalized checkpoint exists") - .slot; - - let mut dropped: Vec = Vec::new(); - self.sidecars_awaiting_parent.retain(|_, parked_columns| { - parked_columns.retain(|parked| { - let keep = parked.slot > finalized_slot; - if !keep { - dropped.push(*parked); - } - keep - }); - !parked_columns.is_empty() - }); - - if !dropped.is_empty() { - info!( - finalized_slot, - count = dropped.len(), - "Evicting parked data column sidecars that finality has superseded" - ); - // The rows go with the keys, so a key dropped from the map never - // leaves its bytes on disk with nothing left to read them. - let keys = dropped - .iter() - .map(|parked| (parked.slot, parked.block_root, parked.index)); - let _ = self - .store - .delete_pending_data_column_sidecars(keys) - .inspect_err(|err| error!(%err, "Failed to drop parked data column sidecars")); - self.publish_sidecars_awaiting_parent(); - } + // And so may a held gloas envelope. + self.note_column_stored_for_envelope(block_root); } /// Once a slot, revisit every held block: release the ones whose columns @@ -3766,20 +3543,98 @@ impl BlockChainServer { /// Whether the clock is at a fork this node does not follow while the /// node's own head is still from one it does. /// - /// Blocks past the fork never reach the live chain, so the freshest block + /// Every fork is followed today, so this is false; it stays as the guard + /// for the next fork the node cannot follow. + /// + /// Blocks past the fork do not reach the live chain from the network, so the freshest block /// the store knows stays at the head and the sync tracker would read the /// network as stalled `NETWORK_STALL_THRESHOLD` slots after the fork and /// report `Synced` for good on a node that follows nothing. Nothing else /// tells the node the network is ahead: gossip blocks past the fork are - /// ignored in p2p, and a node that runs across the fork keeps the fork - /// digest it computed at startup, so peers' Status messages are dropped - /// and no range session starts. The clock is the one signal that is always - /// there, and treating the network as fresh lets head lag decide. + /// ignored in p2p. The fork digest rolls over at runtime, so a node that + /// runs across a fork it can follow switches topics and Status digests on + /// its own; for a fork it cannot follow that switch brings nothing in. The + /// clock is the one signal that is always there, and treating the network + /// as fresh lets head lag decide. fn clock_is_past_followed_forks(config: &Config, current_slot: u64, head_slot: u64) -> bool { let fork_at = |slot: u64| config.fork_at_epoch(compute_epoch_at_slot(slot)); !fork_at(current_slot).is_followed() && fork_at(head_slot).is_followed() } + /// Advances a beacon store's clock to `event_ms` (Unix milliseconds), for + /// an event whose handler reads the clock to judge a sub-slot deadline. + /// + /// The beacon tick fires once per slot, so between ticks the store sits at + /// the slot's start and [`fork_choice::on_block`] would find every + /// current-slot block inside both the attestation and the payload + /// timeliness deadlines. The specification has no such gap: its `on_tick` + /// is called "whenever `time > store.time`", so the clock is wherever the + /// last event put it. The follower's tick was per slot only because it has + /// no sub-slot duties of its own, which nothing here changes. + /// + /// Never backwards (a re-delivered block brings an old arrival), and never + /// across a slot boundary: crossing one is the tick's job, since it also + /// drains deferred aggregates and recomputes the head, which this does + /// not. An event from a later slot than the store reads is clamped to the + /// last instant of the current one. The specification's clock is whole + /// seconds, so the deadlines are judged on the second the event fell in. + fn advance_beacon_clock_to(&mut self, event_ms: u64) { + if self.store.chain() != Chain::Beacon { + return; + } + let config = self.store.config(); + let current_slot = fork_choice::get_current_slot(&self.store, &config); + let slot_end_ms = config + .genesis_time_ms() + .saturating_add( + current_slot + .saturating_add(1) + .saturating_mul(config.slot_duration_ms), + ) + .saturating_sub(1); + let time = event_ms.min(slot_end_ms) / 1000; + let store_time_ms = self.store.time_ms().expect("store time exists"); + if fork_choice::seconds_to_milliseconds(time) > store_time_ms { + fork_choice::on_tick(&mut self.store, time, &config); + } + } + + /// Applies a gossip payload attestation message to fork choice, with the + /// clock first advanced to the instant it arrived: the specification's + /// `on_payload_attestation_message` rejects a vote outside its own slot, + /// and only the arrival says which slot that is. + /// + /// An error is logged at `debug` and the vote dropped. The common cause + /// is a vote for a block not imported here yet. The vote is only valid + /// within its own slot and the specification makes queueing it optional, + /// so it is not held. + /// + /// The message passed gossip validation in `ethlambda-p2p`, which + /// verified its signature and the sender's committee seat, so only the + /// store-state checks run here. + fn apply_payload_attestation_message( + &mut self, + message: &gloas::PayloadAttestationMessage, + arrival: &BlockArrival, + ) { + self.advance_beacon_clock_to(arrival.wire_ms(Instant::now(), unix_now_ms())); + let config = self.store.config(); + let slot = message.data.slot; + let validator = message.validator_index; + let block_root = message.data.beacon_block_root; + let _ = + fork_choice::on_payload_attestation_message(&mut self.store, message, false, &config) + .inspect_err(|err| { + debug!( + slot, + validator, + block_root = %ShortRoot(&block_root.0), + %err, + "Dropping a payload attestation message" + ) + }); + } + /// Milliseconds until this actor's next tick, dispatched by chain: lean's /// interval grid via [`ms_until_next_interval`], beacon's once-per-slot /// cadence via [`ms_until_next_beacon_slot`]. @@ -3897,7 +3752,7 @@ impl BlockChainServer { use ethlambda_network_api::p2p_to_block_chain::{ DataColumnSidecarsAwaitingParent, NewAggregatedAttestation, NewAttestation, NewBeaconAggregate, - NewBlock, NewDataColumnSidecars, + NewBlock, NewDataColumnSidecars, NewExecutionPayloadEnvelope, NewPayloadAttestationMessage, }; impl Handler for BlockChainServer { @@ -4002,7 +3857,49 @@ impl Handler for BlockChainServer { // the import, the catch-up tick included, so it is its own section // rather than the head of the first pass's guards. timings.admit_end = Some(Instant::now()); + // Time the import at the block's arrival on this node, not at the + // slot's start where the tick left the clock: `on_block` reads it for + // the attestation and payload timeliness deadlines. The arrival is the + // wire receipt (`decode_start`), which predates validation and the + // mailbox wait; `arrival_ms` was sampled at pickup. + self.advance_beacon_clock_to(msg.arrival.wire_ms(picked_up, arrival_ms)); self.on_block(msg.block, timings).await; + // A gloas envelope that arrived ahead of this block is judgeable now. + self.settle_envelopes().await; + } +} + +impl Handler for BlockChainServer { + async fn handle(&mut self, msg: NewExecutionPayloadEnvelope, _ctx: &Context) { + // Beacon-only: lean has no execution payload envelopes. + let ChainDuties::Beacon = &self.duties else { + return; + }; + let arrival_ms = unix_now_ms(); + let picked_up = Instant::now(); + self.receive_envelope(*msg.envelope, msg.arrival.wire_ms(picked_up, arrival_ms)) + .await; + self.settle_envelopes().await; + } +} + +impl Handler for BlockChainServer { + async fn handle(&mut self, msg: NewPayloadAttestationMessage, ctx: &Context) { + // Beacon-only: lean has no payload timeliness committee. + let ChainDuties::Beacon = &self.duties else { + return; + }; + // The catch-up tick a block gets: a tick still owed for the vote's + // slot would otherwise leave the store a slot behind and drop every + // vote of the new slot as not current. + let slot = msg.message.data.slot; + let config = self.store.config(); + if self.ms_until_slot_start(slot) == 0 + && fork_choice::get_current_slot(&self.store, &config) < slot + { + self.on_tick(unix_now_ms(), ctx).await; + } + self.apply_payload_attestation_message(&msg.message, &msg.arrival); } } @@ -4053,6 +3950,7 @@ impl Handler for BlockChainServer { impl Handler for BlockChainServer { async fn handle(&mut self, msg: NewDataColumnSidecars, _ctx: &Context) { self.on_checked_data_columns(msg.sidecars).await; + self.settle_envelopes().await; } } @@ -4204,13 +4102,16 @@ mod tests { use ethlambda_state_transition::beacon::fork_choice::seconds_to_milliseconds; use ethlambda_storage::backend::InMemoryBackend; use ethlambda_types::beacon::config::Config; - use ethlambda_types::beacon::containers::{BeaconState, deneb, electra, phase0, shared}; + use ethlambda_types::beacon::containers::{ + BeaconState, deneb, electra, fulu, gloas, phase0, shared, + }; use ethlambda_types::beacon::fork::ForkName; use ethlambda_types::beacon::preset; use ethlambda_types::checkpoint::Checkpoint; use ethlambda_types::state::State; + use libssz::SszEncode as _; - const GENESIS_TIME: u64 = 1_000; + pub(crate) const GENESIS_TIME: u64 = 1_000; fn config(milliseconds_per_slot: u64) -> ChainConfig { ChainConfig::new(GENESIS_TIME, milliseconds_per_slot) @@ -4392,7 +4293,7 @@ mod tests { /// /// No anchor state is written: its only caller asserts on the chain tag /// `Store::init_beacon` sets, which is seeded before any state is. - fn beacon_store(genesis_time: u64, finalized_slot: u64) -> Store { + pub(crate) fn beacon_store(genesis_time: u64, finalized_slot: u64) -> Store { beacon_store_with_config(genesis_time, finalized_slot, Config::mainnet()) } @@ -4492,7 +4393,7 @@ mod tests { // with `beacon::gossip::column`. // ----------------------------------------------------------------- - fn beacon_server(store: Store) -> BlockChainServer { + pub(crate) fn beacon_server(store: Store) -> BlockChainServer { BlockChainServer { events: ChainEvents::new(EventBus::default(), &store), store, @@ -4502,6 +4403,8 @@ mod tests { blocks_awaiting_columns: HashMap::new(), held_timings: HashMap::new(), sidecars_awaiting_parent: HashMap::new(), + column_parking: ColumnParking::default(), + envelopes: EnvelopeQueues::default(), beacon_aggregates: Default::default(), custody_columns: Vec::new(), engine: None, @@ -4518,7 +4421,7 @@ mod tests { /// column cell, all the same length), so the chain checks judge it on its /// parent rather than its shape; every other field is its type's default, /// since nothing under test reads past the header. - fn sidecar_at(slot: u64, parent_root: H256) -> fulu::DataColumnSidecar { + pub(crate) fn sidecar_at(slot: u64, parent_root: H256) -> fulu::DataColumnSidecar { let cell: fulu::Cell = libssz_types::SszVector::try_from(vec![0u8; preset::BYTES_PER_CELL]) .expect("exact cell size"); fulu::DataColumnSidecar { @@ -4553,7 +4456,7 @@ mod tests { /// what `Store::insert_signed_block` writes into `LiveChain`, which is /// all `get_checkpoint_block`'s ancestry walk ever reads. The fork is /// irrelevant to that walk, so the cheapest shape to build stands in. - fn bare_block(slot: u64, parent_root: H256) -> SignedBeaconBlock { + pub(crate) fn bare_block(slot: u64, parent_root: H256) -> SignedBeaconBlock { SignedBeaconBlock::Phase0(phase0::SignedBeaconBlock { message: phase0::BeaconBlock { slot, @@ -4585,7 +4488,7 @@ mod tests { /// `insert_state` takes that as "first state on record" and stores a /// plain snapshot, rather than trying to diff against a parent state /// this helper never creates. - fn bare_state() -> BeaconState { + pub(crate) fn bare_state() -> BeaconState { let block_roots = vec![H256::ZERO; preset::SLOTS_PER_HISTORICAL_ROOT] .try_into() .expect("exactly the preset length"); @@ -4680,9 +4583,9 @@ mod tests { /// to read back. Only `check_data_column_sidecars` and `fetch_block` are /// recorded: nothing under test here sends the others. #[derive(Default)] - struct RecordingP2P { - checks: std::sync::Mutex>>, - fetches: std::sync::Mutex>, + pub(crate) struct RecordingP2P { + pub(crate) checks: std::sync::Mutex>>, + pub(crate) fetches: std::sync::Mutex>, } impl ethlambda_network_api::BlockChainToP2P for RecordingP2P { @@ -4713,7 +4616,7 @@ mod tests { } fn check_data_column_sidecars( &self, - sidecars: Vec, + sidecars: Vec, ) -> Result<(), spawned_concurrency::error::ActorError> { self.checks.lock().unwrap().push(sidecars); Ok(()) @@ -4721,7 +4624,7 @@ mod tests { } /// `beacon_server(store)` with a [`RecordingP2P`] wired in as its p2p ref. - fn beacon_server_recording(store: Store) -> (BlockChainServer, Arc) { + pub(crate) fn beacon_server_recording(store: Store) -> (BlockChainServer, Arc) { let p2p = Arc::new(RecordingP2P::default()); let mut server = beacon_server(store); server.p2p = Some(p2p.clone()); @@ -4729,9 +4632,10 @@ mod tests { } /// A beacon store whose clock reads slot 10, so a sidecar at slot 10 is - /// neither future nor finalized. - fn beacon_store_at_slot_10() -> Store { - let mut store = beacon_store(GENESIS_TIME, 0); + /// neither future nor finalized. Fulu is active from genesis, since a + /// stored sidecar is decoded as the fork its slot names. + pub(crate) fn beacon_store_at_slot_10() -> Store { + let mut store = beacon_store_fulu_at_genesis(GENESIS_TIME, 0); store .set_time_ms(seconds_to_milliseconds( GENESIS_TIME + 10 * Config::mainnet().seconds_per_slot, @@ -4740,229 +4644,192 @@ mod tests { store } + /// How far into its slot the store clock reads, in milliseconds. + fn ms_into_slot(server: &BlockChainServer) -> u64 { + server.store.ms_since_genesis() % server.store.config().slot_duration_ms + } + + /// A payload attestation message advances the clock to its arrival before + /// it is judged, and one naming a block that is not imported here is + /// dropped without touching the store. #[tokio::test] - async fn a_checked_sidecar_is_stored_as_it_is() { - // Nothing on this path judges the sidecar (that is the p2p layer's - // job), so storing it is the whole contract. `keep_data_column` - // rather than `on_checked_data_columns`, since this placeholder - // sidecar would fail the debug-build re-check. + async fn a_payload_attestation_for_an_unknown_block_advances_the_clock_and_is_dropped() { let mut server = beacon_server(beacon_store_at_slot_10()); - let sidecar = sidecar_at(10, H256::repeat_byte(9)); - let block_root = sidecar.signed_block_header.message.hash_tree_root(); + assert_eq!(ms_into_slot(&server), 0); + let mut message = gloas::PayloadAttestationMessage { + validator_index: 0, + data: Default::default(), + signature: Default::default(), + }; + message.data.slot = 10; + message.data.beacon_block_root = H256::repeat_byte(9); - server.keep_data_column(sidecar).await; + server.apply_payload_attestation_message(&message, &BlockArrival::now()); - assert_eq!( + // The wall clock is long past the store's slot, so the clock is + // clamped to that slot's last second. + assert!(ms_into_slot(&server) > 0); + assert!( server .store - .data_column_indices_for(10, &block_root) - .expect("DB read should succeed"), - vec![0] + .payload_timeliness_vote(&message.data.beacon_block_root) + .is_none() ); } - /// The debug-build safety net: a sidecar the p2p layer forwarded as - /// checked, but that the chain checks would not have kept, stops the - /// actor rather than reaching the store. Release builds skip the re-check - /// and store it. - #[cfg(debug_assertions)] - #[tokio::test] - #[should_panic(expected = "the chain checks refuse")] - async fn a_sidecar_the_p2p_layer_never_checked_fails_the_debug_recheck() { - let mut server = beacon_server(beacon_store_at_slot_10()); - // Its parent has no state, so the chain checks would park it, never - // keep it. - let sidecar = sidecar_at(10, H256::repeat_byte(9)); - - server.on_checked_data_columns(vec![sidecar]).await; - } - + /// A fulu-era block arriving after the attestation deadline used to be + /// judged against the slot-start clock the once-per-slot tick leaves the + /// store on, so `block_timeliness` saw zero milliseconds into the slot and + /// called every current-slot block timely. Advancing the clock to the + /// arrival puts the store past the deadline that verdict is read against. #[test] - fn a_data_column_sidecar_naming_a_parent_with_no_state_is_parked_not_stored() { - // `beacon_store` writes no anchor state (see its own doc comment), so - // any parent root at all is unknown here, including the anchor's own. - let (mut server, p2p) = beacon_server_recording(beacon_store_at_slot_10()); - let parent_root = H256::repeat_byte(9); - let sidecar = sidecar_at(10, parent_root); - let block_root = sidecar.signed_block_header.message.hash_tree_root(); - - server.park_data_columns(vec![sidecar]); - - // Not stored: it has not been checked, so it has not been accepted. - assert_eq!( - server - .store - .data_column_indices_for(10, &block_root) - .unwrap(), - Vec::::new() - ); - // But kept, under the parent it is waiting on. Dropping it here is - // what deadlocks a follower running the availability gate: a held - // block writes no post-state, so every sidecar of every child of it - // lands in exactly this branch. + fn a_late_arrival_moves_the_store_clock_past_the_attestation_deadline() { + let mut server = beacon_server(beacon_store_at_slot_10()); + let config = server.store.config(); + let slot_start_ms = config.genesis_time_ms() + 10 * config.slot_duration_ms; + let deadline_ms = fork_choice::get_attestation_due_ms(10 / 32, &config); assert_eq!( - server - .sidecars_awaiting_parent - .get(&parent_root) - .map(HashSet::len), - Some(1) + ms_into_slot(&server), + 0, + "the tick leaves the clock at slot start" ); - assert!(p2p.checks.lock().unwrap().is_empty()); + + server.advance_beacon_clock_to(slot_start_ms + deadline_ms + 500); + + assert!(ms_into_slot(&server) >= deadline_ms); } #[test] - fn a_sidecar_whose_parent_imported_meanwhile_goes_back_for_checks_not_into_the_queue() { - // The race the p2p layer's checks open: they found no parent state, - // then the parent imported and drained its (still empty) queue before - // this message arrived. Parking it now would strand it until - // finality, since that parent never drains again. - let (mut server, p2p) = beacon_server_recording(beacon_store_at_slot_10()); - let parent_root = H256::repeat_byte(9); - server - .store - .insert_state(parent_root, bare_state()) - .expect("insert"); - let sidecar = sidecar_at(10, parent_root); + fn an_early_arrival_keeps_the_store_clock_inside_the_attestation_window() { + let mut server = beacon_server(beacon_store_at_slot_10()); + let config = server.store.config(); + let slot_start_ms = config.genesis_time_ms() + 10 * config.slot_duration_ms; + let deadline_ms = fork_choice::get_attestation_due_ms(10 / 32, &config); - server.park_data_columns(vec![sidecar.clone()]); + server.advance_beacon_clock_to(slot_start_ms + 1_000); - assert!(server.sidecars_awaiting_parent.is_empty()); - assert_eq!(*p2p.checks.lock().unwrap(), vec![vec![sidecar]]); + assert!(ms_into_slot(&server) < deadline_ms); } #[test] - fn a_parked_sidecar_goes_back_for_checks_once_its_parent_gains_a_post_state() { - // The deadlock this closes, in miniature: while the parent has no - // post-state every sidecar under it parks, and if parking were the end - // of the story the queue would only ever grow. What breaks the cycle - // is that gaining a post-state releases them, to the p2p layer's - // checks, since this actor no longer judges a sidecar itself. - let (mut server, p2p) = beacon_server_recording(beacon_store_at_slot_10()); - let parent_root = H256::repeat_byte(9); - let sidecar = sidecar_at(10, parent_root); - let block_root = sidecar.signed_block_header.message.hash_tree_root(); - - server.park_data_columns(vec![sidecar.clone()]); - assert!(server.sidecars_awaiting_parent.contains_key(&parent_root)); + fn the_store_clock_never_moves_backwards_on_an_arrival() { + let mut server = beacon_server(beacon_store_at_slot_10()); + let config = server.store.config(); + let slot_start_ms = config.genesis_time_ms() + 10 * config.slot_duration_ms; + server.advance_beacon_clock_to(slot_start_ms + 6_000); + let before = server.store.time_ms().unwrap(); - server.drain_sidecars_awaiting_parent(parent_root); + // A held block re-delivered with its original, earlier arrival. + server.advance_beacon_clock_to(slot_start_ms + 2_000); - // Gone from the queue and from `PendingDataColumns`, and handed to - // the checks exactly as it was parked. - assert!(!server.sidecars_awaiting_parent.contains_key(&parent_root)); - assert!( - server - .store - .take_pending_data_column_sidecar(10, &block_root, 0) - .expect("DB read should succeed") - .is_none() - ); - assert_eq!(*p2p.checks.lock().unwrap(), vec![vec![sidecar]]); + assert_eq!(server.store.time_ms().unwrap(), before); } #[test] - fn a_parked_sidecar_holds_its_bytes_on_disk_and_not_in_the_queue() { - // The queue's size is chosen by whoever is gossiping, so what it holds - // per entry is the thing that has to stay small: a key, not a cell per - // blob. + fn an_arrival_past_the_slot_does_not_cross_the_boundary_ahead_of_the_tick() { + // Crossing it is the tick's job: it also drains deferred aggregates + // and recomputes the head, which this advance does not. let mut server = beacon_server(beacon_store_at_slot_10()); - let parent_root = H256::repeat_byte(9); - let sidecar = sidecar_at(10, parent_root); - let block_root = sidecar.signed_block_header.message.hash_tree_root(); + let config = server.store.config(); + let slot_start_ms = config.genesis_time_ms() + 10 * config.slot_duration_ms; - server.park_data_columns(vec![sidecar]); + server.advance_beacon_clock_to(slot_start_ms + 3 * config.slot_duration_ms); - assert_eq!( - server.sidecars_awaiting_parent.get(&parent_root), - Some(&HashSet::from([ParkedColumn { - slot: 10, - block_root, - index: 0, - }])) - ); - assert!( - server - .store - .take_pending_data_column_sidecar(10, &block_root, 0) - .expect("DB read should succeed") - .is_some(), - "the sidecar's bytes belong in PendingDataColumns" - ); + assert_eq!(fork_choice::get_current_slot(&server.store, &config), 10); } - #[test] - fn a_parked_sidecar_does_not_satisfy_the_availability_gate() { - // Why the parked rows get a table of their own. Nothing has judged a - // parked sidecar's inclusion proof, its KZG batch or its proposer - // signature, so a peer that could get one counted as custodied would - // be able to release a held block with a column it invented. + #[tokio::test] + async fn a_checked_sidecar_is_stored_as_it_is() { + // Nothing on this path judges the sidecar (that is the p2p layer's + // job), so storing it is the whole contract. `keep_data_column` + // rather than `on_checked_data_columns`, since this placeholder + // sidecar would fail the debug-build re-check. let mut server = beacon_server(beacon_store_at_slot_10()); let sidecar = sidecar_at(10, H256::repeat_byte(9)); let block_root = sidecar.signed_block_header.message.hash_tree_root(); - server.park_data_columns(vec![sidecar]); + server + .keep_data_column(DataColumnSidecar::Fulu(sidecar)) + .await; assert_eq!( server .store .data_column_indices_for(10, &block_root) .expect("DB read should succeed"), - Vec::::new(), - "an unverified sidecar must be invisible to data_column_indices_for" + vec![0] ); } - #[test] - fn a_sidecar_parked_twice_takes_one_slot_in_the_queue() { - // The by-root and by-range fetch paths skip gossip's seen cache - // entirely, so nothing between them and this actor dedups a - // re-delivery while the parent is still stateless; it is ordinary. A - // second entry would leave a key with no row behind it once the - // first replay took it. + /// The debug-build safety net: a sidecar the p2p layer forwarded as + /// checked, but that the chain checks would not have kept, stops the + /// actor rather than reaching the store. Release builds skip the re-check + /// and store it. + #[cfg(debug_assertions)] + #[tokio::test] + #[should_panic(expected = "the chain checks refuse")] + async fn a_sidecar_the_p2p_layer_never_checked_fails_the_debug_recheck() { let mut server = beacon_server(beacon_store_at_slot_10()); - let parent_root = H256::repeat_byte(9); + // Its parent has no state, so the chain checks would park it, never + // keep it. + let sidecar = sidecar_at(10, H256::repeat_byte(9)); - server.park_data_columns(vec![sidecar_at(10, parent_root)]); - server.park_data_columns(vec![sidecar_at(10, parent_root)]); + server + .on_checked_data_columns(vec![DataColumnSidecar::Fulu(sidecar)]) + .await; + } - assert_eq!( - server - .sidecars_awaiting_parent - .get(&parent_root) - .map(HashSet::len), - Some(1) + /// [`beacon_store_at_slot_10`] with gloas active from genesis too, so a + /// slot-10 sidecar decodes as gloas's shape. + pub(crate) fn gloas_store_at_slot_10() -> Store { + let mut store = beacon_store_with_config( + GENESIS_TIME, + 0, + Config::mainnet() + .with_fork_epoch(ForkName::Fulu, 0) + .with_fork_epoch(ForkName::Gloas, 0), ); + store + .set_time_ms(seconds_to_milliseconds( + GENESIS_TIME + 10 * Config::mainnet().seconds_per_slot, + )) + .unwrap(); + store } - #[test] - fn parked_sidecars_are_dropped_once_finality_passes_their_slot() { - // Populated directly rather than through `park_data_columns`: the - // chain checks refuse a sidecar at or below the finalized slot before - // it could ever be parked, so the only way to observe the sweep is to - // park one behind their back. The finalized slot is fixed at init, so - // the store carries it rather than the test moving it. - let mut server = beacon_server(beacon_store(GENESIS_TIME, 10)); - let superseded = H256::repeat_byte(1); - let still_wanted = H256::repeat_byte(2); - let parked_at = |slot: u64| ParkedColumn { + /// One blob's worth of column and proof, the smallest shape the parking + /// lot accepts. + pub(crate) fn gloas_sidecar_at(slot: u64, block_root: H256, index: u64) -> DataColumnSidecar { + let cell: fulu::Cell = libssz_types::SszVector::try_from(vec![0u8; preset::BYTES_PER_CELL]) + .expect("exact cell size"); + DataColumnSidecar::Gloas(gloas::DataColumnSidecar { + index, slot, - block_root: H256::repeat_byte(9), - index: 0, - }; - server - .sidecars_awaiting_parent - .insert(superseded, HashSet::from([parked_at(10)])); - server - .sidecars_awaiting_parent - .insert(still_wanted, HashSet::from([parked_at(20)])); + beacon_block_root: block_root, + column: vec![cell].try_into().expect("one cell"), + kzg_proofs: vec![Default::default()].try_into().expect("one proof"), + }) + } - server.evict_sidecars_awaiting_parent_at_or_below_finality(); + #[tokio::test] + async fn a_checked_gloas_sidecar_is_stored_as_it_is() { + let mut server = beacon_server(gloas_store_at_slot_10()); + let block_root = H256::repeat_byte(5); + let sidecar = gloas_sidecar_at(10, block_root, 3); + + server.keep_data_column(sidecar.clone()).await; - // A parent root that never arrives would otherwise pin its children's - // sidecars for this node's whole uptime; one still above finality is - // a parent that may yet show up. - assert!(!server.sidecars_awaiting_parent.contains_key(&superseded)); - assert!(server.sidecars_awaiting_parent.contains_key(&still_wanted)); + assert_eq!( + server + .store + .data_column_indices_for(10, &block_root) + .expect("DB read should succeed"), + vec![3] + ); + assert_eq!( + server.store.get_data_column(10, &block_root, 3).unwrap(), + Some(sidecar) + ); } // ----------------------------------------------------------------- @@ -4985,7 +4852,11 @@ mod tests { /// caller of this helper is exercising, so nothing here needs to be real: /// only structurally present, so `message_hash_tree_root` and `to_ssz` /// succeed. - fn fulu_block(parent_root: H256, slot: u64, commitment_count: usize) -> SignedBeaconBlock { + pub(crate) fn fulu_block( + parent_root: H256, + slot: u64, + commitment_count: usize, + ) -> SignedBeaconBlock { let payload = deneb::ExecutionPayload { parent_hash: Default::default(), fee_recipient: Default::default(), @@ -5107,7 +4978,8 @@ mod tests { #[test] fn a_block_with_every_custody_column_is_available() { - let store = beacon_store(0, 0); + // Fulu at genesis: stored rows decode by the fork their slot names. + let store = beacon_store_fulu_at_genesis(0, 0); let block = fulu_block_with_commitments(&store, 2); let root = block.message_hash_tree_root(); for index in CUSTODY { @@ -5326,17 +5198,17 @@ mod tests { } #[test] - fn the_node_reports_syncing_once_the_clock_passes_the_fork_it_does_not_follow() { - // No block arrives at all: gossip blocks past the fork are ignored in - // p2p and peers' Status messages are dropped, so the store's freshest - // block stays at the head. The clock alone must keep the node from - // reading the network as stalled. + fn the_clock_reaching_a_followed_fork_does_not_force_syncing() { + // Gloas is followed, so blocks past the fork do reach the live chain + // and the tracker's own stall detection applies as before: with no + // block at all it reads a stalled network as synced, the same as when + // no fork is scheduled. let mut server = syncing_server(Some(1)); let past_the_fork = preset::SLOTS_PER_EPOCH + 8; server.update_sync_status(past_the_fork); assert_eq!( server.sync_status_controller.get(), - crate::metrics::SyncStatus::Syncing + crate::metrics::SyncStatus::Synced ); } @@ -5358,33 +5230,24 @@ mod tests { } #[test] - fn the_clock_rule_holds_for_a_followed_head_once_the_clock_is_at_gloas() { + fn the_clock_rule_never_holds_while_every_fork_is_followed() { + // The rule is a guard for a fork the node cannot follow; with gloas + // followed there is none, whatever the clock and head say. let first_gloas_slot = preset::SLOTS_PER_EPOCH; - assert!(BlockChainServer::clock_is_past_followed_forks( - &gloas_at_epoch_one(), - first_gloas_slot + 8, - 0 - )); - } - - #[test] - fn the_clock_rule_does_not_hold_before_the_clock_reaches_gloas() { - let first_gloas_slot = preset::SLOTS_PER_EPOCH; - assert!(!BlockChainServer::clock_is_past_followed_forks( - &gloas_at_epoch_one(), - first_gloas_slot - 1, - 0 - )); - } - - #[test] - fn the_clock_rule_does_not_hold_for_a_head_already_past_the_fork() { - let first_gloas_slot = preset::SLOTS_PER_EPOCH; - assert!(!BlockChainServer::clock_is_past_followed_forks( - &gloas_at_epoch_one(), - first_gloas_slot + 8, - first_gloas_slot - )); + for (current_slot, head_slot) in [ + (first_gloas_slot + 8, 0), + (first_gloas_slot - 1, 0), + (first_gloas_slot + 8, first_gloas_slot), + ] { + assert!( + !BlockChainServer::clock_is_past_followed_forks( + &gloas_at_epoch_one(), + current_slot, + head_slot + ), + "clock {current_slot}, head {head_slot}" + ); + } } #[tokio::test] @@ -6132,8 +5995,12 @@ mod tests { let sidecar = sidecar_at(10, H256::repeat_byte(9)); let block_root = sidecar.signed_block_header.message.hash_tree_root(); - server.keep_data_column(sidecar.clone()).await; - server.keep_data_column(sidecar).await; + server + .keep_data_column(DataColumnSidecar::Fulu(sidecar.clone())) + .await; + server + .keep_data_column(DataColumnSidecar::Fulu(sidecar)) + .await; let emitted = drain(&mut rx); assert_eq!(emitted.len(), 1, "{emitted:?}"); diff --git a/crates/blockchain/src/metrics.rs b/crates/blockchain/src/metrics.rs index 3a40d0d7f..8d5cf1483 100644 --- a/crates/blockchain/src/metrics.rs +++ b/crates/blockchain/src/metrics.rs @@ -949,6 +949,33 @@ static LEAN_BLOCKS_HELD_FOR_COLUMNS: std::sync::LazyLock = .unwrap() }); +static LEAN_ENVELOPES_AWAITING_BLOCK: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_int_gauge!( + "lean_envelopes_awaiting_block", + "Execution payload envelopes held until their block is imported" + ) + .unwrap() + }); + +static LEAN_ENVELOPES_AWAITING_COLUMNS: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_int_gauge!( + "lean_envelopes_awaiting_columns", + "Execution payload envelopes held until their custody columns are stored" + ) + .unwrap() + }); + +static LEAN_BLOCKS_AWAITING_PARENT_PAYLOAD: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_int_gauge!( + "lean_blocks_awaiting_parent_payload", + "Blocks held until their FULL parent's execution payload envelope is verified" + ) + .unwrap() + }); + static LEAN_SIDECARS_AWAITING_PARENT: std::sync::LazyLock = std::sync::LazyLock::new(|| { register_int_gauge!( @@ -1073,6 +1100,9 @@ pub fn init() { // Data column sidecars. std::sync::LazyLock::force(&LEAN_DATA_COLUMNS_STORED_TOTAL); LEAN_BLOCKS_HELD_FOR_COLUMNS.set(0); + LEAN_ENVELOPES_AWAITING_BLOCK.set(0); + LEAN_ENVELOPES_AWAITING_COLUMNS.set(0); + LEAN_BLOCKS_AWAITING_PARENT_PAYLOAD.set(0); } // --- Public API --- @@ -1424,6 +1454,25 @@ pub fn set_blocks_held_for_columns(count: u64) { LEAN_BLOCKS_HELD_FOR_COLUMNS.set(count as i64); } +/// Execution payload envelopes held until their block is imported. +/// +/// Nobody has vouched for these (the block that would judge them is not here +/// yet), so a value that stays up means envelopes are arriving for blocks this +/// node never gets. +pub fn set_envelopes_awaiting_block(count: u64) { + LEAN_ENVELOPES_AWAITING_BLOCK.set(count as i64); +} + +/// Execution payload envelopes held until every sampled column is stored. +pub fn set_envelopes_awaiting_columns(count: u64) { + LEAN_ENVELOPES_AWAITING_COLUMNS.set(count as i64); +} + +/// Blocks held until their FULL parent's execution payload is verified. +pub fn set_blocks_awaiting_parent_payload(count: u64) { + LEAN_BLOCKS_AWAITING_PARENT_PAYLOAD.set(count as i64); +} + /// Sidecars currently parked against a parent root with no post-state. /// /// Reads as the queue depth of the recovery path the availability gate depends diff --git a/crates/blockchain/state_transition/src/beacon/fork_choice.rs b/crates/blockchain/state_transition/src/beacon/fork_choice.rs index 1a326707c..3fdc1130e 100644 --- a/crates/blockchain/state_transition/src/beacon/fork_choice.rs +++ b/crates/blockchain/state_transition/src/beacon/fork_choice.rs @@ -934,7 +934,9 @@ pub fn is_optimistic_candidate_block( /// unrealized justifications in their own table, and the per-slot/per-epoch /// scratch (`proposer_boost_root`, `block_timeliness`, `equivocating_indices`, /// `latest_messages`, `pow_blocks`) in an in-memory struct cheap enough to -/// rebuild after a restart rather than worth persisting. See +/// rebuild after a restart rather than worth persisting; the exceptions are +/// a gloas block's timeliness and verified payload, which are also stored +/// and reloaded on resume. See /// [`ethlambda_storage::Store`]'s own documentation for the full /// field-by-field accounting; this module reads and writes it exclusively /// through its public accessors. @@ -1139,7 +1141,7 @@ pub fn get_forkchoice_store( store.set_payload_link(anchor_root, anchor_slot, anchor_rules.payload_link(None)); match anchor_rules { ForkRules::Gloas => { - store.set_block_timeliness(anchor_root, [true, true]); + store.set_gloas_block_timeliness(anchor_root, [true, true]); store.set_payload_timeliness_vote(anchor_root, vec![None; preset::PTC_SIZE]); store.set_payload_data_availability_vote(anchor_root, vec![None; preset::PTC_SIZE]); } @@ -2249,6 +2251,9 @@ pub fn get_head_node(store: &Store, config: &Config) -> Result { /// on its first tick. That writer also keeps the canonical `BlockRoots` index /// in step with the branch fork choice just picked. /// +/// The head node's payload status is recorded too ([`Store::head_payload_status`]): +/// the root alone cannot say which payload branch of a gloas block was picked. +/// /// Written unconditionally on every call, not only when the head changes: a /// value written once and then left alone is a second source of truth a bug /// can let drift, and the write is one small metadata row plus an index diff @@ -2256,6 +2261,9 @@ pub fn get_head_node(store: &Store, config: &Config) -> Result { /// tree walk. pub fn get_head(store: &mut Store, config: &Config) -> Result { let head = get_head_node(store, config)?; + // Recorded with its root, so a reader can tell which head the status is + // for whichever of the two writes it sees first. + store.set_head_payload_status(head.root, head.payload_status); store .update_checkpoints(ForkCheckpoints::head_only(head.root)) .expect("record beacon head"); @@ -2842,6 +2850,27 @@ pub fn is_payload_verified(store: &Store, root: Root) -> bool { store.has_verified_payload(&root) || is_known_pre_gloas_block(store, root) } +/// The execution verdict gloas's attestation gossip rules read for `root`'s +/// payload, as the specification's `block_payload_statuses`. +/// +/// A pre-gloas block's payload ran inside the block, so its verdict is the +/// block's own: `NOT_VALIDATED` while it sits in the optimistic set, `VALID` +/// otherwise (an invalidated block is dropped from the store, so is never +/// asked about). This keeps an honest gloas-slot vote for the last pre-gloas +/// block, which this node's boundary rule treats as FULL, from being ignored as +/// optimistic. A gloas root reads [`Store::beacon_block_payload_status`]. +pub fn block_payload_status(store: &Store, root: Root) -> PayloadStatusEnum { + if is_known_pre_gloas_block(store, root) { + if store.is_beacon_optimistic(root) { + PayloadStatusEnum::Syncing + } else { + PayloadStatusEnum::Valid + } + } else { + store.beacon_block_payload_status(root) + } +} + /// `payload_timeliness` (gloas `fork-choice.md`): whether `root`'s payload is /// considered `timely` (or not, when `timely` is `false`), taking into /// account both local availability and the payload timeliness committee's @@ -3261,9 +3290,9 @@ fn payload_status_tiebreaker_with( /// **Implementation choice, not spec text**: a candidate with no recorded /// [`Store::block_timeliness`](ethlambda_storage::Store::block_timeliness) /// entry reads as not timely by either deadline, rather than raising -/// `Error::SpecAssert`. That scratch is in-memory only, so every entry is -/// gone after a restart; reading a gap as "not an early equivocation" -/// rather than aborting the whole weight computation over it is the +/// `Error::SpecAssert`. A gloas block's entry is stored and reloaded on +/// resume, so a gap is a block with no entry at all; reading it as "not an +/// early equivocation" rather than aborting the whole weight computation is the /// conservative answer (it can only ever miss withholding a boost, never /// wrongly withhold one), the same shape of tolerance /// [`gloas_get_attestation_score`]'s own doc gives for a pruned vote. @@ -4740,7 +4769,10 @@ pub fn on_block( // first, and shares the pre-import head's proposer shuffling. Both calls // are infallible: nothing from here to the end of this function can turn // into an `Err`, and the store has already been mutated above. - store.set_block_timeliness(block_root, timeliness); + match rules { + ForkRules::Gloas => store.set_gloas_block_timeliness(block_root, timeliness), + ForkRules::PreGloas => store.set_block_timeliness(block_root, timeliness), + } if let Some(pre_block_head) = pre_block_head { update_proposer_boost_root(store, &index, pre_block_head, block_root, config); } @@ -4793,7 +4825,13 @@ pub fn notify_ptc_messages( data: payload_attestation.data, signature: Default::default(), }; - apply_payload_attestation_message(store, &attested_state, &message, true, config)?; + apply_payload_attestation_message( + store, + &attested_state, + &message, + PtcMessageSource::Block, + config, + )?; } } Ok(()) @@ -4825,7 +4863,56 @@ pub fn on_payload_attestation_message( .ok_or(Error::SpecAssert( "data.beacon_block_root in store.block_states", ))?; - apply_payload_attestation_message(store, &state, ptc_message, is_from_block, config) + let source = if is_from_block { + PtcMessageSource::Block + } else { + PtcMessageSource::Wire + }; + apply_payload_attestation_message(store, &state, ptc_message, source, config) +} + +/// [`on_payload_attestation_message`] for a message that +/// `ethlambda-p2p`'s `payload_attestation_message` gossip validation already +/// accepted: the same store-state checks (the block's state is known, the +/// vote is for the current slot, the validator holds a seat in the block's +/// committee, which also locates the seats to write), without the signature +/// check and the indexed-attestation validation behind it, which cost a BLS +/// verification per vote on the chain actor. +/// +/// Only a gossip-verified message may call this. A message from anywhere else +/// (a block's payload attestations go through [`notify_ptc_messages`], which +/// has its own verified path) would be applied unauthenticated. +pub fn apply_verified_payload_attestation( + store: &mut Store, + ptc_message: &gloas::PayloadAttestationMessage, + config: &Config, +) -> Result<()> { + let state = store + .get_state(&ptc_message.data.beacon_block_root) + .expect("get") + .ok_or(Error::SpecAssert( + "data.beacon_block_root in store.block_states", + ))?; + apply_payload_attestation_message( + store, + &state, + ptc_message, + PtcMessageSource::VerifiedWire, + config, + ) +} + +/// Where a payload attestation message came from, which decides what +/// [`apply_payload_attestation_message`] still has to check. +#[derive(Clone, Copy, PartialEq, Eq)] +enum PtcMessageSource { + /// Inside a block, already verified with it: no slot or signature check. + Block, + /// Straight off the wire: current slot, then the signature. + Wire, + /// Off the wire and already signature-verified by gossip validation: + /// current slot only. + VerifiedWire, } /// The body of [`on_payload_attestation_message`] after its state read, given @@ -4836,7 +4923,7 @@ fn apply_payload_attestation_message( store: &mut Store, state: &BeaconState, ptc_message: &gloas::PayloadAttestationMessage, - is_from_block: bool, + source: PtcMessageSource, config: &Config, ) -> Result<()> { let data = ptc_message.data; @@ -4861,11 +4948,13 @@ fn apply_payload_attestation_message( // Verify the signature and check that it is for the current slot if it is // coming from the wire. - if !is_from_block { + if source != PtcMessageSource::Block { verify( data.slot == get_current_slot(store, config), "data.slot == get_current_slot(store)", )?; + } + if source == PtcMessageSource::Wire { let indexed = gloas::IndexedPayloadAttestation { attesting_indices: gloas::PayloadTimelinessCommitteeIndices::try_from(vec![ ptc_message.validator_index, @@ -4934,6 +5023,28 @@ pub fn on_execution_payload_envelope( config: &Config, sidecars: &[gloas::DataColumnSidecar], engine: &stf::ExecutionEngine, +) -> Result<()> { + check_execution_payload_envelope(store, signed_envelope, config, sidecars, engine)?; + accept_execution_payload_envelope(store, signed_envelope); + Ok(()) +} + +/// Every check of [`on_execution_payload_envelope`], without recording the +/// payload. +/// +/// Split out so a caller whose execution engine answers over the network can +/// run the pure consensus checks first, ask the engine only about an envelope +/// that passed them, and record the payload with +/// [`accept_execution_payload_envelope`] once the answer allows it. The +/// `engine` argument still answers the engine's part of +/// `verify_execution_payload_envelope`; such a caller passes +/// [`stf::ExecutionEngine::valid`] and consults the real engine itself. +pub fn check_execution_payload_envelope( + store: &Store, + signed_envelope: &gloas::SignedExecutionPayloadEnvelope, + config: &Config, + sidecars: &[gloas::DataColumnSidecar], + engine: &stf::ExecutionEngine, ) -> Result<()> { let envelope = &signed_envelope.message; let block_root = envelope.beacon_block_root; @@ -4963,13 +5074,26 @@ pub fn on_execution_payload_envelope( // Verify the execution payload envelope. stf::gloas::verify_execution_payload_envelope(&state, signed_envelope, config, engine)?; - // Add execution payload envelope to the store. Only the fact that it is - // verified is kept: nothing in fork choice reads the payload itself. - store.insert_verified_payload(block_root); - Ok(()) } +/// The recording half of [`on_execution_payload_envelope`]: adds a checked +/// envelope to the store. Persisted, so a restarted follower keeps the full +/// branch of this block. +/// +/// The caller must have run [`check_execution_payload_envelope`] on the same +/// envelope; a block the store does not hold is the only case this refuses. +pub fn accept_execution_payload_envelope( + store: &mut Store, + signed_envelope: &gloas::SignedExecutionPayloadEnvelope, +) { + let block_root = signed_envelope.message.beacon_block_root; + let Some((slot, _parent)) = store.block_entry(&block_root) else { + return; + }; + store.insert_verified_payload(slot, signed_envelope); +} + /// Validates `attestation` and, if valid, records it as each attester's /// latest message. /// @@ -5074,10 +5198,11 @@ pub fn on_block_attestation( /// records it against `attesting_indices`, resolved by the caller's gossip /// validation rather than recomputed here. /// -/// The rules are [`ForkRules::PreGloas`]: an aggregate reaches this function -/// only as a pre-gloas (phase0- or electra-shaped) one, since gloas's own -/// aggregate is refused when the gossip payload is decoded and never gets as -/// far as gossip validation. +/// The rules are those of the vote's own fork, read off `data.slot`: under +/// gloas `data.index` is the payload flag that [`update_latest_messages`] +/// records, and the electra-shaped callers that reach here (a gloas aggregate +/// has its own container but yields the same `AttestationData`) do not say +/// which fork they came from. /// /// `is_from_block` is fixed at `false`, matching [`on_attestation`]'s call for /// this topic: an aggregate here is by definition not carried in a block, so @@ -5094,8 +5219,9 @@ pub fn apply_verified_aggregate( config: &Config, index: &HashMap, ) -> Result<()> { - validate_on_attestation_indexed(store, data, ForkRules::PreGloas, false, config, index)?; - update_latest_messages(store, attesting_indices, data, ForkRules::PreGloas); + let rules = ForkRules::of(config.fork_at_epoch(compute_epoch_at_slot(data.slot))); + validate_on_attestation_indexed(store, data, rules, false, config, index)?; + update_latest_messages(store, attesting_indices, data, rules); Ok(()) } @@ -5256,6 +5382,53 @@ mod tests { }) } + /// Marks `root`'s payload verified by storing a default envelope for it. + /// The block must already be in the store: the envelope's row is keyed by + /// the block's slot. + fn verify_payload(store: &mut Store, root: Root) { + let payload = gloas::ExecutionPayload { + parent_hash: Default::default(), + fee_recipient: Default::default(), + state_root: Default::default(), + receipts_root: Default::default(), + logs_bloom: crate::beacon::containers::bellatrix::LogsBloom::try_from(vec![ + 0u8; + preset::BYTES_PER_LOGS_BLOOM + ]) + .expect("built at exactly BYTES_PER_LOGS_BLOOM"), + prev_randao: Default::default(), + block_number: 0, + gas_limit: 0, + gas_used: 0, + timestamp: 0, + extra_data: Default::default(), + base_fee_per_gas: Default::default(), + block_hash: Default::default(), + transactions: Default::default(), + withdrawals: Default::default(), + blob_gas_used: 0, + excess_blob_gas: 0, + block_access_list: Default::default(), + slot_number: 0, + }; + let envelope = gloas::SignedExecutionPayloadEnvelope { + message: gloas::ExecutionPayloadEnvelope { + payload, + execution_requests: Default::default(), + builder_index: 0, + beacon_block_root: root, + parent_beacon_block_root: Root::ZERO, + }, + signature: Default::default(), + }; + let slot = store + .get_signed_block(&root) + .expect("get") + .expect("the block is in the store") + .slot(); + store.insert_verified_payload(slot, &envelope); + } + /// A fulu signed block with an empty body and a zero signature, for the /// fulu-to-gloas boundary tests below, which need a pre-gloas block /// genuinely of fulu's own fork rather than the phase0-shaped [`block`] @@ -6664,7 +6837,7 @@ mod tests { "no verified payload yet: only the empty branch exists" ); - store.insert_verified_payload(a_root); + verify_payload(&mut store, a_root); assert_eq!( get_node_children(&store, &blocks, node).unwrap(), vec![ @@ -6863,7 +7036,7 @@ mod tests { "a payload nobody verified must not be extended, so full loses the tiebreak" ); - store.insert_verified_payload(a_root); + verify_payload(&mut store, a_root); let strong_votes = vec![Some(true); preset::PTC_SIZE]; store.set_payload_timeliness_vote(a_root, strong_votes.clone()); store.set_payload_data_availability_vote(a_root, strong_votes); @@ -6892,7 +7065,7 @@ mod tests { ), ) .unwrap(); - store.insert_verified_payload(a_root); + verify_payload(&mut store, a_root); let exactly_threshold = vec![Some(true); preset::PAYLOAD_TIMELY_THRESHOLD as usize]; store.set_payload_timeliness_vote(a_root, exactly_threshold.clone()); @@ -6961,7 +7134,7 @@ mod tests { "a payload nobody verified must not be extended" ); - store.insert_verified_payload(a_root); + verify_payload(&mut store, a_root); let weak_votes = vec![Some(false); preset::PTC_SIZE]; store.set_payload_timeliness_vote(a_root, weak_votes.clone()); store.set_payload_data_availability_vote(a_root, weak_votes); @@ -7021,7 +7194,7 @@ mod tests { ), ) .unwrap(); - store.insert_verified_payload(a_root); + verify_payload(&mut store, a_root); let state = test_state::with_validators_at(ForkName::Gloas, 4); store.insert_state(anchor_root, state.clone()).unwrap(); @@ -7397,7 +7570,7 @@ mod tests { ), ) .unwrap(); - store.insert_verified_payload(a_root); + verify_payload(&mut store, a_root); let state = test_state::with_validators_at(ForkName::Gloas, 4); store.insert_state(anchor_root, state.clone()).unwrap(); @@ -7565,6 +7738,31 @@ mod tests { ); } + #[test] + fn a_pre_gloas_root_reads_its_own_validity_as_a_payload_status() { + let mut store = empty_store(); + let root = Root::repeat_byte(0xa1); + store + .insert_signed_block(root, fulu_block(1, Root::ZERO)) + .unwrap(); + assert_eq!(block_payload_status(&store, root), PayloadStatusEnum::Valid); + + store.insert_beacon_optimistic_root(root, 1); + assert_eq!( + block_payload_status(&store, root), + PayloadStatusEnum::Syncing + ); + } + + #[test] + fn a_root_with_no_recorded_verdict_is_not_validated() { + let store = empty_store(); + assert!( + block_payload_status(&store, Root::repeat_byte(0xee)).is_not_validated(), + "the specification's default for a root absent from the map" + ); + } + #[test] fn is_payload_verified_is_false_for_an_unknown_root() { let store = empty_store(); @@ -7695,8 +7893,7 @@ mod tests { // A same-slot, same-proposer sibling of the parent that would be an // early equivocation if its timeliness were known, but whose - // `block_timeliness` entry was never recorded: the scratch is - // in-memory only and does not survive a restart. Deliberately no + // `block_timeliness` entry was never recorded. Deliberately no // `store.set_block_timeliness(twin_root, ...)` call. let twin_root = Root::repeat_byte(0xb1); store @@ -7968,7 +8165,7 @@ mod tests { assert_ne!(failed_assertion(rejected), Some(UNVERIFIED_PAYLOAD_PARENT)); // Once the envelope is verified the full child clears the check too. - store.insert_verified_payload(anchor_root); + verify_payload(&mut store, anchor_root); let rejected = import(&mut store, full_child); assert!(rejected.is_err()); assert_ne!(failed_assertion(rejected), Some(UNVERIFIED_PAYLOAD_PARENT)); @@ -8044,7 +8241,7 @@ mod tests { assert!(validate(&store, vote(2, 1), ForkRules::PreGloas).is_ok()); assert!(validate(&store, vote(2, 2), ForkRules::PreGloas).is_ok()); - store.insert_verified_payload(b_root); + verify_payload(&mut store, b_root); assert!(validate(&store, vote(2, 1), ForkRules::Gloas).is_ok()); } @@ -8079,10 +8276,10 @@ mod tests { } } - #[test] - fn on_payload_attestation_message_writes_every_seat_the_validator_holds() { - let config = Config::active().with_fork_epoch(ForkName::Gloas, 0); - let root = Root::repeat_byte(0xb1); + /// A store holding a gloas block at slot 1 whose state seats validator 3 + /// twice in the slot's committee (every other seat is validator 0), with + /// empty vote vectors. + fn payload_attestation_store(root: Root) -> Store { let mut store = store_anchored_at(root); store .insert_signed_block( @@ -8112,6 +8309,14 @@ mod tests { store.insert_state(root, state).unwrap(); store.set_payload_timeliness_vote(root, vec![None; preset::PTC_SIZE]); store.set_payload_data_availability_vote(root, vec![None; preset::PTC_SIZE]); + store + } + + #[test] + fn on_payload_attestation_message_writes_every_seat_the_validator_holds() { + let config = Config::active().with_fork_epoch(ForkName::Gloas, 0); + let root = Root::repeat_byte(0xb1); + let mut store = payload_attestation_store(root); let message = |validator_index: u64, slot: Slot| gloas::PayloadAttestationMessage { validator_index, @@ -8159,6 +8364,178 @@ mod tests { ); } + fn payload_attestation_message( + root: Root, + validator_index: u64, + slot: Slot, + ) -> gloas::PayloadAttestationMessage { + gloas::PayloadAttestationMessage { + validator_index, + data: gloas::PayloadAttestationData { + beacon_block_root: root, + slot, + payload_present: true, + blob_data_available: true, + }, + signature: Default::default(), + } + } + + /// A gossip-verified vote is applied to every seat of its validator + /// without a signature check (the zero signature here would fail one), + /// but still only in the slot it is for, and only from a committee member. + #[test] + fn apply_verified_payload_attestation_skips_the_signature_but_not_the_slot() { + let config = Config::active().with_fork_epoch(ForkName::Gloas, 0); + let root = Root::repeat_byte(0xb1); + let mut store = payload_attestation_store(root); + let seated = |store: &Store| -> Vec { + store + .payload_timeliness_vote(&root) + .unwrap() + .iter() + .enumerate() + .filter_map(|(seat, vote)| vote.map(|_| seat)) + .collect() + }; + + // The store clock reads slot 0: a vote for slot 1 is not current yet. + assert_eq!( + failed_assertion(apply_verified_payload_attestation( + &mut store, + &payload_attestation_message(root, 3, 1), + &config + )), + Some("data.slot == get_current_slot(store)") + ); + assert!(seated(&store).is_empty()); + + store + .set_time_ms(config.slot_duration_ms) + .expect("set the clock to slot 1"); + // A validator outside the committee still writes nothing. + assert_eq!( + failed_assertion(apply_verified_payload_attestation( + &mut store, + &payload_attestation_message(root, 2, 1), + &config + )), + Some("len(ptc_indices) > 0") + ); + apply_verified_payload_attestation( + &mut store, + &payload_attestation_message(root, 3, 1), + &config, + ) + .unwrap(); + assert_eq!(seated(&store), vec![2, 7]); + + // The unverified entry point refuses the same message: its signature + // is not valid. + let mut fresh = payload_attestation_store(root); + fresh + .set_time_ms(config.slot_duration_ms) + .expect("set the clock to slot 1"); + assert!( + on_payload_attestation_message( + &mut fresh, + &payload_attestation_message(root, 3, 1), + false, + &config + ) + .is_err() + ); + } + + /// A restarted follower holds a verified payload (restored from its table) + /// but no payload-committee votes, which are never persisted. Both the + /// head walk (`payload_timeliness` on the tiebreaker path) and the first + /// child's payload attestations (`notify_ptc_messages` ends in this same + /// vote update) read those vectors, so resume must reseed them empty. + #[test] + fn a_resumed_store_walks_the_head_and_takes_payload_attestations() { + let config = Config::active().with_fork_epoch(ForkName::Gloas, 0); + let anchor_root = Root::repeat_byte(0xb0); + let root = Root::repeat_byte(0xb1); + let backend = Arc::new(InMemoryBackend::new()); + let mut store = store_anchored_at_on(backend.clone(), anchor_root); + store + .insert_signed_block( + anchor_root, + gloas_block( + 0, + Root::ZERO, + ExecutionBlockHash::ZERO, + ExecutionBlockHash::ZERO, + ), + ) + .unwrap(); + store + .insert_signed_block( + root, + gloas_block( + 1, + anchor_root, + ExecutionBlockHash::repeat_byte(0xff), + ExecutionBlockHash::repeat_byte(0x22), + ), + ) + .unwrap(); + let mut state = test_state::with_validators_at(ForkName::Gloas, 4); + store.insert_state(anchor_root, state.clone()).unwrap(); + let BeaconState::Gloas(inner) = &mut state else { + unreachable!("with_validators_at(Gloas) builds a gloas state"); + }; + inner.slot = 1; + let mut committee = vec![0; preset::PTC_SIZE]; + committee[2] = 3; + let window_index = (preset::SLOTS_PER_EPOCH + 1) as usize; + inner.ptc_window[window_index] = committee.try_into().unwrap(); + store.insert_state(root, state).unwrap(); + // What `on_block` records for each gloas block, then the envelope. + for block_root in [anchor_root, root] { + store.set_gloas_block_timeliness(block_root, [true, true]); + store.set_payload_timeliness_vote(block_root, vec![None; preset::PTC_SIZE]); + store.set_payload_data_availability_vote(block_root, vec![None; preset::PTC_SIZE]); + } + verify_payload(&mut store, root); + // Dropping the last handle lets the state writer finish its queue. + drop(store); + + let mut store = Store::from_db_state(backend) + .expect("reopen") + .expect("populated directory"); + assert!(store.has_verified_payload(&root)); + + // The slot after the block's own: the head walk asks whether the + // verified payload is timely, which reads the votes. + store.set_time_ms(2 * config.slot_duration_ms).unwrap(); + let committees = CommitteeCache::default(); + let walked = gloas_get_head(&store, &config, &committees) + .expect("a resumed store must walk the head over a verified payload"); + // The actor's own path: `walk_head` with the payload links re-derived + // (none survive a restart) and the boost and weight helpers. + let head = get_head_node(&store, &config) + .expect("the live head walk must succeed on a resumed store"); + assert_eq!(head.root, walked.root); + assert_eq!(get_head(&mut store, &config).unwrap(), walked.root); + assert_eq!(store.head_payload_status(), Some(walked.payload_status)); + + let message = gloas::PayloadAttestationMessage { + validator_index: 3, + data: gloas::PayloadAttestationData { + beacon_block_root: root, + slot: 1, + payload_present: true, + blob_data_available: true, + }, + signature: Default::default(), + }; + on_payload_attestation_message(&mut store, &message, true, &config) + .expect("the first child's attestations must find the vote vectors"); + assert_eq!(store.payload_timeliness_vote(&root).unwrap()[2], Some(true)); + } + #[test] fn get_forkchoice_store_seeds_a_gloas_anchor_and_leaves_an_earlier_one_alone() { // An exact anchor pair, built as `anchor_pair_with` builds a phase0 one. @@ -8376,7 +8753,7 @@ mod tests { .unwrap(); store.insert_state(root, gloas_state.clone()).unwrap(); if rng.chance(1, 2) { - store.insert_verified_payload(root); + verify_payload(&mut store, root); } for timely_votes in [true, false] { let votes = match rng.below(4) { diff --git a/crates/blockchain/state_transition/src/beacon/gossip/aggregate.rs b/crates/blockchain/state_transition/src/beacon/gossip/aggregate.rs index 260c7e794..7578ce0d4 100644 --- a/crates/blockchain/state_transition/src/beacon/gossip/aggregate.rs +++ b/crates/blockchain/state_transition/src/beacon/gossip/aggregate.rs @@ -1,6 +1,9 @@ //! `beacon_aggregate_and_proof` gossip validation: electra's //! `validate_beacon_aggregate_and_proof_gossip` (`specs/electra/p2p-interface.md`), -//! which fulu (what mainnet runs) inherits unchanged. +//! which fulu inherits unchanged, and gloas's modification of it +//! (`specs/gloas/p2p-interface.md`). The aggregate's variant picks the +//! `data.index` rule and whether the payload-status check runs; it runs last, +//! as in the specification. //! //! # Deviations from the specification //! @@ -54,6 +57,7 @@ use lru::LruCache; use super::{ IgnoreReason, Outcome, RejectReason, ancestor_at, is_current_or_previous_epoch, is_future_slot, + verify_attestation_payload_status, }; use crate::beacon::bls; use crate::beacon::constants::{ @@ -67,11 +71,17 @@ use crate::beacon::helpers::math::bytes_to_uint64; use crate::beacon::helpers::misc::{ compute_epoch_at_slot, compute_signing_root, compute_start_slot_at_epoch, }; +use crate::beacon::preset; use crate::beacon::primitives::{ BlsSignature, CommitteeIndex, Epoch, HashTreeRoot as _, Root, ValidatorIndex, }; use ethlambda_storage::CacheKey; +/// The most aggregation bits a gloas aggregate may carry: electra's type +/// bound (every committee of a slot at full size). Gloas's progressive bitlist +/// has no bound of its own, so the gossip rules enforce this one. +const MAX_AGGREGATION_BITS: usize = preset::MAX_VALIDATORS_PER_SLOT; + /// One committee's worth of aggregation bits, packed a `u64` at a time. /// /// The specification's `Seen.aggregate_data_roots` stores each accepted @@ -240,6 +250,7 @@ fn aggregate_and_proof_root(aggregate: &SignedAggregateAndProof) -> Root { match aggregate { SignedAggregateAndProof::Phase0(signed) => signed.message.hash_tree_root(), SignedAggregateAndProof::Electra(signed) => signed.message.hash_tree_root(), + SignedAggregateAndProof::Gloas(signed) => signed.message.hash_tree_root(), } } @@ -256,12 +267,36 @@ pub fn cheap_checks( let config = store.config(); let data = aggregate.data(); - // [New in Electra:EIP7549] [REJECT] `data.index` is zero: the committee - // now lives in `committee_bits` instead. Phase0 has no `committee_bits` - // and carries the real committee in `data.index`, so this only applies - // to the electra shape. - if matches!(aggregate, SignedAggregateAndProof::Electra(_)) && data.index != 0 { - return Err(Outcome::Reject(RejectReason::NonZeroDataIndex)); + // [REJECT] The bitfield fits the type bound electra gave it. Gloas's + // `AggregationBits` is a progressive bitlist with no bound of its own, and + // everything below (the seen-set verdict first) expands the bits, so an + // oversized one is turned away before anything reads them. The length is + // read without expanding the list. Earlier forks' bitlists carry this bound + // in their type. + if matches!(aggregate, SignedAggregateAndProof::Gloas(_)) + && aggregate.aggregation_bits_len() > MAX_AGGREGATION_BITS + { + return Err(Outcome::Reject(RejectReason::BitsLength)); + } + + match aggregate { + // Phase0 has no `committee_bits` and carries the real committee in + // `data.index`, so there is nothing to require of it. + SignedAggregateAndProof::Phase0(_) => {} + // [New in Electra:EIP7549] [REJECT] `data.index` is zero: the committee + // now lives in `committee_bits` instead. + SignedAggregateAndProof::Electra(_) => { + if data.index != 0 { + return Err(Outcome::Reject(RejectReason::NonZeroDataIndex)); + } + } + // [New in Gloas:EIP7732] [REJECT] `data.index` is 0 or 1: it is the + // payload-present flag now. + SignedAggregateAndProof::Gloas(_) => { + if data.index > 1 { + return Err(Outcome::Reject(RejectReason::DataIndexOutOfRange)); + } + } } // [New in Electra:EIP7549] [REJECT] Exactly one committee is named. // Always `Some` for phase0, whose `data.index` alone names the committee. @@ -426,6 +461,19 @@ pub fn stateful_checks( } indexed.attesting_indices.to_vec() } + SignedAggregateAndProof::Gloas(signed) => { + let gloas_attestation = &signed.message.aggregate; + let indexed = crate::beacon::helpers::gloas::get_indexed_attestation( + &state, + gloas_attestation, + &committees, + ) + .map_err(|_| Outcome::Ignore(IgnoreReason::Internal))?; + if !crate::beacon::helpers::gloas::is_valid_indexed_attestation(&state, &indexed) { + return Err(Outcome::Reject(RejectReason::AggregateSignature)); + } + indexed.attesting_indices.to_vec() + } }; // Ancestry, via the vote state's own history; see this function's @@ -449,6 +497,12 @@ pub fn stateful_checks( return Err(Outcome::Ignore(IgnoreReason::FinalizedNotAncestor)); } + // [New in Gloas:EIP7732] The attested payload status is consistent with the + // block's execution payload. + if matches!(aggregate, SignedAggregateAndProof::Gloas(_)) { + verify_attestation_payload_status(store, &data)?; + } + Ok(attesting_indices) } @@ -518,6 +572,54 @@ mod tests { } } + #[test] + fn a_gloas_aggregate_with_an_oversized_bitfield_is_rejected_first() { + use crate::beacon::containers::gloas; + + let store = store(0); + let now_ms = slot_start_ms(&store, 1); + let mut committee_bits = electra::CommitteeBits::default(); + committee_bits + .set(0, true) + .expect("within MAX_COMMITTEES_PER_SLOT"); + let aggregate_with_len = |len: usize| { + SignedAggregateAndProof::Gloas(gloas::SignedAggregateAndProof { + message: gloas::AggregateAndProof { + aggregator_index: 0, + aggregate: gloas::Attestation { + aggregation_bits: gloas::AggregationBits::with_length(len), + data: shared_data(0), + signature: Default::default(), + committee_bits: committee_bits.clone(), + }, + selection_proof: Default::default(), + }, + signature: Default::default(), + }) + }; + + assert_eq!( + cheap_checks( + &seen_aggregates(), + &store, + &aggregate_with_len(MAX_AGGREGATION_BITS + 1), + now_ms + ), + Err(Outcome::Reject(RejectReason::BitsLength)) + ); + // At the bound it is no longer this check's to reject; it goes on to + // the participants rule, which an empty bitfield fails. + assert_eq!( + cheap_checks( + &seen_aggregates(), + &store, + &aggregate_with_len(MAX_AGGREGATION_BITS), + now_ms + ), + Err(Outcome::Reject(RejectReason::NoParticipants)) + ); + } + // -- PackedBits / superset semantics -------------------------------- #[test] diff --git a/crates/blockchain/state_transition/src/beacon/gossip/attestation.rs b/crates/blockchain/state_transition/src/beacon/gossip/attestation.rs index 4db9cd8bc..bb1975791 100644 --- a/crates/blockchain/state_transition/src/beacon/gossip/attestation.rs +++ b/crates/blockchain/state_transition/src/beacon/gossip/attestation.rs @@ -1,6 +1,8 @@ //! `beacon_attestation_{subnet_id}` gossip validation: electra's modified //! `validate_beacon_attestation_gossip` (`specs/electra/p2p-interface.md`), -//! which fulu (what mainnet runs) inherits unchanged. +//! which fulu inherits unchanged, and gloas's modification of it +//! (`specs/gloas/p2p-interface.md`), picked by the attestation's slot since +//! gloas keeps [`SingleAttestation`]. //! //! EIP-7549 replaced the wire type for this topic with [`SingleAttestation`]: //! one attester's vote, with its committee named explicitly @@ -43,6 +45,7 @@ use lru::LruCache; use super::{ IgnoreReason, Outcome, RejectReason, ancestor_at, is_current_or_previous_epoch, is_future_slot, + is_gloas_slot, verify_attestation_payload_status, }; use crate::beacon::bls; use crate::beacon::config::Config; @@ -134,9 +137,15 @@ pub fn cheap_checks( if seen.contains(target_epoch, attestation.attester_index) { return Err(Outcome::Ignore(IgnoreReason::AlreadySeen)); } - // [New in Electra:EIP7549] [REJECT] `data.index` is zero: the committee - // now travels in `committee_index` instead. - if data.index != 0 { + if is_gloas_slot(&config, data.slot) { + // [New in Gloas:EIP7732] [REJECT] `data.index` is 0 or 1: it is the + // payload-present flag now. + if data.index > 1 { + return Err(Outcome::Reject(RejectReason::DataIndexOutOfRange)); + } + } else if data.index != 0 { + // [New in Electra:EIP7549] [REJECT] `data.index` is zero: the committee + // now travels in `committee_index` instead. return Err(Outcome::Reject(RejectReason::NonZeroDataIndex)); } // [IGNORE] Not from a future slot. @@ -231,6 +240,14 @@ pub fn stateful_checks(store: &Store, attestation: &SingleAttestation, subnet_id return Outcome::Ignore(IgnoreReason::FinalizedNotAncestor); } + // [New in Gloas:EIP7732] The attested payload status is consistent with the + // block's execution payload. + if is_gloas_slot(&store.config(), data.slot) + && let Err(outcome) = verify_attestation_payload_status(store, data) + { + return outcome; + } + Outcome::Accept } diff --git a/crates/blockchain/state_transition/src/beacon/gossip/block.rs b/crates/blockchain/state_transition/src/beacon/gossip/block.rs index fc8168670..76bae3c62 100644 --- a/crates/blockchain/state_transition/src/beacon/gossip/block.rs +++ b/crates/blockchain/state_transition/src/beacon/gossip/block.rs @@ -1,43 +1,38 @@ //! `beacon_block` gossip validation: fulu's `validate_beacon_block_gossip` -//! (`specs/fulu/p2p-interface.md`), with the deviations the design spec lists. +//! (`specs/fulu/p2p-interface.md`) and gloas's modified one +//! (`specs/gloas/p2p-interface.md`), with the deviations the design spec lists. use ethlambda_storage::CacheKey; +use tracing::warn; use super::{ - IgnoreReason, Outcome, QueueReason, RejectReason, SeenBlocks, finalized_ancestry, - finalized_start_slot, is_future_slot, + IgnoreReason, Outcome, QueueReason, RejectReason, SeenBlocks, execution_requests_within_limits, + finalized_ancestry, finalized_start_slot, is_future_slot, }; -use crate::beacon::containers::SignedBeaconBlock; -use crate::beacon::fork_choice::Store; +use crate::beacon::containers::{BeaconState, SignedBeaconBlock, gloas}; +use crate::beacon::fork_choice::{self, Store}; use crate::beacon::helpers::misc::compute_epoch_at_slot; +use crate::beacon::lean_boundary::{lean_block_unreachable, lean_state_unreachable}; use crate::beacon::precheck::{self, PrecheckError, Reference, precheck_block}; -use crate::beacon::primitives::Root; +use crate::beacon::preset; +use crate::beacon::primitives::{ExecutionBlockHash, Root}; use crate::beacon::stf::bellatrix::compute_timestamp_at_slot; /// The rules that read only the block, the clock and the store's metadata. /// /// `Err` carries the verdict; `Ok` sends the block on to [`stateful_checks`]. +/// A gloas block runs [`cheap_checks_gloas`] instead. pub fn cheap_checks( seen: &SeenBlocks, store: &Store, block: &SignedBeaconBlock, now_ms: u64, ) -> Result<(), Outcome> { + if let Some(gloas_block) = as_gloas(block, "block::cheap_checks") { + return cheap_checks_gloas(seen, store, gloas_block, now_ms); + } let config = store.config(); let slot = block.slot(); - // [IGNORE] This build does not validate a gloas-shaped block against - // these (fulu) rules: EIP-7732 changes what a block commits to enough - // that accepting one here would propagate something this node cannot - // itself follow (its node wiring cannot deliver a gloas block's payload - // envelope yet, though `fork_choice::on_block` accepts the block). Checked - // first, so a gloas block never reaches a rule below whose fields it - // happens to still answer (its `blob_kzg_commitment_count` reads the - // builder's bid) but whose verdict would misdescribe it. `Ignore`, not - // `Reject`: the sender did nothing wrong, and every honest peer sends - // exactly this once this node's own clock reaches gloas. - if !block.fork_name().is_followed() { - return Err(Outcome::Ignore(IgnoreReason::UnsupportedFork)); - } // [IGNORE] The block is not from a future slot. if is_future_slot(&config, slot, now_ms) { return Err(Outcome::Ignore(IgnoreReason::FutureSlot)); @@ -59,12 +54,93 @@ pub fn cheap_checks( Ok(()) } +/// `block` as a gloas block, or `None` for every earlier fork. +/// +/// Spelled out arm by arm so that a new fork breaks this match rather than +/// silently taking fulu's rules. +fn as_gloas<'a>( + block: &'a SignedBeaconBlock, + function: &str, +) -> Option<&'a gloas::SignedBeaconBlock> { + match block { + SignedBeaconBlock::Gloas(block) => Some(block), + SignedBeaconBlock::Phase0(_) + | SignedBeaconBlock::Altair(_) + | SignedBeaconBlock::Bellatrix(_) + | SignedBeaconBlock::Capella(_) + | SignedBeaconBlock::Deneb(_) + | SignedBeaconBlock::Electra(_) + | SignedBeaconBlock::Fulu(_) => None, + SignedBeaconBlock::Lean(_) => lean_block_unreachable(function), + } +} + +/// The gloas rules that read only the block, the clock and the store's +/// metadata: the first half of gloas's modified `validate_beacon_block_gossip` +/// (`specs/gloas/p2p-interface.md`), in the specification's order. +/// +/// The blob-count rule sits later in the specification, after the parent +/// lookups, but reads only the bid; it runs here for the reason fulu's does, +/// so a block with too many commitments is refused before any state is read. +fn cheap_checks_gloas( + seen: &SeenBlocks, + store: &Store, + signed_block: &gloas::SignedBeaconBlock, + now_ms: u64, +) -> Result<(), Outcome> { + let config = store.config(); + let block = &signed_block.message; + // [IGNORE] The block is the first with a valid signature for its slot and + // proposer. + if seen.contains(block.slot, block.proposer_index) { + return Err(Outcome::Ignore(IgnoreReason::AlreadySeen)); + } + // [IGNORE] The block is not from a future slot. + if is_future_slot(&config, block.slot, now_ms) { + return Err(Outcome::Ignore(IgnoreReason::FutureSlot)); + } + // [IGNORE] The block is from a slot greater than the latest finalized slot. + if block.slot <= finalized_start_slot(store) { + return Err(Outcome::Ignore(IgnoreReason::Finalized)); + } + // [REJECT] The body's operation counts, and the parent execution + // requests' counts, are within their limits. + if !body_operations_within_limits(&block.body) + || !execution_requests_within_limits(&block.body.parent_execution_requests) + { + return Err(Outcome::Reject(RejectReason::OperationLimit)); + } + // [REJECT] The bid's blob KZG commitment count is within the per-epoch + // limit. + let bid = &block.body.signed_execution_payload_bid.message; + let max_blobs = config.max_blobs_per_block(compute_epoch_at_slot(block.slot)); + if bid.blob_kzg_commitments.len() as u64 > max_blobs { + return Err(Outcome::Reject(RejectReason::TooManyBlobs)); + } + Ok(()) +} + +/// `verify_block_body_operation_limits` (`specs/gloas/p2p-interface.md`). +/// +/// Gloas's operation lists are progressive, so unlike every earlier fork's +/// they carry no bound of their own and decoding does not enforce it. +fn body_operations_within_limits(body: &gloas::BeaconBlockBody) -> bool { + body.proposer_slashings.len() <= preset::MAX_PROPOSER_SLASHINGS + && body.attester_slashings.len() <= preset::MAX_ATTESTER_SLASHINGS_ELECTRA + && body.attestations.len() <= preset::MAX_ATTESTATIONS_ELECTRA + && body.deposits.is_empty() + && body.voluntary_exits.len() <= preset::MAX_VOLUNTARY_EXITS + && body.bls_to_execution_changes.len() <= preset::MAX_BLS_TO_EXECUTION_CHANGES + && body.payload_attestations.len() as u64 <= preset::MAX_PAYLOAD_ATTESTATIONS +} + /// The rules that need the parent's post-state. Runs on a blocking thread. /// /// Reads states only through [`Store::cached_state`]: a miss would otherwise /// rebuild the state from diffs, which is far too slow for a verdict that /// gossipsub waits on. A miss queues the block instead. pub fn stateful_checks(store: &Store, block: &SignedBeaconBlock, block_root: Root) -> Outcome { + let gloas_block = as_gloas(block, "block::stateful_checks"); let config = store.config(); let parent_root = block.parent_root(); let parent_known = store.has_block(&parent_root); @@ -90,12 +166,51 @@ pub fn stateful_checks(store: &Store, block: &SignedBeaconBlock, block_root: Roo if let Err(err) = precheck_block(block, block_root, Reference::Parent(&parent_state), &config) { return Outcome::Reject(err.into()); } + // [IGNORE] (gloas) If the parent block is full, the parent payload is + // valid. The specification checks this between the signature and the + // slot-order rule, but the slot-order and expected-proposer rules are both + // inside `precheck_block` here, so a block that fails either is rejected + // even with an unverified parent payload. The parent's fullness is read + // once: it decodes the parent block, which the bounded validation pool + // should not do twice. + let parent_full = if gloas_block.is_some() { + match fork_choice::is_parent_node_full(store, block) { + Ok(full) => full, + Err(err) => { + warn!(%err, "Could not read a gloas block's parent payload status"); + return Outcome::Ignore(IgnoreReason::Internal); + } + } + } else { + true + }; + if gloas_block.is_some() && parent_full && !fork_choice::is_payload_verified(store, parent_root) + { + return Outcome::Ignore(IgnoreReason::ParentPayloadUnverified); + } // [REJECT] The finalized checkpoint is an ancestor of the block. if let Err(outcome) = finalized_ancestry(store, parent_root).verdict() { return outcome; } - // [REJECT] The execution payload's timestamp is the slot's. - if let Some(timestamp) = block.execution_payload_timestamp() + // A gloas block has no payload to time; earlier forks: [REJECT] the + // execution payload's timestamp is the slot's. + if let Some(gloas_block) = gloas_block { + // [REJECT] The bid's parent equals the block's parent. + let bid = &gloas_block + .message + .body + .signed_execution_payload_bid + .message; + if bid.parent_block_root != gloas_block.message.parent_root { + return Outcome::Reject(RejectReason::BidParentMismatch); + } + // [REJECT] If the parent is not full, the bid builds on the parent's + // execution head. `process_slots` leaves `latest_block_hash` alone, + // so the parent's post-state answers for the advanced one. + if !parent_full && latest_block_hash(&parent_state) != Some(bid.parent_block_hash) { + return Outcome::Reject(RejectReason::BidNotOnParentHead); + } + } else if let Some(timestamp) = block.execution_payload_timestamp() && timestamp != compute_timestamp_at_slot(&parent_state, block.slot(), &config) { return Outcome::Reject(RejectReason::PayloadTimestamp); @@ -110,6 +225,22 @@ pub fn stateful_checks(store: &Store, block: &SignedBeaconBlock, block_root: Roo Outcome::Accept } +/// `state.latest_block_hash`, for a gloas state. `None` for an earlier fork, +/// which has none: its parent counts as full, so no caller asks. +fn latest_block_hash(state: &BeaconState) -> Option { + match state { + BeaconState::Gloas(state) => Some(state.latest_block_hash), + BeaconState::Phase0(_) + | BeaconState::Altair(_) + | BeaconState::Bellatrix(_) + | BeaconState::Capella(_) + | BeaconState::Deneb(_) + | BeaconState::Electra(_) + | BeaconState::Fulu(_) => None, + BeaconState::Lean(_) => lean_state_unreachable("block::latest_block_hash"), + } +} + /// `cheap_checks` then `stateful_checks`, the order the p2p actor runs them in. /// For callers that have no reason to split them, such as the spec vectors. pub fn validate( @@ -162,7 +293,8 @@ mod tests { use crate::beacon::bls; use crate::beacon::config::Config; use crate::beacon::constants::{ - DOMAIN_BEACON_PROPOSER, MAXIMUM_GOSSIP_CLOCK_DISPARITY as DISPARITY, + DEPOSIT_CONTRACT_TREE_DEPTH, DOMAIN_BEACON_PROPOSER, + MAXIMUM_GOSSIP_CLOCK_DISPARITY as DISPARITY, }; use crate::beacon::containers::{BeaconState, electra}; use crate::beacon::fork_choice; @@ -250,6 +382,61 @@ mod tests { sign_block(SignedBeaconBlock::Fulu(inner), parent, config, proposer) } + #[test] + fn a_gloas_body_is_limited_per_operation_and_carries_no_deposits() { + let body = gloas::BeaconBlockBody::default(); + assert!(body_operations_within_limits(&body)); + + let mut with_deposit = body.clone(); + let deposit = crate::beacon::containers::Deposit { + proof: vec![Root::ZERO; DEPOSIT_CONTRACT_TREE_DEPTH + 1] + .try_into() + .expect("a deposit proof has the contract tree's depth plus one roots"), + data: Default::default(), + }; + with_deposit.deposits = vec![deposit] + .try_into() + .expect("progressive lists are unbounded"); + assert!(!body_operations_within_limits(&with_deposit)); + + let mut too_many_exits = body; + too_many_exits.voluntary_exits = vec![Default::default(); preset::MAX_VOLUNTARY_EXITS + 1] + .try_into() + .expect("progressive lists are unbounded"); + assert!(!body_operations_within_limits(&too_many_exits)); + } + + #[test] + fn deposit_requests_are_not_limited_by_gossip() { + let request = crate::beacon::containers::electra::DepositRequest { + pubkey: Default::default(), + withdrawal_credentials: Default::default(), + amount: 0, + signature: Default::default(), + index: 0, + }; + let requests = gloas::ExecutionRequests { + deposits: vec![request; preset::MAX_DEPOSIT_REQUESTS_PER_PAYLOAD + 1] + .try_into() + .expect("progressive lists are unbounded"), + ..Default::default() + }; + assert!(execution_requests_within_limits(&requests)); + } + + #[test] + fn gloas_execution_requests_are_limited_per_list() { + let requests = gloas::ExecutionRequests::default(); + assert!(execution_requests_within_limits(&requests)); + + let mut too_many = requests; + too_many.builder_exits = + vec![Default::default(); preset::MAX_BUILDER_EXIT_REQUESTS_PER_PAYLOAD as usize + 1] + .try_into() + .expect("progressive lists are unbounded"); + assert!(!execution_requests_within_limits(&too_many)); + } + #[test] fn a_block_within_the_clock_disparity_passes() { let store = store(0); diff --git a/crates/blockchain/state_transition/src/beacon/gossip/column.rs b/crates/blockchain/state_transition/src/beacon/gossip/column.rs index 2d7130abd..a607ec58e 100644 --- a/crates/blockchain/state_transition/src/beacon/gossip/column.rs +++ b/crates/blockchain/state_transition/src/beacon/gossip/column.rs @@ -1,29 +1,39 @@ //! `data_column_sidecar_{subnet_id}` gossip validation: fulu's -//! `validate_data_column_sidecar_gossip` (`specs/fulu/p2p-interface.md`). +//! `validate_data_column_sidecar_gossip` (`specs/fulu/p2p-interface.md`), and +//! gloas's modified one (`specs/gloas/p2p-interface.md`) in the `_gloas` +//! functions beside it. //! //! Also every check the chain relies on before it keeps a sidecar gossip did //! not accept ([`chain_checks`]): one fetched over req/resp, one gossip queued -//! or had no free permit for, or a parked one whose parent has since -//! imported. The chain actor stores whatever reaches it without checking it +//! or had no free permit for, or a parked one whose block has since imported +//! (its parent, for a fulu sidecar; its own block, for a gloas one). The chain +//! actor stores whatever reaches it without checking it //! again, so these are the only checks such a sidecar gets. use ethlambda_storage::CacheKey; use super::{ - IgnoreReason, Outcome, QueueReason, RejectReason, SeenColumns, finalized_ancestry, - finalized_start_slot, is_future_slot, + IgnoreReason, Outcome, QueueReason, RejectReason, SeenBlockColumns, SeenColumns, + finalized_ancestry, finalized_start_slot, is_future_slot, }; use crate::beacon::bls; use crate::beacon::config::Config; use crate::beacon::constants::{DATA_COLUMN_SIDECAR_SUBNET_COUNT, DOMAIN_BEACON_PROPOSER}; use crate::beacon::containers::BeaconState; +use crate::beacon::containers::DataColumnSidecar as AnyDataColumnSidecar; +use crate::beacon::containers::SignedBeaconBlock; use crate::beacon::containers::fulu::DataColumnSidecar; +use crate::beacon::containers::gloas; use crate::beacon::fork_choice::{self, Store}; use crate::beacon::helpers::accessors::get_beacon_proposer_index; -use crate::beacon::helpers::misc::{compute_domain, compute_epoch_at_slot, compute_signing_root}; +use crate::beacon::helpers::misc::{ + compute_domain, compute_epoch_at_slot, compute_signing_root, compute_start_slot_at_epoch, +}; +use crate::beacon::lean_boundary::lean_block_unreachable; use crate::beacon::precheck; +use crate::beacon::preset; use crate::beacon::primitives::HashTreeRoot as _; -use crate::beacon::primitives::{Slot, ValidatorIndex}; +use crate::beacon::primitives::{Root, Slot, ValidatorIndex}; use crate::beacon::stf; use crate::metrics; @@ -100,8 +110,10 @@ pub fn stateful_checks(store: &Store, sidecar: &DataColumnSidecar) -> Outcome { pub enum ChainVerdict { /// Passed every rule: the chain stores it as it is. Keep, - /// Its parent has no post-state yet. The chain parks it and sends it back - /// through [`chain_checks`] once the parent imports. + /// The block it is judged against has no post-state yet: its parent for a + /// fulu sidecar, its own block for a gloas one (whose commitments are in + /// that block's bid). The chain parks it and sends it back through + /// [`chain_checks_for`] once that block imports. AwaitParent, /// Not kept. The outcome says why: an `Ignore` or a `Reject`, or /// `Queue(ParentNotReady)` when the parent has a post-state but its @@ -165,6 +177,67 @@ pub fn chain_checks(store: &Store, sidecar: &DataColumnSidecar, now_ms: u64) -> } } +/// [`chain_checks`] or [`chain_checks_gloas`], by the sidecar's shape. +pub fn chain_checks_for( + store: &Store, + sidecar: &AnyDataColumnSidecar, + now_ms: u64, +) -> ChainVerdict { + match sidecar { + AnyDataColumnSidecar::Fulu(sidecar) => chain_checks(store, sidecar, now_ms), + AnyDataColumnSidecar::Gloas(sidecar) => chain_checks_gloas(store, sidecar, now_ms), + } +} + +/// [`chain_checks`] for a gloas sidecar: the gloas gossip rules minus the two +/// that only mean something on a gossip topic (the subnet match and the seen +/// cache). +/// +/// What the stateful half cannot yet judge is [`ChainVerdict::AwaitParent`]: +/// a block not seen, or seen without a post-state. Unlike a fulu header, a +/// gloas sidecar carries no signature, so nothing here can refuse a forged one +/// before it is parked; the future-slot and finalized rules bound how long a +/// made-up key can sit there. +pub fn chain_checks_gloas( + store: &Store, + sidecar: &gloas::DataColumnSidecar, + now_ms: u64, +) -> ChainVerdict { + let config = store.config(); + // [IGNORE] Not from a future slot. Also what bounds how far ahead a + // parked row can name. + if is_future_slot(&config, sidecar.slot, now_ms) { + return ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::FutureSlot)); + } + // [IGNORE] From a slot greater than the latest finalized slot, unless it + // is a column of the finalized block itself. Not a gloas rule: gloas has + // no finalized bound on sidecars, and this one stays only to bound what a + // parked row can name. The finalized block is exempt because its envelope + // can still be waiting on these columns: a checkpoint-synced anchor is the + // finalized block, and no child building on its payload imports until its + // sampled columns arrive. Matched by root rather than by slot, since an + // anchor state advanced past empty slots finalizes a block from before its + // epoch's start slot. A column naming that root is never parked: the block + // and its post-state are both stored, so it is judged at once. + let finalized = store.beacon_finalized_checkpoint(); + if sidecar.slot <= compute_start_slot_at_epoch(finalized.epoch) + && sidecar.beacon_block_root != finalized.root + { + return ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::Finalized)); + } + // [IGNORE] Already stored, as in [`chain_checks`]. + if store.has_data_column(sidecar.slot, &sidecar.beacon_block_root, sidecar.index) { + return ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::AlreadyStored)); + } + match stateful_checks_gloas(store, sidecar) { + Outcome::Accept => ChainVerdict::Keep, + Outcome::Queue(QueueReason::BlockUnknown | QueueReason::BlockNotReady) => { + ChainVerdict::AwaitParent + } + outcome => ChainVerdict::Drop(outcome), + } +} + /// What [`judge_against_parent`] does when the parent state's proposer /// lookahead has no answer for the sidecar's slot. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -306,6 +379,163 @@ pub fn validate( stateful_checks(store, sidecar) } +/// The gloas rules that read only the sidecar, the clock and the store's +/// metadata: the half [`validate_gloas`] runs first, beside fulu's +/// [`cheap_checks`]. +/// +/// Split from [`stateful_checks_gloas`] for the reason fulu's is: a duplicate +/// or a misrouted sidecar is dropped without taking a blocking-pool permit, +/// and the KZG batch stays off the swarm loop. `Err` carries the verdict; `Ok` +/// sends the sidecar on to [`stateful_checks_gloas`]. The caller records the +/// sidecar in `seen` once the whole rule answers [`Outcome::Accept`]. +pub fn cheap_checks_gloas( + seen: &SeenBlockColumns, + store: &Store, + sidecar: &gloas::DataColumnSidecar, + subnet_id: u64, + now_ms: u64, +) -> Result<(), Outcome> { + let config = store.config(); + // [IGNORE] The first sidecar seen for this block root and column index. + if seen.contains(sidecar.beacon_block_root, sidecar.index) { + return Err(Outcome::Ignore(IgnoreReason::AlreadySeen)); + } + // [REJECT] The sidecar is for the correct subnet. + if sidecar.index % DATA_COLUMN_SIDECAR_SUBNET_COUNT != subnet_id { + return Err(Outcome::Reject(RejectReason::WrongSubnet)); + } + // [IGNORE] The sidecar is not from a future slot. + if is_future_slot(&config, sidecar.slot, now_ms) { + return Err(Outcome::Ignore(IgnoreReason::FutureSlot)); + } + // [IGNORE] Already stored, fetched over req/resp before gossip delivered + // it. Not a rule of the specification: it saves the KZG batch. + if store.has_data_column(sidecar.slot, &sidecar.beacon_block_root, sidecar.index) { + return Err(Outcome::Ignore(IgnoreReason::AlreadyStored)); + } + Ok(()) +} + +/// The gloas rules that need the named block and its post-state, then the KZG +/// batch. Runs on a blocking thread, beside fulu's [`stateful_checks`]. +/// +/// A block the store has not seen is [`Outcome::Queue`], the specification's +/// "MAY be queued until block is retrieved". A block seen without a post-state +/// is queued too where the specification rejects it, the same deviation +/// fulu's parent rule makes until a bad-block cache exists. +pub fn stateful_checks_gloas(store: &Store, sidecar: &gloas::DataColumnSidecar) -> Outcome { + let block = match judgeable_block_gloas(store, &sidecar.beacon_block_root) { + Ok(block) => block, + Err(outcome) => return outcome, + }; + // [REJECT] The sidecar's slot matches the slot of the block. + if sidecar.slot != block.slot() { + return Outcome::Reject(RejectReason::SlotMismatch); + } + let kzg_commitments = match &block { + SignedBeaconBlock::Gloas(block) => { + &block + .message + .body + .signed_execution_payload_bid + .message + .blob_kzg_commitments + } + // A block of any other fork has no bid to read commitments from, so + // no sidecar of this shape can be valid for it. + SignedBeaconBlock::Phase0(_) + | SignedBeaconBlock::Altair(_) + | SignedBeaconBlock::Bellatrix(_) + | SignedBeaconBlock::Capella(_) + | SignedBeaconBlock::Deneb(_) + | SignedBeaconBlock::Electra(_) + | SignedBeaconBlock::Fulu(_) => return Outcome::Reject(RejectReason::BlockNotGloas), + SignedBeaconBlock::Lean(_) => lean_block_unreachable("column::stateful_checks_gloas"), + }; + // [REJECT] The sidecar passes structural validation. + if !fork_choice::gloas_verify_data_column_sidecar(sidecar, kzg_commitments) { + return Outcome::Reject(RejectReason::Malformed); + } + // [REJECT] The sidecar's column data passes KZG verification. + let kzg = { + let _timing = metrics::time_data_column_kzg_verify(); + fork_choice::gloas_verify_data_column_sidecar_kzg_proofs(sidecar, kzg_commitments) + }; + if !matches!(kzg, Ok(true)) { + return Outcome::Reject(RejectReason::Kzg); + } + Outcome::Accept +} + +/// The block a gloas sidecar is judged against, or the verdict that says why +/// there is none yet. +/// +/// The one predicate behind "this sidecar can be judged now": the chain actor +/// asks it before sending a parked sidecar back for checks, and +/// [`stateful_checks_gloas`] asks it first thing. Two predicates that differed +/// (a post-state without the block, say) would bounce a sidecar between the two +/// for ever, each side sure the other had what it needed. +pub fn judgeable_block_gloas(store: &Store, root: &Root) -> Result { + // [IGNORE] A block for the sidecar has been seen (MAY queue). + if !store.has_block(root) { + return Err(Outcome::Queue(QueueReason::BlockUnknown)); + } + // [REJECT] The block for the sidecar passes validation. A block seen + // without a post-state is queued rather than rejected, see above. Checked + // before the block is decoded, so one without a state costs no read of it. + match store.has_state(root) { + Ok(true) => {} + Ok(false) => return Err(Outcome::Queue(QueueReason::BlockNotReady)), + Err(_) => return Err(Outcome::Ignore(IgnoreReason::Internal)), + } + match store.get_signed_block(root) { + Ok(Some(block)) => Ok(block), + // Stored a moment ago and gone now (pruned): nothing to judge against. + Ok(None) => Err(Outcome::Queue(QueueReason::BlockUnknown)), + Err(_) => Err(Outcome::Ignore(IgnoreReason::Internal)), + } +} + +/// Whether a gloas sidecar is shaped well enough to be worth parking for its +/// block: the structural rules that need no block, checked before anything is +/// written to disk on its behalf. +/// +/// A gloas sidecar carries no signature, so nothing ties a parked one to a +/// real block; these are the only bounds on what a peer can make the node keep. +/// The slot must fall in gloas's epochs, the column index and the blob count +/// must be in range, the column and proof lists must agree, and the index must +/// be one this node custodies, since no other is ever stored. +pub fn is_parkable_gloas( + config: &Config, + custody_columns: &[u64], + sidecar: &gloas::DataColumnSidecar, +) -> bool { + let epoch = compute_epoch_at_slot(sidecar.slot); + config.fork_at_epoch(epoch).has_payload_envelopes() + && (sidecar.index as usize) < preset::NUMBER_OF_COLUMNS + && !sidecar.column.is_empty() + && sidecar.column.len() == sidecar.kzg_proofs.len() + && sidecar.column.len() as u64 <= config.max_blobs_per_block(epoch) + && custody_columns.contains(&sidecar.index) +} + +/// [`cheap_checks_gloas`] then [`stateful_checks_gloas`], for callers with no +/// reason to split them, such as the spec vectors: gloas's +/// `validate_data_column_sidecar_gossip` (`specs/gloas/p2p-interface.md`, +/// "Modified `data_column_sidecar_{subnet_id}`"), beside fulu's [`validate`]. +pub fn validate_gloas( + seen: &SeenBlockColumns, + store: &Store, + sidecar: &gloas::DataColumnSidecar, + subnet_id: u64, + now_ms: u64, +) -> Outcome { + if let Err(outcome) = cheap_checks_gloas(seen, store, sidecar, subnet_id, now_ms) { + return outcome; + } + stateful_checks_gloas(store, sidecar) +} + /// Whether `parent_state`'s proposer lookahead covers `slot` at all. /// /// Checked last in [`stateful_checks`], after every REJECT-worthy rule: the @@ -371,6 +601,7 @@ mod tests { use super::*; use crate::beacon::constants::MAXIMUM_GOSSIP_CLOCK_DISPARITY as DISPARITY; use crate::beacon::containers::{SignedBeaconBlock, electra, fulu, shared}; + use crate::beacon::fork::ForkName; use crate::beacon::gossip::test_support::{fulu_parent, seen_columns, slot_start_ms, store}; use crate::beacon::gossip::{IgnoreReason, Outcome, QueueReason, RejectReason}; use crate::beacon::helpers::misc::compute_start_slot_at_epoch; @@ -933,4 +1164,209 @@ mod tests { ChainVerdict::Drop(Outcome::Reject(RejectReason::WrongProposer)) ); } + + /// Empty column, so only the rules before the block lookup can be reached. + fn gloas_sidecar(slot: Slot, index: u64) -> gloas::DataColumnSidecar { + gloas::DataColumnSidecar { + index, + slot, + beacon_block_root: Root::from([7; 32]), + ..Default::default() + } + } + + fn block_columns() -> SeenBlockColumns { + SeenBlockColumns::new(std::num::NonZeroUsize::new(8).expect("non-zero")) + } + + #[test] + fn a_gloas_sidecar_for_another_subnet_is_rejected() { + let store = store(0); + let now = slot_start_ms(&store, 5); + assert_eq!( + validate_gloas(&block_columns(), &store, &gloas_sidecar(5, 1), 0, now), + Outcome::Reject(RejectReason::WrongSubnet) + ); + } + + #[test] + fn a_gloas_sidecar_from_a_future_slot_is_ignored() { + let store = store(0); + let now = slot_start_ms(&store, 5) - DISPARITY - 1; + assert_eq!( + validate_gloas(&block_columns(), &store, &gloas_sidecar(5, 0), 0, now), + Outcome::Ignore(IgnoreReason::FutureSlot) + ); + } + + #[test] + fn a_gloas_sidecar_for_an_unseen_block_is_queued() { + let store = store(0); + let now = slot_start_ms(&store, 5); + assert_eq!( + validate_gloas(&block_columns(), &store, &gloas_sidecar(5, 0), 0, now), + Outcome::Queue(QueueReason::BlockUnknown) + ); + } + + #[test] + fn a_gloas_sidecar_seen_for_its_block_root_and_index_is_ignored() { + let store = store(0); + let now = slot_start_ms(&store, 5); + let sidecar = gloas_sidecar(5, 0); + let mut seen = block_columns(); + seen.record(sidecar.beacon_block_root, sidecar.index); + assert_eq!( + validate_gloas(&seen, &store, &sidecar, 0, now), + Outcome::Ignore(IgnoreReason::AlreadySeen) + ); + } + + #[test] + fn a_gloas_sidecar_whose_block_has_no_post_state_is_queued() { + let mut store = store(0); + let card = gloas_sidecar(5, 0); + store + .insert_pending_block(card.beacon_block_root, parent_block(5)) + .expect("insert pending block"); + assert_eq!( + stateful_checks_gloas(&store, &card), + Outcome::Queue(QueueReason::BlockNotReady) + ); + } + + #[test] + fn an_already_stored_gloas_sidecar_is_ignored() { + let store = store(0); + let card = gloas_sidecar(5, 0); + store + .put_data_column_sidecar(card.slot, &card.beacon_block_root, card.index, Vec::new()) + .expect("store the column"); + assert_eq!( + cheap_checks_gloas(&block_columns(), &store, &card, 0, slot_start_ms(&store, 5)), + Err(Outcome::Ignore(IgnoreReason::AlreadyStored)) + ); + } + + /// A checkpoint-synced anchor is the finalized block, and its envelope + /// waits on its columns, so the finalized rule lets that block's own + /// columns through: at the finalized slot, and below it, where an anchor + /// state advanced past empty slots puts its block. Any other block's + /// columns there are still dropped. + #[test] + fn chain_checks_let_the_finalized_blocks_own_gloas_sidecars_through() { + let store = store(32); + let now = slot_start_ms(&store, 40); + let finalized_root = store.beacon_finalized_checkpoint().root; + for slot in [32, 30] { + let of_finalized = gloas::DataColumnSidecar { + beacon_block_root: finalized_root, + ..gloas_sidecar(slot, 0) + }; + // This store holds no block, so reaching the block lookup is as + // far as it goes. + assert_eq!( + chain_checks_gloas(&store, &of_finalized, now), + ChainVerdict::AwaitParent, + "the finalized block's column at slot {slot}" + ); + assert_eq!( + chain_checks_gloas(&store, &gloas_sidecar(slot, 0), now), + ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::Finalized)), + "another block's column at slot {slot}" + ); + } + } + + #[test] + fn a_gloas_sidecar_naming_a_non_gloas_block_is_rejected() { + let mut store = store(0); + let card = gloas_sidecar(5, 0); + store + .insert_pending_block(card.beacon_block_root, parent_block(5)) + .expect("insert the block"); + store + .insert_state(card.beacon_block_root, fulu_parent(3)) + .expect("insert the state"); + assert_eq!( + stateful_checks_gloas(&store, &card), + Outcome::Reject(RejectReason::BlockNotGloas) + ); + } + + fn parkable_config() -> Config { + Config::mainnet() + .with_fork_epoch(ForkName::Fulu, 0) + .with_fork_epoch(ForkName::Gloas, 1) + } + + /// One blob's worth of column and proof, at `slot` and `index`. + fn shaped_gloas_sidecar(slot: Slot, index: u64) -> gloas::DataColumnSidecar { + let cell: fulu::Cell = + libssz_types::SszVector::try_from(vec![0u8; preset::BYTES_PER_CELL]).expect("cell"); + gloas::DataColumnSidecar { + column: vec![cell].try_into().expect("one cell"), + kzg_proofs: vec![Default::default()].try_into().expect("one proof"), + ..gloas_sidecar(slot, index) + } + } + + #[test] + fn a_well_shaped_custodied_gloas_sidecar_is_parkable() { + let config = parkable_config(); + let slot = preset::SLOTS_PER_EPOCH; + assert!(is_parkable_gloas( + &config, + &[3], + &shaped_gloas_sidecar(slot, 3) + )); + } + + #[test] + fn a_gloas_sidecar_is_not_parkable_outside_gloas_or_off_the_custody_set_or_misshapen() { + let config = parkable_config(); + let slot = preset::SLOTS_PER_EPOCH; + let custody = [3u64]; + // A slot before gloas names another fork's shape. + assert!(!is_parkable_gloas( + &config, + &custody, + &shaped_gloas_sidecar(1, 3) + )); + // Not a column this node custodies. + assert!(!is_parkable_gloas( + &config, + &custody, + &shaped_gloas_sidecar(slot, 4) + )); + // Out of range, even if a custody set were to name it. + let beyond = preset::NUMBER_OF_COLUMNS as u64; + assert!(!is_parkable_gloas( + &config, + &[beyond], + &shaped_gloas_sidecar(slot, beyond) + )); + // No blobs. + assert!(!is_parkable_gloas( + &config, + &custody, + &gloas_sidecar(slot, 3) + )); + // Column and proofs disagree. + let mut uneven = shaped_gloas_sidecar(slot, 3); + uneven.kzg_proofs = Default::default(); + assert!(!is_parkable_gloas(&config, &custody, &uneven)); + // More blobs than the schedule allows at that epoch. + let limit = config.max_blobs_per_block(1) as usize; + let cell: fulu::Cell = + libssz_types::SszVector::try_from(vec![0u8; preset::BYTES_PER_CELL]).expect("cell"); + let mut crowded = shaped_gloas_sidecar(slot, 3); + crowded.column = vec![cell; limit + 1] + .try_into() + .expect("within the list bound"); + crowded.kzg_proofs = vec![Default::default(); limit + 1] + .try_into() + .expect("within the list bound"); + assert!(!is_parkable_gloas(&config, &custody, &crowded)); + } } diff --git a/crates/blockchain/state_transition/src/beacon/gossip/envelope.rs b/crates/blockchain/state_transition/src/beacon/gossip/envelope.rs new file mode 100644 index 000000000..6fa257b64 --- /dev/null +++ b/crates/blockchain/state_transition/src/beacon/gossip/envelope.rs @@ -0,0 +1,395 @@ +//! Gossip validation for the gloas `execution_payload` topic: a builder's +//! `SignedExecutionPayloadEnvelope`, revealing the payload its bid committed to. +//! +//! The rules are the specification's `validate_execution_payload_envelope_gossip` +//! (`specs/gloas/p2p-interface.md`), split the way [`super::column`]'s are: +//! [`cheap_checks`] reads only the message, the seen cache and the store's +//! finalized checkpoint, so the p2p actor runs it inline; [`stateful_checks`] +//! reads the envelope's block and its post-state and verifies the signature, so +//! it runs on a blocking thread. +//! +//! Deliberate departures from the specification: +//! +//! - A block seen without a post-state is [`Outcome::Queue`] where the +//! specification rejects it, the same deviation every other topic makes until +//! a bad-block cache exists. A block never seen is `Queue` too (the +//! specification's "MAY be queued"), and so is a block whose post-state +//! exists but is not cached, since the chain actor verifies the signature +//! itself. +//! - The block-independent limits (withdrawal and request counts) run in +//! [`cheap_checks`], so a message that no block could make valid is never +//! queued. +//! - The finalized-slot `IGNORE` runs in [`cheap_checks`], ahead of the block +//! lookups the specification places before it. Both outcomes are `IGNORE`, so +//! only the metric label can differ, and an envelope too old to matter is +//! not parked waiting for its block. + +use std::num::NonZeroUsize; + +use ethlambda_storage::CacheKey; +use lru::LruCache; + +use super::column::judgeable_block_gloas; +use super::{ + IgnoreReason, Outcome, QueueReason, RejectReason, execution_requests_within_limits, + finalized_start_slot, +}; +use crate::beacon::containers::gloas::BuilderIndex; +use crate::beacon::containers::{SignedBeaconBlock, gloas}; +use crate::beacon::fork_choice::Store; +use crate::beacon::lean_boundary::lean_block_unreachable; +use crate::beacon::preset; +use crate::beacon::primitives::{HashTreeRoot as _, Root}; +use crate::beacon::stf::gloas::verify_execution_payload_envelope_signature; + +/// The first valid envelope per `(block root, builder index)`: the +/// specification's `seen.execution_payload_envelopes`. +/// +/// Bounded by capacity rather than pruned on finality, like [`super::SeenBlocks`]. +pub struct SeenEnvelopes(LruCache<(Root, BuilderIndex), ()>); + +impl SeenEnvelopes { + pub fn new(capacity: NonZeroUsize) -> Self { + Self(LruCache::new(capacity)) + } + + pub fn contains(&self, block_root: Root, builder_index: BuilderIndex) -> bool { + self.0.contains(&(block_root, builder_index)) + } + + /// Record the first valid envelope for its key. Returns `false`, changing + /// nothing, when one is already recorded. + pub fn record(&mut self, block_root: Root, builder_index: BuilderIndex) -> bool { + if self.0.contains(&(block_root, builder_index)) { + return false; + } + self.0.put((block_root, builder_index), ()); + true + } +} + +/// The rules that read only the message, the seen cache and the store's +/// finalized checkpoint. The caller records the envelope in `seen` once the +/// whole rule answers [`Outcome::Accept`]. +pub fn cheap_checks( + seen: &SeenEnvelopes, + store: &Store, + envelope: &gloas::SignedExecutionPayloadEnvelope, +) -> Result<(), Outcome> { + let message = &envelope.message; + // [IGNORE] The node has not seen another valid envelope for this block root + // from this builder. + if seen.contains(message.beacon_block_root, message.builder_index) { + return Err(Outcome::Ignore(IgnoreReason::AlreadySeen)); + } + // [IGNORE] The envelope is from a slot greater than or equal to the latest + // finalized slot. See the module documentation for its position. + if message.payload.slot_number < finalized_start_slot(store) { + return Err(Outcome::Ignore(IgnoreReason::Finalized)); + } + // [REJECT] The number of withdrawals is within the limit, and [REJECT] the + // execution request counts are within theirs. Neither depends on the + // block, so both run before anything can be queued: the lists are + // progressive, and without this only the gossip size cap would bound what + // a parked envelope holds. The specification checks them after the block + // lookups; an envelope invalid under every block is rejected either way. + if message.payload.withdrawals.len() > preset::MAX_WITHDRAWALS_PER_PAYLOAD { + return Err(Outcome::Reject(RejectReason::TooManyWithdrawals)); + } + if !execution_requests_within_limits(&message.execution_requests) { + return Err(Outcome::Reject(RejectReason::OperationLimit)); + } + Ok(()) +} + +/// The rules that need the envelope's block and its post-state, then the +/// signature. Runs on a blocking thread. +/// +/// A block that is known and has a post-state, but whose state is not in the +/// cache this reads, is [`Outcome::Queue`] ([`QueueReason::StateNotCached`]), +/// not an ignore. The state is never rebuilt here, since a miss would stall +/// the verdict gossipsub waits on, and a block imported moments ago is +/// exactly when it is missing. The chain actor fully verifies every envelope +/// it receives (bid consistency, the builder's signature against the block's +/// post-state before parking it, then `verify_execution_payload_envelope`), +/// so forwarding it without propagating is safe, whereas dropping it stalls +/// sync: a fetched envelope for a block the actor is holding would be lost +/// until the next per-slot re-ask. +pub fn stateful_checks(store: &Store, envelope: &gloas::SignedExecutionPayloadEnvelope) -> Outcome { + let message = &envelope.message; + let payload = &message.payload; + // [IGNORE] The envelope's block has been seen (MAY queue), and [REJECT] it + // passes validation: see the module documentation for the latter. + let block = match judgeable_block_gloas(store, &message.beacon_block_root) { + Ok(block) => block, + Err(outcome) => return outcome, + }; + let block = match &block { + SignedBeaconBlock::Gloas(block) => &block.message, + // A block of any other fork carries no bid, so no envelope is valid + // for it. + SignedBeaconBlock::Phase0(_) + | SignedBeaconBlock::Altair(_) + | SignedBeaconBlock::Bellatrix(_) + | SignedBeaconBlock::Capella(_) + | SignedBeaconBlock::Deneb(_) + | SignedBeaconBlock::Electra(_) + | SignedBeaconBlock::Fulu(_) => return Outcome::Reject(RejectReason::BlockNotGloas), + SignedBeaconBlock::Lean(_) => lean_block_unreachable("envelope::stateful_checks"), + }; + let bid = &block.body.signed_execution_payload_bid.message; + + // [REJECT] The block's slot matches the payload's slot number. + if block.slot != payload.slot_number { + return Outcome::Reject(RejectReason::SlotMismatch); + } + // [REJECT] The envelope is from the builder committed to by the bid. + if message.builder_index != bid.builder_index { + return Outcome::Reject(RejectReason::BuilderIndexMismatch); + } + // [REJECT] The payload's block hash matches the bid's block hash. + if payload.block_hash != bid.block_hash { + return Outcome::Reject(RejectReason::BlockHashMismatch); + } + // [REJECT] The execution requests root matches the bid's. + if message.execution_requests.hash_tree_root() != bid.execution_requests_root { + return Outcome::Reject(RejectReason::ExecutionRequestsRootMismatch); + } + // The request-count and withdrawal-count limits ran in `cheap_checks`. + // [REJECT] The envelope signature is valid. Read off the block's cached + // post-state, never rebuilt: a miss would stall the verdict gossipsub + // waits on. + let Some(state) = store.cached_state(CacheKey::BlockState(message.beacon_block_root)) else { + return Outcome::Queue(QueueReason::StateNotCached); + }; + // An error (a builder the state has no entry for) is a signature that + // cannot be valid. + if !matches!( + verify_execution_payload_envelope_signature(&state, envelope), + Ok(true) + ) { + return Outcome::Reject(RejectReason::BadSignature); + } + Outcome::Accept +} + +/// [`cheap_checks`] then [`stateful_checks`], for callers with no reason to +/// split them, such as the spec vectors: the specification's +/// `validate_execution_payload_envelope_gossip`. The caller records the +/// envelope in `seen` on [`Outcome::Accept`]. +pub fn validate( + seen: &SeenEnvelopes, + store: &Store, + envelope: &gloas::SignedExecutionPayloadEnvelope, +) -> Outcome { + if let Err(outcome) = cheap_checks(seen, store, envelope) { + return outcome; + } + stateful_checks(store, envelope) +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use super::*; + use crate::beacon::containers::electra; + use crate::beacon::gossip::test_support::{fulu_parent, seen_envelopes, store}; + use crate::beacon::primitives::Slot; + + /// A fulu block at `slot`: a block of the wrong fork for either topic. + fn fulu_block(slot: Slot) -> SignedBeaconBlock { + SignedBeaconBlock::Fulu(electra::SignedBeaconBlock { + message: electra::BeaconBlock { + slot, + proposer_index: 0, + parent_root: Root::ZERO, + state_root: Root::ZERO, + body: electra::BeaconBlockBody::empty(), + }, + signature: Default::default(), + }) + } + + const BLOCK_ROOT: Root = Root::repeat_byte(7); + + fn withdrawal() -> crate::beacon::containers::capella::Withdrawal { + crate::beacon::containers::capella::Withdrawal { + index: 0, + validator_index: 0, + address: Default::default(), + amount: 0, + } + } + + /// An envelope naming `BLOCK_ROOT`, with `withdrawals` withdrawals in its + /// payload. Nothing else in it is meant to be valid. + fn envelope(slot: Slot, withdrawals: usize) -> gloas::SignedExecutionPayloadEnvelope { + let payload = gloas::ExecutionPayload { + parent_hash: Default::default(), + fee_recipient: Default::default(), + state_root: Default::default(), + receipts_root: Default::default(), + logs_bloom: vec![0u8; preset::BYTES_PER_LOGS_BLOOM] + .try_into() + .expect("an exact-length bloom"), + prev_randao: Default::default(), + block_number: 0, + gas_limit: 0, + gas_used: 0, + timestamp: 0, + extra_data: Default::default(), + base_fee_per_gas: Default::default(), + block_hash: Default::default(), + transactions: Default::default(), + withdrawals: vec![withdrawal(); withdrawals] + .try_into() + .expect("progressive lists are unbounded"), + blob_gas_used: 0, + excess_blob_gas: 0, + block_access_list: Default::default(), + slot_number: slot, + }; + gloas::SignedExecutionPayloadEnvelope { + message: gloas::ExecutionPayloadEnvelope { + payload, + execution_requests: Default::default(), + builder_index: 0, + beacon_block_root: BLOCK_ROOT, + parent_beacon_block_root: Root::ZERO, + }, + signature: Default::default(), + } + } + + #[test] + fn too_many_withdrawals_are_rejected_before_any_block_lookup() { + let store = store(0); + let envelope = envelope(5, preset::MAX_WITHDRAWALS_PER_PAYLOAD + 1); + assert_eq!( + cheap_checks(&seen_envelopes(), &store, &envelope), + Err(Outcome::Reject(RejectReason::TooManyWithdrawals)) + ); + // The block is unknown, which would queue a well-formed envelope. + assert_eq!( + validate(&seen_envelopes(), &store, &envelope), + Outcome::Reject(RejectReason::TooManyWithdrawals) + ); + } + + #[test] + fn too_many_requests_are_rejected_before_any_block_lookup() { + let store = store(0); + let mut envelope = envelope(5, 0); + envelope.message.execution_requests.builder_exits = + vec![Default::default(); preset::MAX_BUILDER_EXIT_REQUESTS_PER_PAYLOAD as usize + 1] + .try_into() + .expect("progressive lists are unbounded"); + assert_eq!( + validate(&seen_envelopes(), &store, &envelope), + Outcome::Reject(RejectReason::OperationLimit) + ); + } + + #[test] + fn an_envelope_within_the_limits_for_an_unseen_block_is_queued() { + let store = store(0); + let envelope = envelope(5, preset::MAX_WITHDRAWALS_PER_PAYLOAD); + assert_eq!( + validate(&seen_envelopes(), &store, &envelope), + Outcome::Queue(QueueReason::BlockUnknown) + ); + } + + #[test] + fn an_envelope_whose_block_has_no_post_state_is_queued_not_rejected() { + let mut store = store(0); + store + .insert_pending_block(BLOCK_ROOT, fulu_block(5)) + .expect("insert pending block"); + assert_eq!( + validate(&seen_envelopes(), &store, &envelope(5, 0)), + Outcome::Queue(QueueReason::BlockNotReady) + ); + } + + /// A known gloas block with a post-state on disk, but with that state + /// evicted from the cache the checks read, must be forwarded (queued), not + /// dropped: the observed stall was a fetched envelope lost this way. + #[test] + fn an_envelope_whose_block_state_is_not_cached_is_queued_not_ignored() { + let envelope = envelope(5, 0); + let mut block = gloas::SignedBeaconBlock::default(); + block.message.slot = 5; + block + .message + .body + .signed_execution_payload_bid + .message + .execution_requests_root = envelope.message.execution_requests.hash_tree_root(); + let mut store = store(0); + store + .insert_pending_block(BLOCK_ROOT, SignedBeaconBlock::Gloas(block)) + .expect("insert the block"); + let state = fulu_parent(3); + store + .insert_state(BLOCK_ROOT, state.clone()) + .expect("insert the state"); + // Churn the cache past its capacity with other keys until the block's + // state is evicted; the pending-state buffer still answers `has_state`. + let filler = Arc::new(state); + for epoch in 0..1024 { + store.cache_state( + CacheKey::CheckpointState { + epoch, + root: Root::ZERO, + }, + filler.clone(), + ); + } + assert!( + store + .cached_state(CacheKey::BlockState(BLOCK_ROOT)) + .is_none() + ); + assert_eq!( + validate(&seen_envelopes(), &store, &envelope), + Outcome::Queue(QueueReason::StateNotCached) + ); + } + + #[test] + fn an_envelope_naming_a_non_gloas_block_is_rejected() { + let mut store = store(0); + store + .insert_pending_block(BLOCK_ROOT, fulu_block(5)) + .expect("insert the block"); + store + .insert_state(BLOCK_ROOT, fulu_parent(3)) + .expect("insert the state"); + assert_eq!( + validate(&seen_envelopes(), &store, &envelope(5, 0)), + Outcome::Reject(RejectReason::BlockNotGloas) + ); + } + + #[test] + fn an_envelope_key_records_once_per_root_and_builder() { + let mut seen = SeenEnvelopes::new(NonZeroUsize::new(4).expect("non-zero")); + let root = Root::repeat_byte(1); + assert!(!seen.contains(root, 7)); + assert!(seen.record(root, 7)); + assert!(seen.contains(root, 7)); + assert!(!seen.record(root, 7)); + assert!(seen.record(root, 8)); + assert!(seen.record(Root::repeat_byte(2), 7)); + } + + #[test] + fn the_seen_envelopes_forget_their_oldest_entry_past_capacity() { + let mut seen = SeenEnvelopes::new(NonZeroUsize::new(1).expect("non-zero")); + seen.record(Root::repeat_byte(1), 0); + seen.record(Root::repeat_byte(2), 0); + assert!(!seen.contains(Root::repeat_byte(1), 0)); + } +} diff --git a/crates/blockchain/state_transition/src/beacon/gossip/mod.rs b/crates/blockchain/state_transition/src/beacon/gossip/mod.rs index bbb0e779f..2959c593e 100644 --- a/crates/blockchain/state_transition/src/beacon/gossip/mod.rs +++ b/crates/blockchain/state_transition/src/beacon/gossip/mod.rs @@ -10,6 +10,8 @@ pub mod aggregate; pub mod attestation; pub mod block; pub mod column; +pub mod envelope; +pub mod payload_attestation; #[cfg(test)] pub(crate) mod test_support; @@ -18,6 +20,8 @@ pub(crate) mod test_support; // topic's submodule defines it. pub use aggregate::SeenAggregates; pub use attestation::SeenAttestations; +pub use envelope::SeenEnvelopes; +pub use payload_attestation::SeenPayloadAttestations; use std::num::NonZeroUsize; @@ -25,11 +29,14 @@ use lru::LruCache; use crate::beacon::config::Config; use crate::beacon::constants::MAXIMUM_GOSSIP_CLOCK_DISPARITY; -use crate::beacon::containers::BeaconState; -use crate::beacon::fork_choice::{self, Store}; +use crate::beacon::containers::{AttestationData, BeaconState, gloas}; +use crate::beacon::fork::ForkName; +use crate::beacon::fork_choice::{self, PayloadStatusEnum, Store}; use crate::beacon::helpers::accessors::get_block_root_at_slot; -use crate::beacon::helpers::misc::compute_start_slot_at_epoch; +use crate::beacon::helpers::misc::{compute_epoch_at_slot, compute_start_slot_at_epoch}; +use crate::beacon::lean_boundary::lean_fork_unreachable; use crate::beacon::precheck::PrecheckError; +use crate::beacon::preset; use crate::beacon::primitives::{Epoch, Root, Slot, ValidatorIndex}; /// A gossip message's verdict. @@ -75,6 +82,15 @@ pub enum QueueReason { ParentNotReady, /// Its slot is outside the parent state's proposer lookahead. ShufflingUnavailable, + /// A gloas sidecar names a block that has never been seen. + BlockUnknown, + /// A gloas sidecar names a block that is stored but has no post-state yet. + BlockNotReady, + /// A gloas envelope names a block whose post-state is stored but not in + /// the cache the gossip checks read, as when the block was imported + /// moments ago. See `envelope::stateful_checks` for why this is not an + /// ignore. + StateNotCached, } impl QueueReason { @@ -83,6 +99,9 @@ impl QueueReason { Self::ParentUnknown => "parent_unknown", Self::ParentNotReady => "parent_not_ready", Self::ShufflingUnavailable => "shuffling_unavailable", + Self::BlockUnknown => "block_unknown", + Self::BlockNotReady => "block_not_ready", + Self::StateNotCached => "state_not_cached", } } } @@ -121,6 +140,23 @@ pub enum IgnoreReason { FinalizedNotAncestor, /// An ancestor lies outside what the state's `block_roots` can answer. AncestryUnknown, + /// A gloas block builds on its parent's full payload branch, but the + /// parent's envelope has not been seen and verified (the specification + /// lets it be queued until it is). + ParentPayloadUnverified, + /// A gloas vote names the full payload (`data.index == 1`) of a block whose + /// envelope has not been seen and verified (the specification lets it be + /// queued until it is). + PayloadEnvelopeUnseen, + /// A gloas vote names a payload the execution client has not validated. + PayloadOptimistic, + /// A payload attestation's slot is not the current slot. + NotCurrentSlot, + /// A payload attestation names a block that is not at the attested slot. + BlockNotAtSlot, + /// The head state's payload timeliness committee window cannot answer for + /// the attested slot. + PtcUnavailable, } impl IgnoreReason { @@ -140,6 +176,12 @@ impl IgnoreReason { Self::StateUnavailable => "state_unavailable", Self::FinalizedNotAncestor => "finalized_not_ancestor", Self::AncestryUnknown => "ancestry_unknown", + Self::ParentPayloadUnverified => "parent_payload_unverified", + Self::PayloadEnvelopeUnseen => "payload_envelope_unseen", + Self::PayloadOptimistic => "payload_optimistic", + Self::NotCurrentSlot => "not_current_slot", + Self::BlockNotAtSlot => "block_not_at_slot", + Self::PtcUnavailable => "ptc_unavailable", } } } @@ -160,6 +202,11 @@ pub enum RejectReason { PayloadTimestamp, InclusionProof, Kzg, + /// A gloas sidecar's slot is not the slot of the block it names. + SlotMismatch, + /// A gloas sidecar names a block of an earlier fork, which has no bid to + /// take commitments from. + BlockNotGloas, /// An attestation's target epoch is not its slot's epoch. EpochMismatch, /// An aggregate with no aggregation bit set. @@ -186,6 +233,33 @@ pub enum RejectReason { AggregateSignature, /// The target is not the voted block's ancestor at the target epoch. TargetNotAncestor, + /// A gloas block body (or its parent execution requests) carries more of + /// an operation than its limit, or any deposit. + OperationLimit, + /// A gloas bid's `parent_block_root` is not the block's `parent_root`. + BidParentMismatch, + /// A gloas block builds on its parent's empty branch, but its bid's + /// `parent_block_hash` is not the parent state's `latest_block_hash`. + BidNotOnParentHead, + /// A gloas vote's `data.index` is neither zero nor one. + DataIndexOutOfRange, + /// A gloas vote cast in its block's own slot claims the payload is present. + SameSlotPayloadFlag, + /// A gloas vote names a payload the execution client found invalid. + PayloadInvalid, + /// A gloas envelope's builder is not the one its block's bid committed to. + BuilderIndexMismatch, + /// A gloas envelope's payload block hash is not its bid's. + BlockHashMismatch, + /// The root of a gloas envelope's execution requests is not its bid's. + ExecutionRequestsRootMismatch, + /// A gloas envelope's payload carries more withdrawals than its limit. + TooManyWithdrawals, + /// A payload attestation's slot lies before the gloas fork. + PreGloasSlot, + /// A payload attestation's validator is not in its slot's payload + /// timeliness committee. + NotInPtc, } impl RejectReason { @@ -204,6 +278,8 @@ impl RejectReason { Self::PayloadTimestamp => "payload_timestamp", Self::InclusionProof => "inclusion_proof", Self::Kzg => "kzg", + Self::SlotMismatch => "slot_mismatch", + Self::BlockNotGloas => "block_not_gloas", Self::EpochMismatch => "epoch_mismatch", Self::NoParticipants => "no_participants", Self::NonZeroDataIndex => "non_zero_data_index", @@ -217,6 +293,18 @@ impl RejectReason { Self::AggregatorSignature => "aggregator_signature", Self::AggregateSignature => "aggregate_signature", Self::TargetNotAncestor => "target_not_ancestor", + Self::OperationLimit => "operation_limit", + Self::BidParentMismatch => "bid_parent_mismatch", + Self::BidNotOnParentHead => "bid_not_on_parent_head", + Self::DataIndexOutOfRange => "data_index_out_of_range", + Self::SameSlotPayloadFlag => "same_slot_payload_flag", + Self::PayloadInvalid => "payload_invalid", + Self::BuilderIndexMismatch => "builder_index_mismatch", + Self::BlockHashMismatch => "block_hash_mismatch", + Self::ExecutionRequestsRootMismatch => "execution_requests_root_mismatch", + Self::TooManyWithdrawals => "too_many_withdrawals", + Self::PreGloasSlot => "pre_gloas_slot", + Self::NotInPtc => "not_in_ptc", } } } @@ -284,6 +372,51 @@ impl SeenColumns { } } +/// The first valid gloas sidecar per `(block root, column index)`: the +/// specification's modified `Seen.data_column_sidecar_tuples`. +/// +/// A separate type from [`SeenColumns`] since the key differs: fulu's names the +/// proposer, which a gloas sidecar does not carry, and names the block by +/// `(slot, proposer)` where gloas names it by root. Bounded the same way. +pub struct SeenBlockColumns(LruCache<(Root, u64), ()>); + +impl SeenBlockColumns { + pub fn new(capacity: NonZeroUsize) -> Self { + Self(LruCache::new(capacity)) + } + + pub fn contains(&self, block_root: Root, index: u64) -> bool { + self.0.contains(&(block_root, index)) + } + + /// Record the first valid sidecar for its key. Returns `false`, changing + /// nothing, when one is already recorded. + pub fn record(&mut self, block_root: Root, index: u64) -> bool { + if self.0.contains(&(block_root, index)) { + return false; + } + self.0.put((block_root, index), ()); + true + } +} + +/// `verify_execution_requests_limits` (`specs/gloas/p2p-interface.md`). +/// +/// The four lists the specification names: withdrawals, consolidations, +/// builder deposits and builder exits. Deposit requests are deliberately not +/// checked: gloas's `DepositRequests` is progressive and neither the +/// specification nor the state transition bounds it, so a limit here would +/// reject a block that import accepts. +/// +/// Shared by [`block`] (a block's parent execution requests) and [`envelope`]. +pub(crate) fn execution_requests_within_limits(requests: &gloas::ExecutionRequests) -> bool { + requests.withdrawals.len() <= preset::MAX_WITHDRAWAL_REQUESTS_PER_PAYLOAD + && requests.consolidations.len() <= preset::MAX_CONSOLIDATION_REQUESTS_PER_PAYLOAD + && requests.builder_deposits.len() as u64 + <= preset::MAX_BUILDER_DEPOSIT_REQUESTS_PER_PAYLOAD + && requests.builder_exits.len() as u64 <= preset::MAX_BUILDER_EXIT_REQUESTS_PER_PAYLOAD +} + /// The specification's `compute_time_at_slot_ms`: the clock reading at the /// start of `slot`. pub(crate) fn slot_start_ms(config: &Config, slot: Slot) -> u64 { @@ -298,6 +431,18 @@ pub(crate) fn is_future_slot(config: &Config, slot: Slot, now_ms: u64) -> bool { slot_start_ms(config, slot) > now_ms.saturating_add(MAXIMUM_GOSSIP_CLOCK_DISPARITY) } +/// The specification's `is_current_slot` (`altair/p2p-interface.md`): the +/// clock, with the gossip clock disparity allowance on both ends, falls within +/// `slot`'s own span. That is `is_within_slot_range` with a range of zero, so +/// the far edge is the *next* slot's start. +pub(crate) fn is_current_slot(config: &Config, slot: Slot, now_ms: u64) -> bool { + if now_ms.saturating_add(MAXIMUM_GOSSIP_CLOCK_DISPARITY) < slot_start_ms(config, slot) { + return false; + } + slot_start_ms(config, slot.saturating_add(1)).saturating_add(MAXIMUM_GOSSIP_CLOCK_DISPARITY) + >= now_ms +} + /// The specification's `is_within_epoch`: the clock, with the gossip clock /// disparity allowance on both ends, falls somewhere in `epoch`'s own span of /// slots. @@ -329,6 +474,80 @@ pub(crate) fn is_current_or_previous_epoch(config: &Config, epoch: Epoch, now_ms is_within_epoch(config, epoch, now_ms) || is_within_epoch(config, epoch + 1, now_ms) } +/// Whether `slot` falls in gloas. +/// +/// A gloas subnet attestation is the same `SingleAttestation` as electra's, so +/// the message's shape cannot say which rules apply; its slot's fork does. +/// (A gloas aggregate has its own container, which says so itself.) +pub(crate) fn is_gloas_slot(config: &Config, slot: Slot) -> bool { + match config.fork_at_epoch(compute_epoch_at_slot(slot)) { + ForkName::Gloas => true, + ForkName::Phase0 + | ForkName::Altair + | ForkName::Bellatrix + | ForkName::Capella + | ForkName::Deneb + | ForkName::Electra + | ForkName::Fulu => false, + ForkName::Lean => lean_fork_unreachable("gossip::is_gloas_slot"), + } +} + +/// Gloas's `verify_attestation_payload_status` (`specs/gloas/p2p-interface.md`): +/// the payload flag a vote carries in `data.index` must agree with what this +/// node knows of the voted block's payload. +/// +/// The specification lets a vote on an unseen envelope be queued and asks the +/// node to request it by root; neither is done here, so it is an `IGNORE`. +pub(crate) fn verify_attestation_payload_status( + store: &Store, + data: &AttestationData, +) -> Result<(), Outcome> { + let block_root = data.beacon_block_root; + let Some((block_slot, _)) = store.block_slot_and_state_root(&block_root) else { + // The caller has already checked the block is known. + return Err(Outcome::Ignore(IgnoreReason::UnknownBlock)); + }; + payload_vote_verdict( + block_slot, + data, + || fork_choice::is_payload_verified(store, block_root), + || fork_choice::block_payload_status(store, block_root), + ) +} + +/// The branches of [`verify_attestation_payload_status`], over the facts it +/// reads. The envelope and status lookups are lazy: only a vote for the full +/// payload needs either. +fn payload_vote_verdict( + block_slot: Slot, + data: &AttestationData, + is_payload_verified: impl FnOnce() -> bool, + payload_status: impl FnOnce() -> PayloadStatusEnum, +) -> Result<(), Outcome> { + // [REJECT] For same-slot attestations, the payload cannot yet be present. + if block_slot == data.slot && data.index != 0 { + return Err(Outcome::Reject(RejectReason::SameSlotPayloadFlag)); + } + if data.index != 1 { + return Ok(()); + } + // [IGNORE] The envelope has been seen and verified. + if !is_payload_verified() { + return Err(Outcome::Ignore(IgnoreReason::PayloadEnvelopeUnseen)); + } + let status = payload_status(); + // [IGNORE] The attested payload is optimistic. + if status.is_not_validated() { + return Err(Outcome::Ignore(IgnoreReason::PayloadOptimistic)); + } + // [REJECT] The attested payload is processed and invalid. + if status.is_invalidated() { + return Err(Outcome::Reject(RejectReason::PayloadInvalid)); + } + Ok(()) +} + /// Which block `state`'s own history names as the ancestor at `slot`, given /// that `state` is `at_block_root`'s post-state. /// @@ -455,6 +674,71 @@ mod tests { NonZeroUsize::new(n).expect("non-zero") } + fn vote(slot: Slot, index: u64) -> AttestationData { + AttestationData { + slot, + index, + ..Default::default() + } + } + + fn never() -> bool { + unreachable!("a vote that does not claim the full payload reads no envelope") + } + + fn no_status() -> PayloadStatusEnum { + unreachable!("a vote that does not claim the full payload reads no status") + } + + #[test] + fn a_same_slot_vote_for_the_payload_is_rejected() { + assert_eq!( + payload_vote_verdict(5, &vote(5, 1), never, no_status), + Err(Outcome::Reject(RejectReason::SameSlotPayloadFlag)) + ); + } + + #[test] + fn a_vote_without_the_payload_flag_reads_no_payload_state() { + assert_eq!( + payload_vote_verdict(5, &vote(5, 0), never, no_status), + Ok(()) + ); + assert_eq!( + payload_vote_verdict(4, &vote(5, 0), never, no_status), + Ok(()) + ); + } + + #[test] + fn a_full_payload_vote_without_a_verified_envelope_is_ignored() { + assert_eq!( + payload_vote_verdict(4, &vote(5, 1), || false, no_status), + Err(Outcome::Ignore(IgnoreReason::PayloadEnvelopeUnseen)) + ); + } + + #[test] + fn a_full_payload_vote_judges_the_execution_verdict() { + let verdict = |status| payload_vote_verdict(4, &vote(5, 1), || true, move || status); + assert_eq!(verdict(PayloadStatusEnum::Valid), Ok(())); + for status in [PayloadStatusEnum::Syncing, PayloadStatusEnum::Accepted] { + assert_eq!( + verdict(status), + Err(Outcome::Ignore(IgnoreReason::PayloadOptimistic)) + ); + } + for status in [ + PayloadStatusEnum::Invalid, + PayloadStatusEnum::InvalidBlockHash, + ] { + assert_eq!( + verdict(status), + Err(Outcome::Reject(RejectReason::PayloadInvalid)) + ); + } + } + #[test] fn a_slot_is_future_only_past_the_clock_disparity() { let config = Config { @@ -474,6 +758,36 @@ mod tests { )); } + #[test] + fn the_current_slot_holds_within_the_clock_disparity_at_either_edge() { + let config = Config { + genesis_time: 0, + ..Config::mainnet() + }; + let start_ms = slot_start_ms(&config, 3); + let next_start_ms = slot_start_ms(&config, 4); + assert!(is_current_slot( + &config, + 3, + start_ms - MAXIMUM_GOSSIP_CLOCK_DISPARITY + )); + assert!(!is_current_slot( + &config, + 3, + start_ms - MAXIMUM_GOSSIP_CLOCK_DISPARITY - 1 + )); + assert!(is_current_slot( + &config, + 3, + next_start_ms + MAXIMUM_GOSSIP_CLOCK_DISPARITY + )); + assert!(!is_current_slot( + &config, + 3, + next_start_ms + MAXIMUM_GOSSIP_CLOCK_DISPARITY + 1 + )); + } + #[test] fn a_block_key_records_once() { let mut seen = SeenBlocks::new(capacity(4)); @@ -493,6 +807,16 @@ mod tests { assert!(seen.record(10, 3, 1)); } + #[test] + fn a_block_column_key_records_once_per_root_and_index() { + let mut seen = SeenBlockColumns::new(capacity(4)); + let root = Root::from([1; 32]); + assert!(seen.record(root, 0)); + assert!(!seen.record(root, 0)); + assert!(seen.record(root, 1)); + assert!(seen.record(Root::from([2; 32]), 0)); + } + #[test] fn the_caches_forget_their_oldest_entry_past_capacity() { let mut seen = SeenBlocks::new(capacity(2)); diff --git a/crates/blockchain/state_transition/src/beacon/gossip/payload_attestation.rs b/crates/blockchain/state_transition/src/beacon/gossip/payload_attestation.rs new file mode 100644 index 000000000..1ba89616a --- /dev/null +++ b/crates/blockchain/state_transition/src/beacon/gossip/payload_attestation.rs @@ -0,0 +1,307 @@ +//! Gossip validation for the gloas `payload_attestation_message` topic: a +//! payload timeliness committee member's vote on whether a block's payload +//! arrived in time. +//! +//! The rules are the specification's `validate_payload_attestation_message_gossip` +//! (`specs/gloas/p2p-interface.md`), split like [`super::envelope`]'s: +//! [`cheap_checks`] reads the message, the clock, the seen cache and the +//! config; [`stateful_checks`] reads the voted block and the head state's +//! committee and verifies the signature. +//! +//! Deliberate departures from the specification: +//! +//! - A voted block seen without a post-state is `IGNORE` where the +//! specification rejects it, until a bad-block cache exists. An unseen block +//! is `IGNORE` as the specification says; nothing is queued, since the vote +//! is only valid within its own slot. +//! - The head state is read from the state cache, never rebuilt. A miss is +//! `IGNORE`, like the other topics' uncached states. + +use std::num::NonZeroUsize; + +use ethlambda_storage::CacheKey; +use lru::LruCache; + +use super::{IgnoreReason, Outcome, RejectReason, is_current_slot}; +use crate::beacon::bls; +use crate::beacon::constants::DOMAIN_PTC_ATTESTER; +use crate::beacon::containers::gloas; +use crate::beacon::fork_choice::Store; +use crate::beacon::helpers::accessors::get_domain; +use crate::beacon::helpers::gloas::get_ptc; +use crate::beacon::helpers::misc::{compute_epoch_at_slot, compute_signing_root}; +use crate::beacon::primitives::{HashTreeRoot as _, Slot, ValidatorIndex}; + +/// The first valid payload attestation per `(slot, validator index)`: the +/// specification's `seen.payload_attestation_validators`. +/// +/// Bounded the same way as [`super::SeenBlocks`]. +pub struct SeenPayloadAttestations(LruCache<(Slot, ValidatorIndex), ()>); + +impl SeenPayloadAttestations { + pub fn new(capacity: NonZeroUsize) -> Self { + Self(LruCache::new(capacity)) + } + + pub fn contains(&self, slot: Slot, validator_index: ValidatorIndex) -> bool { + self.0.contains(&(slot, validator_index)) + } + + /// Record the first valid attestation for its key. Returns `false`, + /// changing nothing, when one is already recorded. + pub fn record(&mut self, slot: Slot, validator_index: ValidatorIndex) -> bool { + if self.0.contains(&(slot, validator_index)) { + return false; + } + self.0.put((slot, validator_index), ()); + true + } +} + +/// The rules that read only the message, the clock, the seen cache and the +/// config. The caller records the message in `seen` once the whole rule +/// answers [`Outcome::Accept`]. +pub fn cheap_checks( + seen: &SeenPayloadAttestations, + store: &Store, + message: &gloas::PayloadAttestationMessage, + now_ms: u64, +) -> Result<(), Outcome> { + let config = store.config(); + let slot = message.data.slot; + // [REJECT] The payload attestation's slot is at or after the gloas fork. + if compute_epoch_at_slot(slot) < config.gloas_fork_epoch { + return Err(Outcome::Reject(RejectReason::PreGloasSlot)); + } + // [IGNORE] This is the first valid payload attestation from this validator. + if seen.contains(slot, message.validator_index) { + return Err(Outcome::Ignore(IgnoreReason::AlreadySeen)); + } + // [IGNORE] The payload attestation's slot is the current slot. + if !is_current_slot(&config, slot, now_ms) { + return Err(Outcome::Ignore(IgnoreReason::NotCurrentSlot)); + } + Ok(()) +} + +/// The rules that need the voted block and the head state. Runs on a blocking +/// thread. +pub fn stateful_checks(store: &Store, message: &gloas::PayloadAttestationMessage) -> Outcome { + let data = &message.data; + let validator_index = message.validator_index; + // [IGNORE] The voted block has been seen. + let Some((block_slot, _)) = store.block_slot_and_state_root(&data.beacon_block_root) else { + return Outcome::Ignore(IgnoreReason::UnknownBlock); + }; + // [REJECT] The voted block passes validation: see the module documentation. + match store.has_state(&data.beacon_block_root) { + Ok(true) => {} + Ok(false) => return Outcome::Ignore(IgnoreReason::StateUnavailable), + Err(_) => return Outcome::Ignore(IgnoreReason::Internal), + } + // [IGNORE] The voted block is at the assigned slot. + if block_slot != data.slot { + return Outcome::Ignore(IgnoreReason::BlockNotAtSlot); + } + + // The committee is read off the head state, as the specification says. + // `head` is the root alone: `beacon_head` would decode the whole head block + // for a slot nothing here reads. + let Ok(head_root) = store.head() else { + return Outcome::Ignore(IgnoreReason::Internal); + }; + let Some(state) = store.cached_state(CacheKey::BlockState(head_root)) else { + return Outcome::Ignore(IgnoreReason::StateUnavailable); + }; + // [REJECT] The validator index is valid. + let Ok(validator) = state.validator(validator_index) else { + return Outcome::Reject(RejectReason::UnknownValidator); + }; + // [REJECT] The validator is a member of the payload timeliness committee. + // A slot the head's committee window cannot answer is not the sender's fault. + let Ok(ptc) = get_ptc(&state, data.slot, &store.config()) else { + return Outcome::Ignore(IgnoreReason::PtcUnavailable); + }; + if !ptc.contains(&validator_index) { + return Outcome::Reject(RejectReason::NotInPtc); + } + // [REJECT] The signature is valid. + let domain = get_domain( + &state, + DOMAIN_PTC_ATTESTER, + Some(compute_epoch_at_slot(data.slot)), + ); + let signing_root = compute_signing_root(data.hash_tree_root(), domain); + if !bls::verify(&validator.pubkey, signing_root, &message.signature) { + return Outcome::Reject(RejectReason::BadSignature); + } + Outcome::Accept +} + +/// [`cheap_checks`] then [`stateful_checks`], for callers with no reason to +/// split them, such as the spec vectors: the specification's +/// `validate_payload_attestation_message_gossip`. The caller records the +/// message in `seen` on [`Outcome::Accept`]. +pub fn validate( + seen: &SeenPayloadAttestations, + store: &Store, + message: &gloas::PayloadAttestationMessage, + now_ms: u64, +) -> Outcome { + if let Err(outcome) = cheap_checks(seen, store, message, now_ms) { + return outcome; + } + stateful_checks(store, message) +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use ethlambda_storage::backend::InMemoryBackend; + use ethlambda_types::checkpoint::Checkpoint; + + use super::*; + use crate::beacon::config::Config; + use crate::beacon::containers::{SignedBeaconBlock, electra}; + use crate::beacon::fork::ForkName; + use crate::beacon::gossip::test_support::{GENESIS_TIME, store}; + use crate::beacon::helpers::test_state::with_validators_at; + use crate::beacon::preset; + use crate::beacon::primitives::Root; + + /// A fulu block at `slot`: a block of the wrong fork for either topic. + fn fulu_block(slot: Slot) -> SignedBeaconBlock { + SignedBeaconBlock::Fulu(electra::SignedBeaconBlock { + message: electra::BeaconBlock { + slot, + proposer_index: 0, + parent_root: Root::ZERO, + state_root: Root::ZERO, + body: electra::BeaconBlockBody::empty(), + }, + signature: Default::default(), + }) + } + + const BLOCK_ROOT: Root = Root::repeat_byte(7); + + fn message(slot: Slot) -> gloas::PayloadAttestationMessage { + gloas::PayloadAttestationMessage { + validator_index: 0, + data: gloas::PayloadAttestationData { + beacon_block_root: BLOCK_ROOT, + slot, + payload_present: true, + blob_data_available: true, + }, + signature: Default::default(), + } + } + + /// A store that is gloas from genesis, so the committee lookup is reached. + fn gloas_store() -> Store { + Store::init_beacon( + Arc::new(InMemoryBackend::new()), + GENESIS_TIME, + Config::mainnet() + .with_fork_epoch(ForkName::Fulu, 0) + .with_fork_epoch(ForkName::Gloas, 0), + Root::ZERO, + Checkpoint { + root: Root::ZERO, + slot: 0, + }, + 0, + ) + } + + #[test] + fn a_vote_for_an_unseen_block_is_ignored() { + let store = store(0); + assert_eq!( + stateful_checks(&store, &message(5)), + Outcome::Ignore(IgnoreReason::UnknownBlock) + ); + } + + #[test] + fn a_vote_for_a_block_without_a_post_state_is_ignored_not_rejected() { + let mut store = store(0); + store + .insert_pending_block(BLOCK_ROOT, fulu_block(5)) + .expect("insert pending block"); + assert_eq!( + stateful_checks(&store, &message(5)), + Outcome::Ignore(IgnoreReason::StateUnavailable) + ); + } + + #[test] + fn a_vote_is_ignored_when_the_head_state_is_not_cached() { + let mut store = store(0); + store + .insert_pending_block(BLOCK_ROOT, fulu_block(5)) + .expect("insert the block"); + // Only the voted block has a state; the head (the anchor root) has none. + store + .insert_state(BLOCK_ROOT, with_validators_at(ForkName::Gloas, 8)) + .expect("insert the state"); + assert_eq!( + stateful_checks(&store, &message(5)), + Outcome::Ignore(IgnoreReason::StateUnavailable) + ); + } + + #[test] + fn a_vote_outside_the_head_states_committee_window_is_ignored_with_its_own_reason() { + let mut store = gloas_store(); + let slot = 10 * preset::SLOTS_PER_EPOCH; + store + .insert_pending_block(BLOCK_ROOT, fulu_block(slot)) + .expect("insert the block"); + store + .insert_state(BLOCK_ROOT, with_validators_at(ForkName::Gloas, 8)) + .expect("insert the voted block's state"); + let head = store.head().expect("head root"); + store + .insert_state(head, with_validators_at(ForkName::Gloas, 8)) + .expect("insert the head state"); + assert_eq!( + stateful_checks(&store, &message(slot)), + Outcome::Ignore(IgnoreReason::PtcUnavailable) + ); + } + + #[test] + fn a_validator_the_head_state_lacks_is_rejected() { + let mut store = gloas_store(); + store + .insert_pending_block(BLOCK_ROOT, fulu_block(5)) + .expect("insert the block"); + store + .insert_state(BLOCK_ROOT, with_validators_at(ForkName::Gloas, 8)) + .expect("insert the voted block's state"); + let head = store.head().expect("head root"); + store + .insert_state(head, with_validators_at(ForkName::Gloas, 8)) + .expect("insert the head state"); + let mut message = message(5); + message.validator_index = 8; + assert_eq!( + stateful_checks(&store, &message), + Outcome::Reject(RejectReason::UnknownValidator) + ); + } + + #[test] + fn a_payload_attestation_key_records_once_per_slot_and_validator() { + let mut seen = SeenPayloadAttestations::new(NonZeroUsize::new(4).expect("non-zero")); + assert!(!seen.contains(5, 9)); + assert!(seen.record(5, 9)); + assert!(seen.contains(5, 9)); + assert!(!seen.record(5, 9)); + assert!(seen.record(5, 10)); + assert!(seen.record(6, 9)); + } +} diff --git a/crates/blockchain/state_transition/src/beacon/gossip/test_support.rs b/crates/blockchain/state_transition/src/beacon/gossip/test_support.rs index 593b2dc3b..4b9e6b40a 100644 --- a/crates/blockchain/state_transition/src/beacon/gossip/test_support.rs +++ b/crates/blockchain/state_transition/src/beacon/gossip/test_support.rs @@ -157,6 +157,11 @@ pub(crate) fn seen_aggregates() -> SeenAggregates { SeenAggregates::new(capacity, capacity) } +/// The [`seen_blocks`] counterpart for gloas execution payload envelopes. +pub(crate) fn seen_envelopes() -> super::SeenEnvelopes { + super::SeenEnvelopes::new(NonZeroUsize::new(8).expect("non-zero")) +} + /// The [`seen_blocks`] counterpart for subnet attestations. pub(crate) fn seen_attestations() -> SeenAttestations { SeenAttestations::new(NonZeroUsize::new(8).expect("non-zero")) diff --git a/crates/blockchain/state_transition/src/beacon/precheck.rs b/crates/blockchain/state_transition/src/beacon/precheck.rs index 25d66d032..c34511a17 100644 --- a/crates/blockchain/state_transition/src/beacon/precheck.rs +++ b/crates/blockchain/state_transition/src/beacon/precheck.rs @@ -133,8 +133,8 @@ pub fn precheck_block( /// The proposer `parent_state` has already fixed for `slot`, if it has. /// -/// Fulu's `proposer_lookahead` (EIP-7917) holds the proposers of the state's -/// current epoch and of the `MIN_SEED_LOOKAHEAD` epochs after it. Each entry +/// Fulu's (and gloas's) `proposer_lookahead` (EIP-7917) holds the proposers of +/// the state's current epoch and of the `MIN_SEED_LOOKAHEAD` epochs after it. Each entry /// is fixed when it enters the window: epoch processing only shifts the /// window along and appends a new last epoch. A slot inside the window is /// therefore answered exactly as the state advanced to that slot would answer @@ -142,24 +142,21 @@ pub fn precheck_block( /// state, which is the import's job; the specification lets such a block /// through rather than rejecting it, and so does this. pub(crate) fn fixed_proposer(parent_state: &BeaconState, slot: Slot) -> Option { - let state = match parent_state { - BeaconState::Fulu(state) => state, - // Gloas keeps the lookahead, but its gossip rules are not applied - // here yet (block and column gossip refuse it before this is asked), - // so no window is read from a gloas parent and the caller lets the - // block through as it does before fulu. + // Gloas keeps fulu's lookahead unchanged, so both answer the same way. + let (state_slot, lookahead) = match parent_state { + BeaconState::Fulu(state) => (state.slot, &state.proposer_lookahead), + BeaconState::Gloas(state) => (state.slot, &state.proposer_lookahead), BeaconState::Phase0(_) | BeaconState::Altair(_) | BeaconState::Bellatrix(_) | BeaconState::Capella(_) | BeaconState::Deneb(_) | BeaconState::Electra(_) - | BeaconState::Gloas(_) | BeaconState::Lean(_) => return None, }; - let window_start = compute_start_slot_at_epoch(compute_epoch_at_slot(state.slot)); + let window_start = compute_start_slot_at_epoch(compute_epoch_at_slot(state_slot)); let offset = usize::try_from(slot.checked_sub(window_start)?).ok()?; - state.proposer_lookahead.get(offset).copied() + lookahead.get(offset).copied() } #[cfg(test)] @@ -261,6 +258,25 @@ mod tests { precheck_block(block, block.message_hash_tree_root(), reference, config) } + #[test] + fn a_gloas_parents_lookahead_fixes_the_proposer_like_a_fulu_one() { + let mut parent = with_validators_at(ForkName::Gloas, 8); + let BeaconState::Gloas(inner) = &mut parent else { + unreachable!("with_validators_at(Gloas) builds a gloas state"); + }; + for (offset, entry) in inner.proposer_lookahead.iter_mut().enumerate() { + *entry = offset as ValidatorIndex; + } + let window_start = compute_start_slot_at_epoch(compute_epoch_at_slot(parent.slot())); + let slot = parent.slot() + 1; + assert_eq!( + fixed_proposer(&parent, slot), + Some((slot - window_start) as ValidatorIndex) + ); + let beyond = window_start + preset::PROPOSER_LOOKAHEAD_LENGTH as Slot; + assert_eq!(fixed_proposer(&parent, beyond), None); + } + #[test] fn a_block_signed_by_its_expected_proposer_passes() { let config = Config::mainnet(); diff --git a/crates/blockchain/state_transition/src/beacon/stf/gloas.rs b/crates/blockchain/state_transition/src/beacon/stf/gloas.rs index 303dd403e..fe982ed14 100644 --- a/crates/blockchain/state_transition/src/beacon/stf/gloas.rs +++ b/crates/blockchain/state_transition/src/beacon/stf/gloas.rs @@ -512,14 +512,14 @@ pub fn process_withdrawals(state: &mut BeaconState) -> Result<()> { /// built from the revealed envelope's own `execution_requests`. Not from /// `state.latest_execution_payload_bid`: the bid commits to that list only /// by its hash ([`gloas::ExecutionPayloadBid::execution_requests_root`]), it -/// never carries the list itself. Nothing calls this: envelope verification -/// does not build the request list, since [`ExecutionEngine`] collapses the -/// whole `verify_and_notify_new_payload` interface to one boolean and never -/// inspects it (see [`crate::beacon::stf::deneb::process_execution_payload`]'s -/// own documentation). Electra's own -/// [`crate::beacon::stf::electra::get_execution_requests_list`] is called, by -/// the live follower's Engine API client (`ethlambda-blockchain`'s -/// `beacon_engine`), which sends the list to a real execution client. +/// never carries the list itself. [`verify_execution_payload_envelope`] does +/// not call this, since [`ExecutionEngine`] collapses the whole +/// `verify_and_notify_new_payload` interface to one boolean and never inspects +/// it (see [`crate::beacon::stf::deneb::process_execution_payload`]'s own +/// documentation). The live follower's Engine API client does: `ethlambda-blockchain`'s +/// `beacon_engine` builds the `engine_newPayloadV5` request list with this, +/// and electra's [`crate::beacon::stf::electra::get_execution_requests_list`] +/// for the earlier forks. pub fn get_execution_requests_list(requests: &gloas::ExecutionRequests) -> Vec> { let mut list = Vec::new(); diff --git a/crates/blockchain/state_transition/tests/beacon_spec/gossip.rs b/crates/blockchain/state_transition/tests/beacon_spec/gossip.rs index a3c0a81f2..e98ec5ed2 100644 --- a/crates/blockchain/state_transition/tests/beacon_spec/gossip.rs +++ b/crates/blockchain/state_transition/tests/beacon_spec/gossip.rs @@ -16,16 +16,19 @@ use std::sync::Arc; use ethlambda_state_transition::beacon::ForkName; use ethlambda_state_transition::beacon::config::Config; use ethlambda_state_transition::beacon::containers::{ - BeaconState, Checkpoint, SignedAggregateAndProof, SignedBeaconBlock, electra, fulu, phase0, + BeaconState, Checkpoint, DataColumnSidecar, SignedAggregateAndProof, SignedBeaconBlock, + electra, gloas, phase0, }; use ethlambda_state_transition::beacon::fork_choice::{ - self, DataAvailability, PayloadValidity, Store, + self, DataAvailability, PayloadStatusEnum, PayloadValidity, Store, }; use ethlambda_state_transition::beacon::gossip::{ - self as rules, Outcome, SeenAggregates, SeenAttestations, SeenBlocks, SeenColumns, + self as rules, Outcome, SeenAggregates, SeenAttestations, SeenBlockColumns, SeenBlocks, + SeenColumns, SeenEnvelopes, SeenPayloadAttestations, }; use ethlambda_state_transition::beacon::helpers::accessors::CommitteeCache; use ethlambda_state_transition::beacon::primitives::Root; +use ethlambda_state_transition::beacon::stf::ExecutionEngine; use ethlambda_storage::ForkCheckpoints; use libssz::SszDecode; use libtest_mimic::{Failed, Trial}; @@ -38,8 +41,33 @@ const HANDLERS: &[&str] = &[ "gossip_data_column_sidecar", "gossip_beacon_aggregate_and_proof", "gossip_beacon_attestation", + "gossip_execution_payload_envelope", + "gossip_payload_attestation_message", ]; +/// The forks each of [`HANDLERS`] validates. A case from any other fork is +/// reported as ignored rather than run, the same "known gap, not a silent one" +/// treatment `case.in_scope()` gives a fork past `HIGHEST_IMPLEMENTED_FORK`. +/// +/// `case.in_scope()` alone would pass every fork up to gloas, since the state +/// transition handles them all; the gossip rules do not. The block rules +/// (`beacon::gossip::block`) and the column rules (`beacon::gossip::column`) +/// have both fulu's and gloas's, and so do the aggregate and attestation +/// rules (`beacon::gossip::{aggregate, attestation}`): electra's, which fulu +/// keeps, and gloas's payload-flag modification of them. +fn validated_forks(handler: &str) -> &'static [ForkName] { + match handler { + "gossip_data_column_sidecar" | "gossip_beacon_block" => &[ForkName::Fulu, ForkName::Gloas], + "gossip_beacon_aggregate_and_proof" | "gossip_beacon_attestation" => { + &[ForkName::Fulu, ForkName::Gloas] + } + "gossip_execution_payload_envelope" | "gossip_payload_attestation_message" => { + &[ForkName::Gloas] + } + other => panic!("{other} is not in HANDLERS, so it has no validated forks"), + } +} + /// Every other `gossip_*` handler the fixture tree ships, none of which this /// node validates yet: the topics it neither subscribes to nor has a /// validator for. Reported as ignored, named individually, rather than left @@ -49,11 +77,10 @@ const HANDLERS: &[&str] = &[ /// list, the same way [`super::UNMODELED_FORKS`] forces a decision on a new /// fork directory. /// -/// `gossip_execution_payload_bid`, `gossip_execution_payload_envelope`, -/// `gossip_payload_attestation_message`, and `gossip_proposer_preferences` -/// are gloas's own topics (EIP-7732 ePBS): the builder's bid, the revealed -/// payload envelope, the payload timeliness committee's vote, and a -/// builder's advertised preferences, respectively. None of them existed +/// `gossip_execution_payload_bid` and `gossip_proposer_preferences` are +/// gloas's own topics (EIP-7732 ePBS): the builder's bid and a builder's +/// advertised preferences, respectively. (Its envelope and payload +/// attestation topics are in [`HANDLERS`].) None of them existed /// until gloas's fixture directory started parsing (`ForkName::Gloas`), so /// they land here rather than silently in `unknown` the first time this /// runner sees them. Alphabetized with the rest rather than kept together. @@ -62,9 +89,7 @@ const IGNORED_HANDLERS: &[&str] = &[ "gossip_blob_sidecar", "gossip_bls_to_execution_change", "gossip_execution_payload_bid", - "gossip_execution_payload_envelope", "gossip_partial_data_column_sidecar", - "gossip_payload_attestation_message", "gossip_proposer_preferences", "gossip_proposer_slashing", "gossip_sync_committee_contribution_and_proof", @@ -78,10 +103,18 @@ const SKIPPED: &[(&str, &str)] = &[ "gossip_beacon_block__reject_parent_consensus_failed_execution_not_verified", "a parent seen without a post-state is queued, not rejected, until a bad-block cache exists", ), + ( + "gossip_beacon_block__reject_parent_failed_validation", + "a parent seen without a post-state is queued, not rejected, until a bad-block cache exists", + ), ( "gossip_data_column_sidecar__reject_parent_failed_validation", "a parent seen without a post-state is queued, not rejected, until a bad-block cache exists", ), + ( + "gossip_data_column_sidecar__reject_block_failed_validation", + "a block seen without a post-state is queued, not rejected, until a bad-block cache exists", + ), ( "gossip_beacon_aggregate_and_proof__reject_block_failed_validation", "a vote block seen without a post-state is ignored, not rejected, until a bad-block cache exists", @@ -90,6 +123,14 @@ const SKIPPED: &[(&str, &str)] = &[ "gossip_beacon_attestation__reject_block_failed_validation", "a vote block seen without a post-state is ignored, not rejected, until a bad-block cache exists", ), + ( + "gossip_execution_payload_envelope__reject_block_failed_validation", + "a block seen without a post-state is queued, not rejected, until a bad-block cache exists", + ), + ( + "gossip_payload_attestation_message__reject_block_failed_validation", + "a vote block seen without a post-state is ignored, not rejected, until a bad-block cache exists", + ), ]; /// Capacity for a case's seen caches. A case sends a handful of messages. @@ -113,6 +154,9 @@ struct StoreBlock { #[serde(default)] pending: bool, payload_status: Option, + /// The block's execution payload envelope, delivered once the block is + /// imported so `is_payload_verified` answers true for its root. + payload: Option, } #[derive(serde::Deserialize)] @@ -124,13 +168,38 @@ struct FinalizedOverride { #[derive(serde::Deserialize)] struct GossipMessage { - offset_ms: u64, + /// The format's offset from `meta.current_time_ms`. Some of gloas's + /// column vectors carry `current_time_ms` instead; see + /// [`GossipMessage::arrival_ms`]. + offset_ms: Option, + current_time_ms: Option, subnet_id: Option, message: String, expected: String, reason: Option, } +impl GossipMessage { + /// The clock reading, in milliseconds since genesis, the message arrives + /// at: the format's `meta.current_time_ms + offset_ms`. + /// + /// Some of gloas's column vectors carry a per-message `current_time_ms` + /// instead: the generator (`test/gloas/networking/ + /// test_gossip_data_column_sidecar.py`) writes `compute_time_at_slot_ms` + /// plus 500 there, the same basis as `meta.current_time_ms`, so it is the + /// absolute reading. + fn arrival_ms(&self, base_ms: u64) -> Result { + match (self.offset_ms, self.current_time_ms) { + (Some(offset), _) => Ok(base_ms + offset), + (None, Some(absolute)) => Ok(absolute), + (None, None) => Err(format!( + "{}: names neither offset_ms nor current_time_ms", + self.message + )), + } + } +} + fn parse_root(hex_root: &str) -> Root { let stripped = hex_root.strip_prefix("0x").unwrap_or(hex_root); let bytes = hex::decode(stripped).expect("the fixture's root is valid hex"); @@ -147,9 +216,7 @@ fn decode_block(case: &Case, name: &str) -> Result { /// needed one yet): every fork through deneb shares phase0's shape, and /// electra and fulu share electra's, exactly the split /// [`SignedAggregateAndProof`]'s own doc describes for -/// [`SignedBeaconBlock::Fulu`]. Gloas's own aggregate has no modeled variant, -/// and this runner does not run its cases (see [`trials`]), so it is refused -/// by name. +/// [`SignedBeaconBlock::Fulu`]. Gloas has its own, progressive one. fn decode_signed_aggregate(case: &Case, name: &str) -> Result { let bytes = case.ssz_bytes(name); match case.fork { @@ -165,9 +232,11 @@ fn decode_signed_aggregate(case: &Case, name: &str) -> Result { - Err(format!("no aggregate shape for fork {:?}", case.fork)) - } + ForkName::Gloas => Ok(SignedAggregateAndProof::Gloas( + gloas::SignedAggregateAndProof::from_ssz_bytes(&bytes) + .map_err(|err| format!("decoding {name}: {err:?}"))?, + )), + ForkName::Lean => Err(format!("no aggregate shape for fork {:?}", case.fork)), } } @@ -188,6 +257,31 @@ fn case_config(case: &Case, state: &BeaconState) -> Config { config } +/// Delivers `entry`'s execution payload envelope, if it lists one, so +/// `is_payload_verified` answers true for its block. +fn deliver_payload( + store: &mut Store, + case: &Case, + entry: &StoreBlock, + config: &Config, +) -> Result<(), String> { + let Some(name) = &entry.payload else { + return Ok(()); + }; + let envelope = gloas::SignedExecutionPayloadEnvelope::from_ssz_bytes(&case.ssz_bytes(name)) + .map_err(|err| format!("decoding {name}: {err:?}"))?; + // No sampled columns are named, so the empty retrieval reads as available, + // as in the fork-choice runner's envelope step. + fork_choice::on_execution_payload_envelope( + store, + &envelope, + config, + &[], + &ExecutionEngine::valid(), + ) + .map_err(|err| format!("delivering {name}: {err:?}")) +} + /// The store the case describes: its anchor, then each listed block. fn build_store( case: &Case, @@ -205,23 +299,41 @@ fn build_store( .map_err(|err| format!("get_forkchoice_store: {err:?}"))?; // The store's clock at the case's base time, so `on_block` accepts every - // listed block. - let now_s = (config.genesis_time_ms() + meta.current_time_ms) / 1000; - fork_choice::on_tick(&mut store, now_s, config); + // listed block; a block from a slot past that time advances it to that + // slot's start, since a vector may place its base time just before the + // slot a clock-disparity case sends its sidecar for. + let mut clock_s = (config.genesis_time_ms() + meta.current_time_ms) / 1000; + fork_choice::on_tick(&mut store, clock_s, config); + deliver_payload(&mut store, case, anchor, config)?; for entry in rest { let block = decode_block(case, &entry.block)?; let root = block.message_hash_tree_root(); - // Seen without a post-state. An `INVALIDATED` payload lands here too: - // this store never keeps a post-state for one, since `on_block` fails - // it and invalidates the branch. - if entry.failed || entry.pending || entry.payload_status.as_deref() == Some("INVALIDATED") { + let block_slot = block.slot(); + let block_start_s = + (config.genesis_time_ms() + block.slot() * config.slot_duration_ms) / 1000; + if block_start_s > clock_s { + clock_s = block_start_s; + fork_choice::on_tick(&mut store, clock_s, config); + } + // Gloas's `payload_status` is the verdict on the block's envelope, not + // on the block, which is imported regardless; it feeds the attestation + // rules' `block_payload_statuses` instead. + let gloas = case.fork == ForkName::Gloas; + // Seen without a post-state. A pre-gloas `INVALIDATED` payload lands + // here too: this store never keeps a post-state for one, since + // `on_block` fails it and invalidates the branch. + if entry.failed + || entry.pending + || (!gloas && entry.payload_status.as_deref() == Some("INVALIDATED")) + { store .insert_pending_block(root, block) .map_err(|err| format!("storing {}: {err}", entry.block))?; continue; } let validity = match entry.payload_status.as_deref() { + _ if gloas => PayloadValidity::NotRequired, None => PayloadValidity::NotRequired, Some("VALID") => PayloadValidity::Validated, Some("NOT_VALIDATED") => PayloadValidity::Optimistic, @@ -236,6 +348,16 @@ fn build_store( &CommitteeCache::default(), ) .map_err(|err| format!("importing {}: {err:?}", entry.block))?; + deliver_payload(&mut store, case, entry, config)?; + if gloas && let Some(status) = entry.payload_status.as_deref() { + let status = match status { + "VALID" => PayloadStatusEnum::Valid, + "NOT_VALIDATED" => PayloadStatusEnum::Syncing, + "INVALIDATED" => PayloadStatusEnum::Invalid, + other => return Err(format!("unknown payload_status {other}")), + }; + store.insert_beacon_block_payload_status(root, block_slot, status); + } } if let Some(finalized) = &meta.finalized_checkpoint { @@ -288,11 +410,17 @@ fn run_case(case: &Case) -> Result<(), String> { let capacity = NonZeroUsize::new(SEEN_CAPACITY).expect("non-zero"); let mut seen_blocks = SeenBlocks::new(capacity); let mut seen_columns = SeenColumns::new(capacity); + let mut seen_block_columns = SeenBlockColumns::new(capacity); let mut seen_aggregates = SeenAggregates::new(capacity, capacity); let mut seen_attestations = SeenAttestations::new(capacity); + let mut seen_envelopes = SeenEnvelopes::new(capacity); + let mut seen_payload_attestations = SeenPayloadAttestations::new(capacity); for (index, message) in meta.messages.iter().enumerate() { - let now_ms = config.genesis_time_ms() + meta.current_time_ms + message.offset_ms; + let now_ms = config.genesis_time_ms() + + message + .arrival_ms(meta.current_time_ms) + .map_err(|err| format!("message {index}: {err}"))?; let outcome = match meta.topic.as_str() { "beacon_block" => { let block = decode_block(case, &message.message)?; @@ -305,18 +433,40 @@ fn run_case(case: &Case) -> Result<(), String> { } "data_column_sidecar" => { let sidecar = - fulu::DataColumnSidecar::from_ssz_bytes(&case.ssz_bytes(&message.message)) + DataColumnSidecar::from_ssz(case.fork, &case.ssz_bytes(&message.message)) .map_err(|err| format!("decoding {}: {err:?}", message.message))?; let subnet_id = message .subnet_id .ok_or("a data_column_sidecar message names its subnet")?; - let outcome = - rules::column::validate(&seen_columns, &store, &sidecar, subnet_id, now_ms); - if outcome == Outcome::Accept { - let header = &sidecar.signed_block_header.message; - seen_columns.record(header.slot, header.proposer_index, sidecar.index); + match sidecar { + DataColumnSidecar::Fulu(sidecar) => { + let outcome = rules::column::validate( + &seen_columns, + &store, + &sidecar, + subnet_id, + now_ms, + ); + if outcome == Outcome::Accept { + let header = &sidecar.signed_block_header.message; + seen_columns.record(header.slot, header.proposer_index, sidecar.index); + } + outcome + } + DataColumnSidecar::Gloas(sidecar) => { + let outcome = rules::column::validate_gloas( + &seen_block_columns, + &store, + &sidecar, + subnet_id, + now_ms, + ); + if outcome == Outcome::Accept { + seen_block_columns.record(sidecar.beacon_block_root, sidecar.index); + } + outcome + } } - outcome } "beacon_aggregate_and_proof" => { let aggregate = decode_signed_aggregate(case, &message.message)?; @@ -353,6 +503,37 @@ fn run_case(case: &Case) -> Result<(), String> { } outcome } + "execution_payload" => { + let envelope = gloas::SignedExecutionPayloadEnvelope::from_ssz_bytes( + &case.ssz_bytes(&message.message), + ) + .map_err(|err| format!("decoding {}: {err:?}", message.message))?; + let outcome = rules::envelope::validate(&seen_envelopes, &store, &envelope); + if outcome == Outcome::Accept { + seen_envelopes.record( + envelope.message.beacon_block_root, + envelope.message.builder_index, + ); + } + outcome + } + "payload_attestation_message" => { + let attestation = gloas::PayloadAttestationMessage::from_ssz_bytes( + &case.ssz_bytes(&message.message), + ) + .map_err(|err| format!("decoding {}: {err:?}", message.message))?; + let outcome = rules::payload_attestation::validate( + &seen_payload_attestations, + &store, + &attestation, + now_ms, + ); + if outcome == Outcome::Accept { + seen_payload_attestations + .record(attestation.data.slot, attestation.validator_index); + } + outcome + } other => return Err(format!("topic {other} has no runner")), }; check(message, outcome).map_err(|err| format!("message {index}: {err}"))?; @@ -369,21 +550,10 @@ pub fn trials() -> Vec { cases.len(), )); for case in cases { - // The block and column rules (`beacon::gossip::{block, column}`) - // are fulu's own `validate_beacon_block_gossip` and - // `validate_data_column_sidecar_gossip`. The aggregate and - // attestation rules (`beacon::gossip::{aggregate, attestation}`) - // are electra's (`p2p-interface.md`), which fulu keeps. Either - // way a case from any other fork is ignored rather than run, the - // same "known gap, not a silent one" treatment `case.in_scope()` - // already gives a fork - // past `HIGHEST_IMPLEMENTED_FORK`. `case.in_scope()` alone would - // pass every fork up to gloas, since the state transition handles - // them all; the gossip rules do not. In particular the node does - // not validate gloas gossip yet, so a gloas case stays ignored - // here even though its state transition runs. + // A fork the handler's rules do not cover is ignored rather than + // run; see [`validated_forks`]. let ignored = !case.in_scope() - || case.fork != ForkName::Fulu + || !validated_forks(handler).contains(&case.fork) || SKIPPED.iter().any(|(name, _)| *name == case.name); trials.push(super::case_trial("gossip", case, run_case).with_ignored_flag(ignored)); } diff --git a/crates/common/types/src/beacon/containers/mod.rs b/crates/common/types/src/beacon/containers/mod.rs index d624b1e9c..1e8b09e1c 100644 --- a/crates/common/types/src/beacon/containers/mod.rs +++ b/crates/common/types/src/beacon/containers/mod.rs @@ -1061,13 +1061,26 @@ impl BeaconState { } } +/// The one committee `bits` names, or `None` for none or several. +fn named_committee(bits: &electra::CommitteeBits) -> Option { + let mut named = (0..bits.len()).filter(|&index| bits.get(index).unwrap_or(false)); + let first = named.next()?; + // A second named committee disqualifies the aggregate outright. + match named.next() { + None => Some(first as CommitteeIndex), + Some(_) => None, + } +} + /// An aggregate attestation with the proof its aggregator was selected, in /// whichever fork's shape it currently has. /// -/// Two variants, not one per fork, for the reason +/// Three variants, not one per fork, for the reason /// [`SignedBeaconBlock::Fulu`] wraps electra's block: every fork through deneb /// shares [`phase0::SignedAggregateAndProof`] outright, and fulu shares -/// electra's the same way. +/// electra's the same way. Gloas has its own: same bytes on the wire, but its +/// `Attestation` is a progressive container (EIP-7688), so the +/// `hash_tree_root` the aggregator's signature covers differs. /// /// Here rather than beside the gossip decode in `ethlambda-p2p`, where it was /// first declared, because the gossip path no longer ends at that decode: an @@ -1083,6 +1096,7 @@ impl BeaconState { pub enum SignedAggregateAndProof { Phase0(phase0::SignedAggregateAndProof), Electra(electra::SignedAggregateAndProof), + Gloas(gloas::SignedAggregateAndProof), } impl SignedAggregateAndProof { @@ -1091,6 +1105,7 @@ impl SignedAggregateAndProof { match self { Self::Phase0(signed) => signed.message.aggregator_index, Self::Electra(signed) => signed.message.aggregator_index, + Self::Gloas(signed) => signed.message.aggregator_index, } } @@ -1099,6 +1114,7 @@ impl SignedAggregateAndProof { match self { Self::Phase0(signed) => signed.message.aggregate.data.slot, Self::Electra(signed) => signed.message.aggregate.data.slot, + Self::Gloas(signed) => signed.message.aggregate.data.slot, } } @@ -1107,6 +1123,7 @@ impl SignedAggregateAndProof { match self { Self::Phase0(signed) => signed.message.aggregate.data, Self::Electra(signed) => signed.message.aggregate.data, + Self::Gloas(signed) => signed.message.aggregate.data, } } @@ -1122,6 +1139,7 @@ impl SignedAggregateAndProof { match self { Self::Phase0(signed) => signed.message.selection_proof, Self::Electra(signed) => signed.message.selection_proof, + Self::Gloas(signed) => signed.message.selection_proof, } } @@ -1130,6 +1148,7 @@ impl SignedAggregateAndProof { match self { Self::Phase0(signed) => signed.signature, Self::Electra(signed) => signed.signature, + Self::Gloas(signed) => signed.signature, } } @@ -1152,16 +1171,21 @@ impl SignedAggregateAndProof { pub fn committee_index(&self) -> Option { match self { Self::Phase0(signed) => Some(signed.message.aggregate.data.index), - Self::Electra(signed) => { - let bits = &signed.message.aggregate.committee_bits; - let mut named = (0..bits.len()).filter(|&index| bits.get(index).unwrap_or(false)); - let first = named.next()?; - // A second named committee disqualifies the aggregate outright. - match named.next() { - None => Some(first as CommitteeIndex), - Some(_) => None, - } - } + // Gloas reuses electra's `CommitteeBits` outright. + Self::Electra(signed) => named_committee(&signed.message.aggregate.committee_bits), + Self::Gloas(signed) => named_committee(&signed.message.aggregate.committee_bits), + } + } + + /// The length of the aggregation bitfield, read without expanding it. + /// + /// For callers that must bound the bitfield before anything iterates it: + /// gloas's is unbounded by its type. + pub fn aggregation_bits_len(&self) -> usize { + match self { + Self::Phase0(signed) => signed.message.aggregate.aggregation_bits.len(), + Self::Electra(signed) => signed.message.aggregate.aggregation_bits.len(), + Self::Gloas(signed) => signed.message.aggregate.aggregation_bits.len(), } } @@ -1175,6 +1199,7 @@ impl SignedAggregateAndProof { match self { Self::Phase0(signed) => signed.message.aggregate.aggregation_bits.count_ones(), Self::Electra(signed) => signed.message.aggregate.aggregation_bits.count_ones(), + Self::Gloas(signed) => signed.message.aggregate.aggregation_bits.count_ones(), } } @@ -1198,6 +1223,12 @@ impl SignedAggregateAndProof { .map(|i| bits.get(i).unwrap_or(false)) .collect() } + Self::Gloas(signed) => { + let bits = &signed.message.aggregate.aggregation_bits; + (0..bits.len()) + .map(|i| bits.get(i).unwrap_or(false)) + .collect() + } } } } @@ -1515,6 +1546,90 @@ signed_beacon_block_accessors!( ], ); +/// A data column sidecar in either shape: fulu's carries a signed header and +/// an inclusion proof, gloas's names its block by root and reads its +/// commitments from that block's bid. +/// +/// Two variants rather than one per fork, since the shape changes only at +/// gloas; [`DataColumnSidecar::fork`] answers which fork's rules apply. +// Fulu's carries a header and an inclusion proof inline, so the variants differ +// in size; a sidecar is held and passed by value, and boxing one would only make +// every reader dereference it. +#[allow(clippy::large_enum_variant)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum DataColumnSidecar { + Fulu(fulu::DataColumnSidecar), + Gloas(gloas::DataColumnSidecar), +} + +impl DataColumnSidecar { + /// The column this sidecar carries. + pub fn index(&self) -> u64 { + match self { + Self::Fulu(sidecar) => sidecar.index, + Self::Gloas(sidecar) => sidecar.index, + } + } + + /// The slot of the block this sidecar belongs to. + pub fn slot(&self) -> Slot { + match self { + Self::Fulu(sidecar) => sidecar.signed_block_header.message.slot, + Self::Gloas(sidecar) => sidecar.slot, + } + } + + /// The root of the block this sidecar belongs to: fulu's is the header's + /// hash tree root, gloas's is named outright. + pub fn block_root(&self) -> Root { + match self { + Self::Fulu(sidecar) => sidecar.signed_block_header.message.hash_tree_root(), + Self::Gloas(sidecar) => sidecar.beacon_block_root, + } + } + + /// The fork whose rules apply to this sidecar. + pub fn fork(&self) -> ForkName { + match self { + Self::Fulu(_) => ForkName::Fulu, + Self::Gloas(_) => ForkName::Gloas, + } + } + + /// Decodes a sidecar of a known fork; the bytes carry no tag, so the fork + /// comes from context (the gossip topic's digest, a request's fork digest). + /// + /// Forks before fulu have no data columns and lean has none at all, so + /// they answer with an error rather than a panic: the fork comes off the + /// wire. + pub fn from_ssz(fork: ForkName, bytes: &[u8]) -> Result { + match fork { + ForkName::Fulu => Ok(Self::Fulu(fulu::DataColumnSidecar::from_ssz_bytes(bytes)?)), + ForkName::Gloas => Ok(Self::Gloas(gloas::DataColumnSidecar::from_ssz_bytes( + bytes, + )?)), + ForkName::Phase0 + | ForkName::Altair + | ForkName::Bellatrix + | ForkName::Capella + | ForkName::Deneb + | ForkName::Electra + | ForkName::Lean => Err(Error::UnsupportedForFork { + function: "DataColumnSidecar", + fork, + }), + } + } + + /// Encodes the sidecar. + pub fn to_ssz(&self) -> Vec { + match self { + Self::Fulu(sidecar) => sidecar.to_ssz(), + Self::Gloas(sidecar) => sidecar.to_ssz(), + } + } +} + #[cfg(test)] mod tests { use super::*; @@ -2204,4 +2319,79 @@ mod tests { assert_eq!(block.blob_kzg_commitment_count(), 0); assert_eq!(block.execution_payload_timestamp(), None); } + + fn fulu_sidecar(index: u64, header: BeaconBlockHeader) -> fulu::DataColumnSidecar { + fulu::DataColumnSidecar { + index, + column: Default::default(), + kzg_commitments: Default::default(), + kzg_proofs: Default::default(), + signed_block_header: SignedBeaconBlockHeader { + message: header, + signature: Default::default(), + }, + kzg_commitments_inclusion_proof: vec![ + Root::ZERO; + preset::KZG_COMMITMENTS_INCLUSION_PROOF_DEPTH + ] + .try_into() + .unwrap(), + } + } + + #[test] + fn a_data_column_sidecar_round_trips_per_fork() { + let header = BeaconBlockHeader { + slot: 9, + ..Default::default() + }; + let fulu = DataColumnSidecar::Fulu(fulu_sidecar(7, header)); + let gloas = DataColumnSidecar::Gloas(gloas::DataColumnSidecar { + index: 3, + slot: 11, + beacon_block_root: Root::from([5; 32]), + ..Default::default() + }); + for sidecar in [fulu, gloas] { + let decoded = DataColumnSidecar::from_ssz(sidecar.fork(), &sidecar.to_ssz()).unwrap(); + assert_eq!(decoded, sidecar); + } + } + + #[test] + fn a_data_column_sidecar_reports_its_block_root_per_variant() { + let header = BeaconBlockHeader { + slot: 9, + proposer_index: 2, + ..Default::default() + }; + let fulu = DataColumnSidecar::Fulu(fulu_sidecar(7, header.clone())); + assert_eq!(fulu.block_root(), header.hash_tree_root()); + assert_eq!((fulu.index(), fulu.slot()), (7, 9)); + assert_eq!(fulu.fork(), ForkName::Fulu); + + let root = Root::from([5; 32]); + let gloas = DataColumnSidecar::Gloas(gloas::DataColumnSidecar { + index: 3, + slot: 11, + beacon_block_root: root, + ..Default::default() + }); + assert_eq!(gloas.block_root(), root); + assert_eq!((gloas.index(), gloas.slot()), (3, 11)); + assert_eq!(gloas.fork(), ForkName::Gloas); + } + + #[test] + fn a_data_column_sidecar_does_not_decode_before_fulu_or_as_lean() { + let valid = DataColumnSidecar::Fulu(fulu_sidecar(7, BeaconBlockHeader::default())).to_ssz(); + // Bytes that decode fine as fulu's, so the error can only be the fork. + assert!(DataColumnSidecar::from_ssz(ForkName::Fulu, &valid).is_ok()); + for fork in [ForkName::Electra, ForkName::Lean] { + assert!(matches!( + DataColumnSidecar::from_ssz(fork, &valid), + Err(Error::UnsupportedForFork { fork: got, .. }) if got == fork + )); + } + } } diff --git a/crates/common/types/src/beacon/fork.rs b/crates/common/types/src/beacon/fork.rs index 79584df4c..c3ccd42a2 100644 --- a/crates/common/types/src/beacon/fork.rs +++ b/crates/common/types/src/beacon/fork.rs @@ -112,12 +112,13 @@ impl ForkName { /// Whether this node follows a chain that has reached this fork. /// - /// Phase0 through fulu are followed. Gloas is not: nothing delivers the - /// payload envelopes or payload attestations a gloas chain needs, so the - /// node refuses gloas blocks and anchors, ignores gloas gossip and reports - /// syncing once the clock reaches the fork. This is the one place to - /// change when the node starts following gloas; every site whose rule is - /// "does this node follow the fork" calls it. + /// Every beacon fork is followed. Gloas needs the payload envelopes and + /// payload attestations that phase0 through fulu do not: the chain actor + /// imports the envelopes, the network side delivers them and the two + /// gloas gossip topics, and fork choice keeps the head's payload status. + /// The question stays a function so a fork this node cannot follow has + /// one place to say so; every site whose rule is "does this node follow + /// the fork" calls it. /// /// # Panics /// @@ -131,12 +132,33 @@ impl ForkName { | ForkName::Capella | ForkName::Deneb | ForkName::Electra - | ForkName::Fulu => true, - ForkName::Gloas => false, + | ForkName::Fulu + | ForkName::Gloas => true, ForkName::Lean => super::lean_fork_unreachable("ForkName::is_followed"), } } + /// Whether blocks of this fork reveal their execution payload in a separate + /// envelope (gloas, EIP-7732, and any fork after it). + /// + /// An explicit match rather than `self >= ForkName::Gloas`: the derived + /// order puts [`ForkName::Lean`] after every beacon fork, so that + /// comparison is true for lean, which has no envelopes. A fork added after + /// gloas must be listed here, so the compiler forces the decision. + pub const fn has_payload_envelopes(self) -> bool { + match self { + ForkName::Gloas => true, + ForkName::Phase0 + | ForkName::Altair + | ForkName::Bellatrix + | ForkName::Capella + | ForkName::Deneb + | ForkName::Electra + | ForkName::Fulu + | ForkName::Lean => false, + } + } + /// The one-byte tag this fork is stored under in a `States` value. /// /// Spelled out rather than `self as u8`. The variant order is already @@ -221,6 +243,14 @@ mod tests { assert!(ForkName::Fulu > ForkName::Phase0); } + #[test] + fn only_gloas_and_later_beacon_forks_have_payload_envelopes() { + for fork in ForkName::ALL { + assert_eq!(fork.has_payload_envelopes(), fork >= ForkName::Gloas); + } + assert!(!ForkName::Lean.has_payload_envelopes()); + } + #[test] fn parse_round_trips_every_fork() { for fork in ForkName::ALL { @@ -300,9 +330,9 @@ mod tests { } #[test] - fn every_beacon_fork_through_fulu_is_followed_and_gloas_is_not() { + fn every_beacon_fork_through_gloas_is_followed() { for fork in ForkName::ALL { - assert_eq!(fork.is_followed(), fork != ForkName::Gloas, "{fork:?}"); + assert!(fork.is_followed(), "{fork:?}"); } } diff --git a/crates/common/types/src/beacon/fork_digest.rs b/crates/common/types/src/beacon/fork_digest.rs index 69a42488a..33e40475e 100644 --- a/crates/common/types/src/beacon/fork_digest.rs +++ b/crates/common/types/src/beacon/fork_digest.rs @@ -10,7 +10,6 @@ use sha2::{Digest as _, Sha256}; use crate::beacon::config::Config; -use crate::beacon::constants; use crate::beacon::containers::shared::ForkData; use crate::beacon::fork::ForkName; use crate::beacon::primitives::{Epoch, ForkDigest, HashTreeRoot as _, Root, Version}; @@ -56,21 +55,6 @@ pub fn compute_fork_digest( core::array::from_fn(|index| base.0[index] ^ mask[index]) } -/// The next epoch at which [`compute_fork_digest`] changes, if there is one. -/// -/// Both fork activations and blob-schedule entries qualify: crossing either one -/// strands a running node on topic names no peer is publishing to. Unscheduled -/// forks carry [`constants::FAR_FUTURE_EPOCH`], a real value rather than a -/// `None`, so they are filtered out before the minimum is taken. -pub fn next_fork_boundary(config: &Config, epoch: Epoch) -> Option { - ForkName::ALL - .into_iter() - .map(|fork| config.fork_epoch(fork)) - .chain(config.blob_schedule.iter().map(|entry| entry.epoch)) - .filter(|&boundary| boundary != constants::FAR_FUTURE_EPOCH && boundary > epoch) - .min() -} - #[cfg(test)] mod tests { use super::*; @@ -130,26 +114,6 @@ mod tests { ); } - #[test] - fn next_boundary_covers_both_fork_and_blob_schedule_epochs() { - let config = Config::mainnet(); - // A plain fork boundary. - assert_eq!(next_fork_boundary(&config, 0), Some(74_240)); - // A blob-parameter-only fork is a boundary too: it moves the digest. - assert_eq!(next_fork_boundary(&config, 411_392), Some(412_672)); - assert_eq!(next_fork_boundary(&config, 412_672), Some(419_072)); - // Past the last scheduled boundary there is nothing left to warn about. - assert_eq!(next_fork_boundary(&config, 419_072), None); - } - - #[test] - fn far_future_forks_are_not_boundaries() { - // Minimal leaves every fork after phase0 at FAR_FUTURE_EPOCH, which is a - // real, enormous Epoch rather than a None; treating it as a boundary - // would schedule a warning for the heat death of the universe. - assert_eq!(next_fork_boundary(&Config::minimal(), 0), None); - } - #[test] fn fork_data_root_binds_the_version_and_the_chain() { let a = compute_fork_data_root([1, 0, 0, 0], Root::ZERO); diff --git a/crates/net/api/src/lib.rs b/crates/net/api/src/lib.rs index f310132e8..4d61d2969 100644 --- a/crates/net/api/src/lib.rs +++ b/crates/net/api/src/lib.rs @@ -3,8 +3,9 @@ use std::time::Instant; use ethlambda_types::{ attestation::{SignedAggregatedAttestation, SignedAttestation}, beacon::containers::{ - SignedAggregateAndProof, SignedBeaconBlock, electra::SingleAttestation, - fulu::DataColumnSidecar, + DataColumnSidecar, SignedAggregateAndProof, SignedBeaconBlock, + electra::SingleAttestation, + gloas::{PayloadAttestationMessage, SignedExecutionPayloadEnvelope}, }, beacon::primitives::ValidatorIndex, block::SignedBlock, @@ -27,7 +28,8 @@ pub trait BlockChainToP2P: Send + Sync { /// Ask peers for whatever of one block this node is missing. fn fetch_block(&self, request: FetchRequest) -> Result<(), ActorError>; /// Run the chain checks on sidecars the chain actor had parked, now that - /// their parent has a post-state. + /// the block they are checked against has a post-state: their parent for a + /// fulu sidecar, their own block for a gloas one. /// /// The chain actor keeps a sidecar without checking it, so it hands these /// back rather than judging them itself: the p2p layer runs every column @@ -62,6 +64,10 @@ pub struct FetchRequest { /// reading its own store to find out would pay a DB read per request to /// re-derive what the caller already knew. pub needs_block: bool, + /// Whether the block's execution payload envelope is missing (gloas): the + /// block is known, and it names a payload this node has no envelope for. + /// Fetched by root, like the block. + pub needs_envelope: bool, /// Columns of this block that this node custodies and does not have. /// /// Empty when nothing is missing, or when the block itself is, since a @@ -178,6 +184,21 @@ pub struct DeferredFrom { } impl BlockArrival { + /// The Unix millisecond this delivery came off the wire, given that the + /// actor picked it up at `picked_up` (an [`Instant`]) at `picked_up_ms` + /// (Unix milliseconds). + /// + /// The wire receipt predates validation and the mailbox wait, and it is + /// what a sub-slot deadline should be judged against. Meant for every + /// delivery carrying a [`BlockArrival`], blocks, envelopes and payload + /// attestations alike. + pub fn wire_ms(&self, picked_up: Instant, picked_up_ms: u64) -> u64 { + let received = self.decode_start.unwrap_or(self.handed_off); + let in_flight_ms = u64::try_from(picked_up.saturating_duration_since(received).as_millis()) + .unwrap_or(u64::MAX); + picked_up_ms.saturating_sub(in_flight_ms) + } + /// An arrival whose earliest knowable moment is now. /// /// For producers that did not decode the block themselves, so have no @@ -237,9 +258,10 @@ pub trait P2PToBlockChain: Send + Sync { /// again. fn new_data_column_sidecars(&self, sidecars: Vec) -> Result<(), ActorError>; /// Data column sidecars the chain checks could not judge yet, because - /// their parent has no post-state: the chain actor parks them and sends - /// them back through [`BlockChainToP2P::check_data_column_sidecars`] once - /// the parent imports. + /// the block they are judged against has no post-state (a fulu sidecar's + /// parent, a gloas sidecar's own block): the chain actor parks them and + /// sends them back through [`BlockChainToP2P::check_data_column_sidecars`] + /// once that block imports. fn data_column_sidecars_awaiting_parent( &self, sidecars: Vec, @@ -279,6 +301,27 @@ pub trait P2PToBlockChain: Send + Sync { attesting_indices: Vec, arrival: AggregateArrival, ) -> Result<(), ActorError>; + /// A gloas execution payload envelope that passed gossip validation (or + /// was fetched), for the chain actor to verify against its block's + /// post-state and apply to fork choice. + /// + /// Boxed for the reason [`Self::new_beacon_aggregate`] is: the envelope + /// carries a whole execution payload. `arrival` is the same shape a block + /// brings, since the actor reads the clock the envelope arrived at to + /// judge its payload timeliness. + fn new_execution_payload_envelope( + &self, + envelope: Box, + arrival: BlockArrival, + ) -> Result<(), ActorError>; + /// A gloas payload attestation message that passed gossip validation, for + /// the chain actor to apply to fork choice. Not queued when its block is + /// not imported yet: a vote is only valid within its own slot. + fn new_payload_attestation_message( + &self, + message: PayloadAttestationMessage, + arrival: BlockArrival, + ) -> Result<(), ActorError>; } /// When an aggregate reached this node, for the one metric that cannot be diff --git a/crates/net/engine/src/client.rs b/crates/net/engine/src/client.rs index 00418181a..25a4db2e9 100644 --- a/crates/net/engine/src/client.rs +++ b/crates/net/engine/src/client.rs @@ -2,7 +2,7 @@ use std::time::Duration; -use ethlambda_types::beacon::containers::deneb; +use ethlambda_types::beacon::containers::{deneb, gloas}; use ethlambda_types::beacon::primitives::{Bytes32, Root}; use serde_json::json; use tracing::{debug, warn}; @@ -10,8 +10,8 @@ use tracing::{debug, warn}; use crate::auth::JwtSecret; use crate::error::EngineError; use crate::types::{ - ClientVersionV1, ExecutionPayloadV3, ForkchoiceStateV1, ForkchoiceUpdatedResponse, - PayloadStatusV1, data, + ClientVersionV1, CustodyColumns, ExecutionPayloadV3, ExecutionPayloadV4, ForkchoiceStateV1, + ForkchoiceUpdatedResponse, PayloadStatusV1, data, }; /// Per-attempt timeout. @@ -21,6 +21,11 @@ use crate::types::{ /// `newPayload` on mainnet answers in 50-500 ms. pub const ENGINE_TIMEOUT: Duration = Duration::from_secs(8); +/// Per-attempt timeout for `engine_newPayloadV5`, the value `amsterdam.md` +/// gives that method. Tighter than [`ENGINE_TIMEOUT`], which stays the default +/// for the older methods. +pub const ENGINE_NEW_PAYLOAD_V5_TIMEOUT: Duration = Duration::from_secs(6); + /// How many times one call is attempted before it is given up on. /// /// Three, not the ten `ethlambda-p2p` uses for a peer fetch. Those are different @@ -86,12 +91,22 @@ impl EngineClient { &self, method: &str, params: serde_json::Value, + ) -> Result { + self.call_with_timeout(method, params, ENGINE_TIMEOUT).await + } + + /// [`Self::call`] with a per-attempt `timeout` of the method's own. + async fn call_with_timeout( + &self, + method: &str, + params: serde_json::Value, + timeout: Duration, ) -> Result { let mut backoff = ENGINE_INITIAL_BACKOFF; let mut last = EngineError::Transport("no attempt was made".to_string()); for attempt in 1..=ENGINE_MAX_ATTEMPTS { - match self.call_once(method, ¶ms).await { + match self.call_once(method, ¶ms, timeout).await { Ok(value) => return Ok(value), // An answer, not a failure to get one. Do not retry. Err(err @ EngineError::Rpc { .. }) => return Err(err), @@ -113,6 +128,7 @@ impl EngineClient { &self, method: &str, params: &serde_json::Value, + timeout: Duration, ) -> Result { let body = json!({ "jsonrpc": "2.0", @@ -125,12 +141,13 @@ impl EngineClient { .http .post(&self.endpoint) .bearer_auth(self.secret.token()) + .timeout(timeout) .json(&body) .send() .await .map_err(|err| { if err.is_timeout() { - EngineError::Timeout(ENGINE_TIMEOUT) + EngineError::Timeout(timeout) } else { EngineError::Transport(err.to_string()) } @@ -202,6 +219,36 @@ impl EngineClient { self.call("engine_newPayloadV4", params).await } + /// `engine_newPayloadV5`: Amsterdam's `newPayload`, which gloas asks about + /// a revealed execution payload envelope. + /// + /// The parameters are the same four as V4's, with the payload as + /// `ExecutionPayloadV4` (adding `blockAccessList` and `slotNumber`). The + /// response is V4's. An execution client answers `-38005` when the + /// payload's timestamp is not Amsterdam's, which surfaces as + /// [`EngineError::Rpc`] and is not retried. + pub async fn new_payload_v5( + &self, + payload: &gloas::ExecutionPayload, + versioned_hashes: &[Bytes32], + parent_beacon_block_root: Root, + execution_requests: &[Vec], + ) -> Result { + let hashes: Vec = versioned_hashes.iter().map(|hash| data(&hash.0)).collect(); + let requests: Vec = execution_requests + .iter() + .map(|request| data(request)) + .collect(); + let params = json!([ + ExecutionPayloadV4(payload), + hashes, + data(&parent_beacon_block_root.0), + requests, + ]); + self.call_with_timeout("engine_newPayloadV5", params, ENGINE_NEW_PAYLOAD_V5_TIMEOUT) + .await + } + /// `engine_forkchoiceUpdatedV3`, always with a `null` `payloadAttributes`. /// /// A follower never proposes, so it never asks an execution client to start @@ -217,6 +264,24 @@ impl EngineClient { Ok(response.payload_status) } + /// `engine_forkchoiceUpdatedV4`: Amsterdam's fork choice notification, with + /// a `null` `payloadAttributes` and the consensus client's custody set. + /// + /// Follower only: there is no variant taking `PayloadAttributesV4`, since a + /// follower never asks for a build. `custody_columns` of `None` sends + /// `null`, meaning the consensus client provides no custody. The response + /// is V3's. + pub async fn forkchoice_updated_v4( + &self, + state: &ForkchoiceStateV1, + custody_columns: Option, + ) -> Result { + let params = forkchoice_updated_v4_params(state, custody_columns); + let response: ForkchoiceUpdatedResponse = + self.call("engine_forkchoiceUpdatedV4", params).await?; + Ok(response.payload_status) + } + /// `engine_forkchoiceUpdatedV3` with `payloadAttributes`: the same fork /// choice notification, plus a request to start building a payload on the /// head for the slot the attributes describe. @@ -264,10 +329,17 @@ impl EngineClient { /// Runs the startup handshake, logging what the execution client is and /// warning about any method this client needs that it does not advertise. /// + /// `gloas_scheduled` says whether the network's fork schedule includes + /// gloas, which is what makes `engine_newPayloadV5` a needed method. + /// /// Warns rather than refuses: an execution client that under-reports its /// capabilities still works, and refusing to start over a handshake would /// turn a cosmetic mismatch into an outage. - pub async fn handshake(&self, ours: &ClientVersionV1) -> Result<(), EngineError> { + pub async fn handshake( + &self, + ours: &ClientVersionV1, + gloas_scheduled: bool, + ) -> Result<(), EngineError> { let theirs = self .exchange_capabilities(crate::ETHLAMBDA_ENGINE_CAPABILITIES) .await?; @@ -279,6 +351,19 @@ impl EngineClient { ); } } + // Kept out of the list above: these are only called from gloas on, so a + // network that does not schedule gloas never needs it, and an + // execution client that predates Amsterdam is right not to offer it. + if gloas_scheduled { + for required in ["engine_newPayloadV5", "engine_forkchoiceUpdatedV4"] { + if !theirs.iter().any(|method| method == required) { + warn!( + method = required, + "The execution client does not advertise a method this node needs from gloas on" + ); + } + } + } match self.client_version(ours).await { Ok(versions) => { for version in versions { @@ -297,3 +382,46 @@ impl EngineClient { Ok(()) } } + +/// The three positional parameters of `engine_forkchoiceUpdatedV4`: the state, +/// a `null` `payloadAttributes`, and the custody bitmap or `null`. +fn forkchoice_updated_v4_params( + state: &ForkchoiceStateV1, + custody_columns: Option, +) -> serde_json::Value { + json!([state, serde_json::Value::Null, custody_columns]) +} + +#[cfg(test)] +mod tests { + use ethlambda_types::beacon::primitives::ExecutionBlockHash; + + use super::*; + + fn state() -> ForkchoiceStateV1 { + ForkchoiceStateV1 { + head_block_hash: ExecutionBlockHash::repeat_byte(1), + safe_block_hash: ExecutionBlockHash::repeat_byte(2), + finalized_block_hash: ExecutionBlockHash::repeat_byte(3), + } + } + + #[test] + fn forkchoice_updated_v4_params_are_state_null_and_bitmap() { + let columns = CustodyColumns::from_indices([0, 9]); + let params = forkchoice_updated_v4_params(&state(), columns); + let params = params.as_array().expect("an array of params"); + assert_eq!(params.len(), 3); + assert!(params[0].get("headBlockHash").is_some()); + assert!(params[1].is_null()); + assert_eq!(params[2], "0x01020000000000000000000000000000"); + } + + #[test] + fn forkchoice_updated_v4_without_custody_sends_a_third_null() { + let params = forkchoice_updated_v4_params(&state(), None); + let params = params.as_array().expect("an array of params"); + assert_eq!(params.len(), 3); + assert!(params[2].is_null()); + } +} diff --git a/crates/net/engine/src/error.rs b/crates/net/engine/src/error.rs index 4d2c717cd..0e575e777 100644 --- a/crates/net/engine/src/error.rs +++ b/crates/net/engine/src/error.rs @@ -19,4 +19,9 @@ pub enum EngineError { Rpc { code: i64, message: String }, #[error("decoding the response: {0}")] Decode(String), + /// A caller asked about an envelope that does not belong to the block it + /// paired it with. Raised before any call is made, so it is no verdict on + /// the payload itself: the caller must not import either. + #[error("envelope does not match its block: {0}")] + EnvelopeMismatch(String), } diff --git a/crates/net/engine/src/lib.rs b/crates/net/engine/src/lib.rs index 922ec474a..f3cf1d737 100644 --- a/crates/net/engine/src/lib.rs +++ b/crates/net/engine/src/lib.rs @@ -11,7 +11,7 @@ //! //! Osaka introduces no new `newPayload`: its own document adds only //! `engine_getPayloadV5` and `engine_getBlobsV2`/`V3`, and `engine_newPayloadV5` -//! belongs to Amsterdam. So the Osaka-current call for a payload is Prague's +//! belongs to Amsterdam, where gloas's envelope import uses it. So the Osaka-current call for a payload is Prague's //! `engine_newPayloadV4`, and the Osaka-current fork choice notification is //! Cancun's `engine_forkchoiceUpdatedV3`. //! @@ -31,7 +31,7 @@ pub mod types; pub use auth::JwtSecret; pub use client::EngineClient; pub use error::EngineError; -pub use types::{ForkchoiceStateV1, PayloadStatusV1, PayloadStatusValue}; +pub use types::{CustodyColumns, ForkchoiceStateV1, PayloadStatusV1, PayloadStatusValue}; /// The methods this client will call, sent in the `engine_exchangeCapabilities` /// handshake. @@ -40,7 +40,9 @@ pub use types::{ForkchoiceStateV1, PayloadStatusV1, PayloadStatusValue}; /// requires `engine_exchangeCapabilities` itself not to appear. pub const ETHLAMBDA_ENGINE_CAPABILITIES: &[&str] = &[ "engine_newPayloadV4", + "engine_newPayloadV5", "engine_forkchoiceUpdatedV3", + "engine_forkchoiceUpdatedV4", "engine_getPayloadV5", "engine_getClientVersionV1", ]; diff --git a/crates/net/engine/src/types.rs b/crates/net/engine/src/types.rs index dee89790f..6e142812e 100644 --- a/crates/net/engine/src/types.rs +++ b/crates/net/engine/src/types.rs @@ -9,7 +9,9 @@ //! number crossing this boundary is `QUANTITY`, `0x`-prefixed with leading zeros //! stripped: `0x0` for zero, never `0x00` and never `0x`. -use ethlambda_types::beacon::containers::deneb; +use ethlambda_types::beacon::containers::capella::Withdrawal; +use ethlambda_types::beacon::containers::{deneb, gloas}; +use ethlambda_types::beacon::preset; use ethlambda_types::beacon::primitives::{ExecutionBlockHash, Uint256}; use ethlambda_types::beacon::serde_helpers::HexPrefixed; use serde::{Deserialize, Serialize, Serializer}; @@ -120,31 +122,117 @@ impl Serialize for ExecutionPayloadV3<'_> { .collect(); out.serialize_field("transactions", &transactions)?; - let withdrawals: Vec = payload - .withdrawals + let withdrawals: Vec = + payload.withdrawals.iter().map(withdrawal_json).collect(); + out.serialize_field("withdrawals", &withdrawals)?; + + out.serialize_field("blobGasUsed", &quantity(payload.blob_gas_used))?; + out.serialize_field("excessBlobGas", &quantity(payload.excess_blob_gas))?; + out.end() + } +} + +/// `ExecutionPayloadV4` (Amsterdam), borrowed from a decoded envelope. +/// +/// `ExecutionPayloadV3`'s fields followed by `blockAccessList` (the RLP bytes +/// the payload already carries, as `DATA`) and `slotNumber`. Gloas's payload +/// has progressive `transactions` and `withdrawals`, a different type from +/// deneb's, so it needs its own serializer; the wire shape of those two is +/// unchanged. +pub struct ExecutionPayloadV4<'a>(pub &'a gloas::ExecutionPayload); + +impl Serialize for ExecutionPayloadV4<'_> { + fn serialize(&self, serializer: S) -> Result { + use serde::ser::SerializeStruct as _; + + let payload = self.0; + let mut out = serializer.serialize_struct("ExecutionPayloadV4", 19)?; + out.serialize_field("parentHash", &data(&payload.parent_hash.0))?; + out.serialize_field("feeRecipient", &data(&payload.fee_recipient.0))?; + out.serialize_field("stateRoot", &data(&payload.state_root.0))?; + out.serialize_field("receiptsRoot", &data(&payload.receipts_root.0))?; + out.serialize_field("logsBloom", &data(payload.logs_bloom.as_ref()))?; + out.serialize_field("prevRandao", &data(&payload.prev_randao.0))?; + out.serialize_field("blockNumber", &quantity(payload.block_number))?; + out.serialize_field("gasLimit", &quantity(payload.gas_limit))?; + out.serialize_field("gasUsed", &quantity(payload.gas_used))?; + out.serialize_field("timestamp", &quantity(payload.timestamp))?; + out.serialize_field("extraData", &data(payload.extra_data.as_ref()))?; + out.serialize_field("baseFeePerGas", &uint256(&payload.base_fee_per_gas))?; + out.serialize_field("blockHash", &data(&payload.block_hash.0))?; + + let transactions: Vec = payload + .transactions .iter() - .map(|withdrawal| { - serde_json::json!({ - "index": quantity(withdrawal.index), - "validatorIndex": quantity(withdrawal.validator_index), - "address": data(&withdrawal.address.0), - "amount": quantity(withdrawal.amount), - }) - }) + .map(|transaction| data(transaction.as_ref())) .collect(); + out.serialize_field("transactions", &transactions)?; + + let withdrawals: Vec = + payload.withdrawals.iter().map(withdrawal_json).collect(); out.serialize_field("withdrawals", &withdrawals)?; out.serialize_field("blobGasUsed", &quantity(payload.blob_gas_used))?; out.serialize_field("excessBlobGas", &quantity(payload.excess_blob_gas))?; + out.serialize_field("blockAccessList", &data(payload.block_access_list.as_ref()))?; + out.serialize_field("slotNumber", &quantity(payload.slot_number))?; out.end() } } +/// `WithdrawalV1`. +fn withdrawal_json(withdrawal: &Withdrawal) -> serde_json::Value { + serde_json::json!({ + "index": quantity(withdrawal.index), + "validatorIndex": quantity(withdrawal.validator_index), + "address": data(&withdrawal.address.0), + "amount": quantity(withdrawal.amount), + }) +} + /// Encodes a `DATA`: `0x`-prefixed, every byte rendered, no stripping. pub fn data(bytes: &[u8]) -> String { HexPrefixed(bytes).to_string() } +/// The `custodyColumns` parameter of `engine_forkchoiceUpdatedV4`: a bitarray +/// of `CELLS_PER_EXT_BLOB` bits marking the columns the consensus client +/// custodies. +/// +/// Little-endian bit order, as an SSZ `Bitvector` has it: column `i` is bit +/// `i % 8` of byte `i / 8`. ethrex parses it the same way (`u128::from_le_bytes`). +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct CustodyColumns([u8; Self::BYTES]); + +impl CustodyColumns { + /// Wire length of the bitmap, which the specification fixes at 16 bytes. + pub const BYTES: usize = 16; + + /// Builds the bitmap from column indices, or `None` when any index is not + /// below `NUMBER_OF_COLUMNS`. Duplicates are harmless. + pub fn from_indices(indices: impl IntoIterator) -> Option { + let mut bits = [0u8; Self::BYTES]; + for index in indices { + if index >= preset::NUMBER_OF_COLUMNS as u64 { + return None; + } + bits[(index / 8) as usize] |= 1 << (index % 8); + } + Some(Self(bits)) + } + + /// The `DATA` encoding: `0x` and 32 hex digits. + pub fn to_data(self) -> String { + data(&self.0) + } +} + +impl Serialize for CustodyColumns { + fn serialize(&self, serializer: S) -> Result { + serializer.serialize_str(&self.to_data()) + } +} + /// Encodes a `QUANTITY`: `0x`-prefixed, leading zeros stripped, `0x0` for zero. pub fn quantity(value: u64) -> String { format!("0x{value:x}") @@ -170,11 +258,31 @@ pub fn uint256(value: &Uint256) -> String { #[cfg(test)] mod tests { use ethlambda_types::beacon::containers::bellatrix; - use ethlambda_types::beacon::preset; use ethlambda_types::beacon::primitives::{Bytes32, ExecutionAddress, U256}; use super::*; + #[test] + fn custody_columns_use_little_endian_bit_order() { + // Column 0 is bit 0 of byte 0; column 9 is bit 1 of byte 1; column 127 + // is bit 7 of byte 15. + let columns = CustodyColumns::from_indices([0, 9, 127]).unwrap(); + assert_eq!(columns.to_data(), "0x01020000000000000000000000000080"); + } + + #[test] + fn custody_columns_reject_an_out_of_range_index() { + let bound = preset::NUMBER_OF_COLUMNS as u64; + assert!(CustodyColumns::from_indices([bound - 1]).is_some()); + assert!(CustodyColumns::from_indices([0, bound]).is_none()); + } + + #[test] + fn empty_custody_columns_encode_as_sixteen_zero_bytes() { + let columns = CustodyColumns::from_indices([]).unwrap(); + assert_eq!(columns.to_data(), format!("0x{}", "00".repeat(16))); + } + /// An otherwise-zero deneb execution payload whose `block_hash` is /// `block_hash`. /// @@ -294,6 +402,77 @@ mod tests { ); } + #[test] + fn a_gloas_payload_serializes_to_the_amsterdam_shape() { + let mut payload = gloas::ExecutionPayload { + parent_hash: ExecutionBlockHash::repeat_byte(1), + fee_recipient: ExecutionAddress::repeat_byte(2), + state_root: Bytes32::repeat_byte(3), + receipts_root: Bytes32::repeat_byte(4), + logs_bloom: bellatrix::LogsBloom::try_from(vec![0u8; preset::BYTES_PER_LOGS_BLOOM]) + .expect("built at exactly BYTES_PER_LOGS_BLOOM"), + prev_randao: Bytes32::repeat_byte(5), + block_number: 16, + gas_limit: 30_000_000, + gas_used: 0, + timestamp: 255, + extra_data: Default::default(), + base_fee_per_gas: Uint256::from_u128(7), + block_hash: ExecutionBlockHash::repeat_byte(6), + transactions: Default::default(), + withdrawals: Default::default(), + blob_gas_used: 131072, + excess_blob_gas: 0, + block_access_list: Default::default(), + slot_number: 4096, + }; + let transaction: gloas::Transaction = vec![0x02u8, 0xab].try_into().expect("fits"); + payload.transactions.push(transaction); + payload.withdrawals.push(Withdrawal { + index: 9, + validator_index: 10, + address: ExecutionAddress::repeat_byte(8), + amount: 32, + }); + for byte in [0xc1u8, 0x80] { + payload.block_access_list.push(byte); + } + + let json = serde_json::to_value(ExecutionPayloadV4(&payload)).expect("serializes"); + + let expected = serde_json::json!({ + "parentHash": format!("0x{}", "01".repeat(32)), + "feeRecipient": format!("0x{}", "02".repeat(20)), + "stateRoot": format!("0x{}", "03".repeat(32)), + "receiptsRoot": format!("0x{}", "04".repeat(32)), + "logsBloom": format!("0x{}", "00".repeat(256)), + "prevRandao": format!("0x{}", "05".repeat(32)), + "blockNumber": "0x10", + "gasLimit": "0x1c9c380", + "gasUsed": "0x0", + "timestamp": "0xff", + "extraData": "0x", + "baseFeePerGas": "0x7", + "blockHash": format!("0x{}", "06".repeat(32)), + "transactions": ["0x02ab"], + "withdrawals": [{ + "index": "0x9", + "validatorIndex": "0xa", + "address": format!("0x{}", "08".repeat(20)), + "amount": "0x20", + }], + "blobGasUsed": "0x20000", + "excessBlobGas": "0x0", + "blockAccessList": "0xc180", + "slotNumber": "0x1000", + }); + assert_eq!(json, expected); + + // The spec's order, with the two Amsterdam fields last. + let text = serde_json::to_string(&ExecutionPayloadV4(&payload)).expect("serializes"); + assert!(text.ends_with(r#""blockAccessList":"0xc180","slotNumber":"0x1000"}"#)); + } + #[test] fn quantities_strip_leading_zeros_and_zero_is_a_single_digit() { assert_eq!(quantity(0), "0x0"); diff --git a/crates/net/engine/tests/wire_smoke.rs b/crates/net/engine/tests/wire_smoke.rs index 78ed116bf..72d943efd 100644 --- a/crates/net/engine/tests/wire_smoke.rs +++ b/crates/net/engine/tests/wire_smoke.rs @@ -9,7 +9,7 @@ use std::sync::Arc; -use ethlambda_engine::types::{ForkchoiceStateV1, PayloadStatusValue}; +use ethlambda_engine::types::{CustodyColumns, ForkchoiceStateV1, PayloadStatusValue}; use ethlambda_engine::{EngineClient, EngineError, JwtSecret}; use ethlambda_types::beacon::primitives::ExecutionBlockHash; use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; @@ -134,3 +134,90 @@ async fn an_rpc_error_envelope_surfaces_typed_and_is_not_retried() { other => panic!("expected an Rpc error, got {other:?}"), } } + +#[tokio::test] +async fn new_payload_v5_sends_four_params_under_its_own_method() { + use ethlambda_types::beacon::containers::{bellatrix, gloas}; + use ethlambda_types::beacon::preset; + use ethlambda_types::beacon::primitives::{Bytes32, ExecutionAddress, Root, Uint256}; + + let (endpoint, seen) = serve_once( + r#"{"jsonrpc":"2.0","id":1,"result":{"status":"VALID","latestValidHash":"0x0505050505050505050505050505050505050505050505050505050505050505","validationError":null}}"#, + ) + .await; + + let payload = gloas::ExecutionPayload { + parent_hash: ExecutionBlockHash::ZERO, + fee_recipient: ExecutionAddress::ZERO, + state_root: Bytes32::ZERO, + receipts_root: Bytes32::ZERO, + logs_bloom: bellatrix::LogsBloom::try_from(vec![0u8; preset::BYTES_PER_LOGS_BLOOM]) + .expect("built at exactly BYTES_PER_LOGS_BLOOM"), + prev_randao: Bytes32::ZERO, + block_number: 1, + gas_limit: 1, + gas_used: 0, + timestamp: 1, + extra_data: Default::default(), + base_fee_per_gas: Uint256::ZERO, + block_hash: ExecutionBlockHash::repeat_byte(5), + transactions: Default::default(), + withdrawals: Default::default(), + blob_gas_used: 0, + excess_blob_gas: 0, + // 0xc0 is the RLP of an empty list, the smallest BAL an execution + // client accepts as well-formed. + block_access_list: vec![0xc0u8].try_into().expect("fits"), + slot_number: 77, + }; + let status = client(endpoint) + .new_payload_v5( + &payload, + &[Bytes32::repeat_byte(1)], + Root::repeat_byte(2), + &[vec![0x00, 0xaa]], + ) + .await + .expect("the mock answers"); + + assert_eq!(status.status, PayloadStatusValue::Valid); + + let request = seen.lock().await.clone().expect("the mock saw a request"); + let (_, body) = request.split_once("\r\n\r\n").expect("a request body"); + let body: serde_json::Value = serde_json::from_str(body).expect("a JSON body"); + assert_eq!(body["method"], "engine_newPayloadV5"); + let params = body["params"].as_array().expect("an array of params"); + assert_eq!(params.len(), 4); + assert_eq!(params[0]["slotNumber"], "0x4d"); + assert_eq!(params[0]["blockAccessList"], "0xc0"); + assert_eq!(params[1][0], format!("0x{}", "01".repeat(32))); + assert_eq!(params[2], format!("0x{}", "02".repeat(32))); + assert_eq!(params[3][0], "0x00aa"); +} + +#[tokio::test] +async fn forkchoice_updated_v4_sends_three_params_and_parses_syncing() { + let (endpoint, seen) = serve_once( + r#"{"jsonrpc":"2.0","id":1,"result":{"payloadStatus":{"status":"SYNCING","latestValidHash":null,"validationError":null},"payloadId":null}}"#, + ) + .await; + + let state = ForkchoiceStateV1 { + head_block_hash: ExecutionBlockHash::repeat_byte(1), + safe_block_hash: ExecutionBlockHash::repeat_byte(2), + finalized_block_hash: ExecutionBlockHash::repeat_byte(3), + }; + let columns = CustodyColumns::from_indices([0, 9]); + let status = client(endpoint) + .forkchoice_updated_v4(&state, columns) + .await + .expect("the mock answers"); + + assert_eq!(status.status, PayloadStatusValue::Syncing); + + let request = seen.lock().await.clone().expect("the mock saw a request"); + assert!(request.contains("engine_forkchoiceUpdatedV4")); + assert!(request.contains("headBlockHash")); + // The attributes are null and the bitmap follows them. + assert!(request.contains(",null,\"0x01020000000000000000000000000000\"]")); +} diff --git a/crates/net/p2p/src/beacon/column_checks.rs b/crates/net/p2p/src/beacon/column_checks.rs index cc4163d41..91fbd1efe 100644 --- a/crates/net/p2p/src/beacon/column_checks.rs +++ b/crates/net/p2p/src/beacon/column_checks.rs @@ -9,8 +9,9 @@ //! - a gossiped sidecar reported `Queue` or `Ignore(Overloaded)`, whose //! stateful checks stopped early or never ran; //! - every `DataColumnsByRoot` and `DataColumnsByRange` answer; -//! - sidecars the chain actor had parked, handed back once their parent -//! imported (`BlockChainToP2P::check_data_column_sidecars`). +//! - sidecars the chain actor had parked, handed back once the block they +//! wait on imported (`BlockChainToP2P::check_data_column_sidecars`): a fulu +//! sidecar's parent, a gloas sidecar's own block. //! //! Each sidecar is checked on a blocking thread, bounded by //! [`P2PServer::column_check_permits`]. Unlike a gossip verdict, nothing here @@ -19,14 +20,14 @@ use ethlambda_state_transition::beacon::gossip::column::{self, ChainVerdict}; use ethlambda_state_transition::beacon::gossip::{IgnoreReason, Outcome}; -use ethlambda_types::beacon::containers::fulu::DataColumnSidecar; +use ethlambda_types::beacon::containers::DataColumnSidecar; use ethlambda_types::time::unix_now_ms; use tracing::{error, warn}; use crate::{P2PServer, metrics}; /// Run the chain checks on `sidecars`, then send the chain actor the ones that -/// passed as one batch and the ones waiting on a parent as another. +/// passed as one batch and the ones waiting on a block as another. /// /// Returns at once: the checks run on a task of their own, so the p2p actor /// goes on handling swarm events while a range batch's KZG proofs are @@ -51,7 +52,7 @@ pub(crate) fn check_and_forward(server: &P2PServer, sidecars: Vec Result<(), ActorError> { Ok(()) } + fn new_execution_payload_envelope( + &self, + _envelope: Box< + ethlambda_types::beacon::containers::gloas::SignedExecutionPayloadEnvelope, + >, + _arrival: BlockArrival, + ) -> Result<(), ActorError> { + Ok(()) + } + fn new_payload_attestation_message( + &self, + _message: ethlambda_types::beacon::containers::gloas::PayloadAttestationMessage, + _arrival: BlockArrival, + ) -> Result<(), ActorError> { + Ok(()) + } } /// A sidecar the checks cannot judge yet goes back to the chain actor to @@ -188,15 +205,56 @@ mod tests { let mut malformed = valid_shaped_sidecar(5, 1); malformed.kzg_commitments = Default::default(); - check_and_forward(&server, vec![orphan.clone(), malformed]); + check_and_forward( + &server, + vec![ + DataColumnSidecar::Fulu(orphan.clone()), + DataColumnSidecar::Fulu(malformed), + ], + ); assert_eq!( received.recv().await, - Some(Forwarded::AwaitingParent(vec![orphan])) + Some(Forwarded::AwaitingParent(vec![DataColumnSidecar::Fulu( + orphan + )])) ); // The task sends at most one message per kind and has now finished, // so the channel closes with nothing else in it. drop(server); assert_eq!(received.recv().await, None); } + + /// A gloas sidecar is judged against its own block, not a parent: one + /// whose block this node has not seen goes back to the chain actor to be + /// parked, and one from a slot that has not started is dropped, so neither + /// reaches the batch the chain actor stores unchecked. + #[tokio::test] + async fn a_gloas_sidecar_for_an_unknown_block_is_parked_and_a_future_one_dropped() { + let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; + let (sender, mut received) = mpsc::unbounded_channel(); + server.blockchain = Some(Arc::new(RecordingChain(sender))); + + let unknown_block = DataColumnSidecar::Gloas(gloas::DataColumnSidecar { + index: 0, + slot: 5, + beacon_block_root: Root::repeat_byte(9), + ..Default::default() + }); + let future = DataColumnSidecar::Gloas(gloas::DataColumnSidecar { + index: 1, + slot: u64::MAX / 2, + beacon_block_root: Root::repeat_byte(9), + ..Default::default() + }); + + check_and_forward(&server, vec![unknown_block.clone(), future]); + + assert_eq!( + received.recv().await, + Some(Forwarded::AwaitingParent(vec![unknown_block])) + ); + drop(server); + assert_eq!(received.recv().await, None); + } } diff --git a/crates/net/p2p/src/beacon/decode.rs b/crates/net/p2p/src/beacon/decode.rs index 84da94623..b4fac1c01 100644 --- a/crates/net/p2p/src/beacon/decode.rs +++ b/crates/net/p2p/src/beacon/decode.rs @@ -6,9 +6,9 @@ //! other four topics carry containers whose shape has not changed since the //! fork that introduced them, so they decode with no fork lookup at all. //! `data_column_sidecar_{subnet_id}`, the one family among the subscribed -//! topics rather than a fixed name, decodes with no lookup either, for a -//! different reason: fulu is the only fork that defines the container, so -//! there is no ladder to begin with (see [`decode_data_column_sidecar`]). +//! topics rather than a fixed name, decodes by the fork its topic's digest +//! names, since fulu and gloas define different containers under the one name +//! (see [`decode_data_column_sidecar`]). //! `beacon_attestation_{subnet_id}` is the exception to reading the fork off //! the payload: electra moved its slot, so its fork comes from the topic's //! digest instead (see [`decode_attestation`]). @@ -22,16 +22,16 @@ //! | `voluntary_exit`, `proposer_slashing` | No | //! | `bls_to_execution_change` | No, capella onward | //! | `sync_committee_contribution_and_proof` | No, altair onward | -//! | `data_column_sidecar_{subnet_id}` | No, fulu only | +//! | `data_column_sidecar_{subnet_id}` | Yes, at gloas, by topic digest | +//! | `execution_payload`, `payload_attestation_message` | No, gloas onward | use ethlambda_types::beacon::config::Config; use ethlambda_types::beacon::containers::{ - SignedBeaconBlock, altair, capella, electra, fulu, phase0, shared, + DataColumnSidecar, SignedBeaconBlock, altair, capella, electra, gloas, phase0, shared, }; use ethlambda_types::beacon::fork::ForkName; use ethlambda_types::beacon::preset; use ethlambda_types::beacon::primitives::Slot; -use ethlambda_types::time::unix_now_ms; use libssz::SszDecode as _; use super::topics; @@ -47,11 +47,12 @@ use super::topics; pub use ethlambda_types::beacon::containers::SignedAggregateAndProof; /// Slashing evidence, in whichever shape the slot's fork gives it. Electra -/// widened `IndexedAttestation`'s committee bound. +/// widened `IndexedAttestation`'s committee bound; gloas made it progressive. #[derive(Debug, Clone, PartialEq)] pub enum AttesterSlashing { Phase0(phase0::AttesterSlashing), Electra(electra::AttesterSlashing), + Gloas(gloas::AttesterSlashing), } /// A decoded gossip payload, one variant per subscribed topic. @@ -183,33 +184,14 @@ pub fn fork_at_slot(config: &Config, slot: Slot) -> ForkName { config.fork_at_epoch(slot / preset::SLOTS_PER_EPOCH) } -/// The fork this node's own wall clock says is active right now. -/// -/// [`decode_data_column_sidecar`] has no slot to look a fork up by the way -/// [`fork_at_slot`] does: only fulu's container is modeled, so nothing reads -/// a slot out of the bytes first. A caller on the gossip path uses this to -/// approximate, once that decode has already failed, whether the failure is -/// this build's own gap rather than the sender's fault. An approximation -/// only, since gossip topic subscriptions are frozen at startup and this -/// reads the live clock instead of whatever fork the topic was actually -/// built for; see [`decode_data_column_sidecar`]'s own doc and its caller in -/// `gossipsub::handler::triage_data_column`. -pub(crate) fn current_fork(config: &Config) -> ForkName { - let slot = unix_now_ms() - .saturating_sub(config.genesis_time_ms()) - .checked_div(config.slot_duration_ms) - .unwrap_or(0); - fork_at_slot(config, slot) -} - /// Decode a `beacon_block` payload, at the fork its own slot names. /// /// Separate from [`decode_gossip`] because the two topics worth logging in /// detail are dispatched by name, and a handler that already knows it is /// holding a block should not have to unwrap a [`BeaconGossip`] to find one. /// -/// A block whose slot is at a fork this node does not follow (gloas) and that -/// fails to decode is `UnsupportedFork`, not `Ssz`: the container this build +/// A block whose slot is at a fork this node does not follow (none today) and +/// that fails to decode is `UnsupportedFork`, not `Ssz`: the container this build /// models for that fork is the one at the pinned specification release, and /// an honest peer's block may follow a later one. Bytes too short to name a /// slot, and a block at a followed fork, stay `Ssz` or `Truncated`. @@ -224,22 +206,35 @@ pub fn decode_block(config: &Config, bytes: &[u8]) -> Result Result { - fulu::DataColumnSidecar::from_ssz_bytes(bytes).map_err(|_| DecodeError::Ssz) +/// Fulu and gloas define different containers under this name (gloas drops the +/// header and inclusion proof for a `slot` and a block root; see +/// `containers::gloas::DataColumnSidecar`'s doc), and the bytes alone do not say +/// which one they are. The fork therefore comes from context: a gossip topic's +/// digest, or a req/resp chunk's context bytes. A fork with no data columns +/// fails like a malformed payload. +pub fn decode_data_column_sidecar( + fork: ForkName, + bytes: &[u8], +) -> Result { + DataColumnSidecar::from_ssz(fork, bytes).map_err(|_| DecodeError::Ssz) +} + +/// Decode an `execution_payload` payload. The topic exists from gloas on, so +/// the container needs no fork lookup. +pub fn decode_execution_payload_envelope( + bytes: &[u8], +) -> Result { + gloas::SignedExecutionPayloadEnvelope::from_ssz_bytes(bytes).map_err(|_| DecodeError::Ssz) +} + +/// Decode a `payload_attestation_message` payload. Gloas on, like +/// [`decode_execution_payload_envelope`]. +pub fn decode_payload_attestation_message( + bytes: &[u8], +) -> Result { + gloas::PayloadAttestationMessage::from_ssz_bytes(bytes).map_err(|_| DecodeError::Ssz) } /// Decode a `beacon_aggregate_and_proof` payload, at the fork its slot names. @@ -253,17 +248,16 @@ pub fn decode_aggregate_and_proof( electra::SignedAggregateAndProof::from_ssz_bytes(bytes) .map(SignedAggregateAndProof::Electra) } + // Electra's bytes, but a progressive `Attestation` (EIP-7688), so the + // root the aggregator signs differs; `data.index` is the payload flag. + ForkName::Gloas => gloas::SignedAggregateAndProof::from_ssz_bytes(bytes) + .map(SignedAggregateAndProof::Gloas), ForkName::Phase0 | ForkName::Altair | ForkName::Bellatrix | ForkName::Capella | ForkName::Deneb => phase0::SignedAggregateAndProof::from_ssz_bytes(bytes) .map(SignedAggregateAndProof::Phase0), - // Gloas's own `AggregateAndProof` has no modeled variant here yet - // (its attestation is progressive-list-shaped, EIP-7549 continued), - // so this is refused rather than mis-decoded as electra's. - // `UnsupportedFork`, not `Ssz`: the sender did nothing wrong. - ForkName::Gloas => return Err(DecodeError::UnsupportedFork), ForkName::Lean => { unreachable!("fork_at_slot never returns Lean: it is absent from ForkName::ALL") } @@ -307,7 +301,9 @@ impl Attestation { /// subscribed digest was computed at. pub fn decode_attestation(fork: ForkName, bytes: &[u8]) -> Result { match fork { - ForkName::Electra | ForkName::Fulu => { + // Gloas's `SingleAttestation` has electra's bytes; its `data.index` + // carries the payload flag, which the gossip rules read. + ForkName::Electra | ForkName::Fulu | ForkName::Gloas => { electra::SingleAttestation::from_ssz_bytes(bytes).map(Attestation::Electra) } ForkName::Phase0 @@ -315,11 +311,6 @@ pub fn decode_attestation(fork: ForkName, bytes: &[u8]) -> Result phase0::Attestation::from_ssz_bytes(bytes).map(Attestation::Phase0), - // Gloas's `SingleAttestation` has the same bytes as electra's, but its - // `data.index` carries the payload-availability signal instead of - // being zero, so electra's gossip rules would reject an honest vote. - // `UnsupportedFork`, not `Ssz`: the sender did nothing wrong. - ForkName::Gloas => return Err(DecodeError::UnsupportedFork), ForkName::Lean => { unreachable!("a beacon topic's fork is never Lean: it is absent from ForkName::ALL") } @@ -349,6 +340,9 @@ pub fn decode_gossip( ForkName::Electra | ForkName::Fulu => { electra::AttesterSlashing::from_ssz_bytes(bytes).map(AttesterSlashing::Electra) } + ForkName::Gloas => { + gloas::AttesterSlashing::from_ssz_bytes(bytes).map(AttesterSlashing::Gloas) + } ForkName::Phase0 | ForkName::Altair | ForkName::Bellatrix @@ -356,9 +350,6 @@ pub fn decode_gossip( | ForkName::Deneb => { phase0::AttesterSlashing::from_ssz_bytes(bytes).map(AttesterSlashing::Phase0) } - // Gloas's own `AttesterSlashing` has no modeled variant here - // yet; see `decode_aggregate_and_proof`'s matching arm. - ForkName::Gloas => return Err(DecodeError::UnsupportedFork), ForkName::Lean => { unreachable!("fork_at_slot never returns Lean: it is absent from ForkName::ALL") } @@ -390,6 +381,7 @@ pub fn decode_gossip( #[cfg(test)] mod tests { use super::*; + use ethlambda_types::beacon::containers::fulu; use ethlambda_types::beacon::primitives::{BlsSignature, Bytes32, Root}; use libssz::SszEncode as _; @@ -473,17 +465,14 @@ mod tests { } #[test] - fn an_undecodable_block_is_unsupported_only_at_a_fork_the_node_does_not_follow() { + fn an_undecodable_block_is_an_ssz_failure_at_gloas_as_at_any_followed_fork() { let mut config = Config::mainnet(); config.gloas_fork_epoch = config.fulu_fork_epoch + 1; // Phase0-shaped bytes are not a valid block at either fork. let at_fulu = phase0_block(slot_of(config.fulu_fork_epoch)).to_ssz(); let at_gloas = phase0_block(slot_of(config.gloas_fork_epoch)).to_ssz(); assert_eq!(decode_block(&config, &at_fulu), Err(DecodeError::Ssz)); - assert_eq!( - decode_block(&config, &at_gloas), - Err(DecodeError::UnsupportedFork) - ); + assert_eq!(decode_block(&config, &at_gloas), Err(DecodeError::Ssz)); assert_eq!( decode_block(&config, &[0xff; 3]), Err(DecodeError::Truncated) @@ -540,8 +529,28 @@ mod tests { .expect("exactly the required depth"), }; let bytes = sidecar.to_ssz(); - assert_eq!(decode_data_column_sidecar(&bytes).unwrap().index, 3); - assert!(decode_data_column_sidecar(&bytes[..bytes.len() - 1]).is_err()); + assert_eq!( + decode_data_column_sidecar(ForkName::Fulu, &bytes) + .unwrap() + .index(), + 3 + ); + assert!(decode_data_column_sidecar(ForkName::Fulu, &bytes[..bytes.len() - 1]).is_err()); + } + + #[test] + fn a_gloas_sidecar_decodes_by_the_topics_fork_and_not_as_fulu() { + let sidecar = ethlambda_types::beacon::containers::gloas::DataColumnSidecar { + index: 5, + slot: 77, + ..Default::default() + }; + let bytes = sidecar.to_ssz(); + let decoded = decode_data_column_sidecar(ForkName::Gloas, &bytes).unwrap(); + assert_eq!((decoded.index(), decoded.slot()), (5, 77)); + assert_eq!(decoded, DataColumnSidecar::Gloas(sidecar)); + assert!(decode_data_column_sidecar(ForkName::Fulu, &bytes).is_err()); + assert!(decode_data_column_sidecar(ForkName::Electra, &bytes).is_err()); } /// One attester's vote at `slot`, in the shape electra puts on a subnet. @@ -607,14 +616,13 @@ mod tests { } #[test] - fn a_subnet_attestation_at_gloas_is_unsupported_rather_than_malformed() { - // Gloas reads `data.index` as the payload-availability signal, so - // decoding these bytes with electra's rules would reject honest - // votes. The bytes are valid, so the refusal must not be `Ssz`. - let single = single_attestation(slot_of(10)).to_ssz(); + fn a_subnet_attestation_at_gloas_decodes_as_electras_shape() { + // Gloas keeps electra's bytes and reads `data.index` as the payload + // flag; the gossip rules, not the decoder, interpret it. + let single = single_attestation(slot_of(10)); assert_eq!( - decode_attestation(ForkName::Gloas, &single), - Err(DecodeError::UnsupportedFork) + decode_attestation(ForkName::Gloas, &single.to_ssz()), + Ok(Attestation::Electra(single)) ); } @@ -626,6 +634,34 @@ mod tests { } } + #[test] + fn a_payload_attestation_message_round_trips() { + let message = gloas::PayloadAttestationMessage { + validator_index: 17, + data: gloas::PayloadAttestationData { + beacon_block_root: Root::repeat_byte(3), + slot: slot_of(10), + payload_present: true, + blob_data_available: false, + }, + signature: Default::default(), + }; + let bytes = message.to_ssz(); + assert_eq!(decode_payload_attestation_message(&bytes), Ok(message)); + for length in 0..bytes.len() { + assert!(decode_payload_attestation_message(&bytes[..length]).is_err()); + } + } + + #[test] + fn an_execution_payload_envelope_round_trips() { + let envelope = crate::test_support::envelope(5, 9); + let bytes = envelope.to_ssz(); + assert_eq!(decode_execution_payload_envelope(&bytes), Ok(envelope)); + assert!(decode_execution_payload_envelope(&bytes[..bytes.len() - 1]).is_err()); + assert!(decode_execution_payload_envelope(&[0xff; 3]).is_err()); + } + #[test] fn an_unsubscribed_topic_is_refused() { let config = Config::mainnet(); diff --git a/crates/net/p2p/src/beacon/encoding.rs b/crates/net/p2p/src/beacon/encoding.rs index d5791b445..ea7df9bee 100644 --- a/crates/net/p2p/src/beacon/encoding.rs +++ b/crates/net/p2p/src/beacon/encoding.rs @@ -34,8 +34,10 @@ use std::io; use ethlambda_types::beacon::config::Config; +use ethlambda_types::beacon::containers::DataColumnSidecar; use ethlambda_types::beacon::containers::SignedBeaconBlock; -use ethlambda_types::beacon::containers::fulu::DataColumnSidecar; +use ethlambda_types::beacon::containers::gloas::SignedExecutionPayloadEnvelope; +use ethlambda_types::beacon::fork::ForkName; use ethlambda_types::beacon::fork_digest::compute_fork_digest; use ethlambda_types::beacon::preset; use ethlambda_types::beacon::primitives::Root; @@ -44,6 +46,7 @@ use libssz::{SszDecode, SszEncode}; use tracing::warn; use super::decode; +use super::fork_schedule::ForkSchedule; use super::messages::{ BeaconBlocksByRangeRequest, BeaconMetaData, BeaconStatus, MetaDataV1, MetaDataV2, MetaDataV3, StatusV1, StatusV2, @@ -253,16 +256,15 @@ where let encoded = sidecar.to_ssz(); if encoded.len() > MAX_PAYLOAD_SIZE - 1024 { warn!( - index = sidecar.index, + index = sidecar.index(), size = encoded.len(), "Skipping oversized data column sidecar in response" ); continue; } - // The sidecar's own epoch, taken from the header it carries rather - // than the one this node runs on, so a backfill labels each chunk - // with its own fork. - let epoch = sidecar.signed_block_header.message.slot / preset::SLOTS_PER_EPOCH; + // The sidecar's own epoch rather than the one this node runs on, so a + // backfill labels each chunk with its own fork. + let epoch = sidecar.slot() / preset::SLOTS_PER_EPOCH; let digest = compute_fork_digest(config, genesis_validators_root, epoch); write_success_chunk(io, label, &digest, encoded).await?; } @@ -272,14 +274,16 @@ where /// Read a data column sidecar response: one `DataColumnSidecar` per chunk, /// until the peer closes. /// -/// The counterpart of [`decode_blocks_response`]. Only fulu defines this -/// container, so there is no fork ladder to select a decoder from the way a -/// block chunk's slot selects one; [`super::decode::decode_data_column_sidecar`] -/// decodes unconditionally. The context bytes are still checked against the -/// digest the sidecar's own slot implies, for the same reason a block chunk's -/// are: it catches a peer whose `genesis_validators_root` or fork schedule -/// differs from ours, which a signature failure would otherwise be the only -/// way to notice. +/// The counterpart of [`decode_blocks_response`]. A gloas sidecar carries no +/// header to read a slot from ahead of decoding, so the fork comes from the +/// chunk's context bytes instead ([`ForkSchedule::fork_for_digest`]), and +/// [`super::decode::decode_data_column_sidecar`] decodes by it. A digest no +/// scheduled fork uses ends the stream like any other mismatch. The context +/// bytes are then checked against the digest the sidecar's own slot implies, +/// for the same reason a block chunk's are: it catches a peer whose +/// `genesis_validators_root` or fork schedule differs from ours, which a +/// signature failure would otherwise be the only way to notice. It also +/// catches a sidecar of one fork's shape sent under another's context. /// /// A mismatch ends the stream rather than skipping the chunk, matching /// [`decode_blocks_response`]: a peer that disagrees about a historical digest @@ -300,16 +304,25 @@ where // column protocol; see `protocols::max_request_data_column_sidecars`. max_chunks: protocols::max_request_data_column_sidecars() as usize, }; + let schedule = ForkSchedule::new(config, genesis_validators_root); read_chunked_response(io, protocol_label, limits, |context, payload| { - let sidecar = decode::decode_data_column_sidecar(payload) + let Some(fork) = <[u8; 4]>::try_from(context) + .ok() + .and_then(|digest| schedule.fork_for_digest(digest)) + else { + return Err(invalid(format!( + "data column sidecar chunk context {} is no scheduled fork digest", + hex::encode(context), + ))); + }; + let sidecar = decode::decode_data_column_sidecar(fork, payload) .map_err(|err| invalid(format!("data column sidecar chunk: {err}")))?; - let slot = sidecar.signed_block_header.message.slot; - let epoch = slot / preset::SLOTS_PER_EPOCH; - let expected = compute_fork_digest(config, genesis_validators_root, epoch); + let slot = sidecar.slot(); + let expected = schedule.digest_at(slot / preset::SLOTS_PER_EPOCH); if context != expected { warn!( slot, - index = sidecar.index, + index = sidecar.index(), peer_context = %hex::encode(context), our_context = %hex::encode(expected), "Data column sidecar chunk names another fork digest" @@ -325,3 +338,157 @@ where }) .await } + +/// Write an envelope response: one result code, one `ForkDigest` and one +/// payload per envelope. +/// +/// The counterpart of [`write_blocks_response`]. The spec keys the context +/// epoch off the block the envelope fulfills; an envelope carries no slot of +/// its own, but `verify_execution_payload_envelope` pins +/// `payload.slot_number` to that block's slot, so a stored envelope's +/// `slot_number` is the block's slot. +pub async fn write_execution_payload_envelopes_response( + io: &mut T, + label: &'static str, + config: &Config, + genesis_validators_root: Root, + envelopes: &[SignedExecutionPayloadEnvelope], +) -> io::Result<()> +where + T: AsyncWrite + Unpin + Send, +{ + for envelope in envelopes { + let encoded = envelope.to_ssz(); + let slot = envelope.message.payload.slot_number; + if encoded.len() > MAX_PAYLOAD_SIZE - 1024 { + warn!( + slot, + size = encoded.len(), + "Skipping oversized execution payload envelope in response" + ); + continue; + } + let digest = compute_fork_digest( + config, + genesis_validators_root, + slot / preset::SLOTS_PER_EPOCH, + ); + write_success_chunk(io, label, &digest, encoded).await?; + } + Ok(()) +} + +/// Whether `fork` has execution payload envelopes: gloas and every beacon fork +/// after it, which the schedule's digests can name. Lean is not on the beacon +/// timeline, though it sorts after every beacon fork. +fn carries_envelopes(fork: ForkName) -> bool { + fork >= ForkName::Gloas && fork != ForkName::Lean +} + +/// Read an envelope response: one `SignedExecutionPayloadEnvelope` per chunk, +/// until the peer closes. +/// +/// The counterpart of [`decode_data_column_sidecars_response`]: the fork comes +/// from the context bytes, and a digest no scheduled fork uses ends the stream. +/// Envelopes exist only from gloas on, so the digest of an earlier fork is +/// refused too ([`carries_envelopes`]). +/// The context is then checked against the digest the payload's own +/// `slot_number` implies. +pub async fn decode_execution_payload_envelopes_response( + io: &mut T, + protocol_label: &str, + config: &Config, + genesis_validators_root: Root, +) -> io::Result> +where + T: AsyncRead + Unpin + Send, +{ + let limits = ChunkLimits { + has_context: true, + max_chunks: protocols::MAX_REQUEST_PAYLOADS as usize, + }; + let schedule = ForkSchedule::new(config, genesis_validators_root); + read_chunked_response(io, protocol_label, limits, |context, payload| { + let fork = <[u8; 4]>::try_from(context) + .ok() + .and_then(|digest| schedule.fork_for_digest(digest)); + if !fork.is_some_and(carries_envelopes) { + return Err(invalid(format!( + "execution payload envelope chunk context {} is not a gloas-or-later fork digest", + hex::encode(context), + ))); + } + let envelope = SignedExecutionPayloadEnvelope::from_ssz_bytes(payload) + .map_err(|err| invalid(format!("execution payload envelope chunk: {err:?}")))?; + let slot = envelope.message.payload.slot_number; + let expected = schedule.digest_at(slot / preset::SLOTS_PER_EPOCH); + if context != expected { + warn!( + slot, + peer_context = %hex::encode(context), + our_context = %hex::encode(expected), + "Execution payload envelope chunk names another fork digest" + ); + return Err(invalid(format!( + "execution payload envelope chunk context {} does not match {} for slot {}", + hex::encode(context), + hex::encode(expected), + slot, + ))); + } + Ok(envelope) + }) + .await +} + +/// Test fixtures shared by the codec and handler tests. +#[cfg(test)] +pub(crate) mod test_support { + use ethlambda_types::beacon::containers::gloas::{ + ExecutionPayload, ExecutionPayloadEnvelope, SignedExecutionPayloadEnvelope, + }; + use ethlambda_types::beacon::preset; + use ethlambda_types::beacon::primitives::{ExecutionBlockHash, Root}; + + /// An otherwise-empty envelope for the block `beacon_block_root` at `slot`, + /// revealing the payload `block_hash`. + pub(crate) fn envelope( + beacon_block_root: Root, + slot: u64, + block_hash: ExecutionBlockHash, + ) -> SignedExecutionPayloadEnvelope { + let payload = ExecutionPayload { + parent_hash: Default::default(), + fee_recipient: Default::default(), + state_root: Default::default(), + receipts_root: Default::default(), + logs_bloom: vec![0u8; preset::BYTES_PER_LOGS_BLOOM] + .try_into() + .expect("built at exactly BYTES_PER_LOGS_BLOOM"), + prev_randao: Default::default(), + block_number: 0, + gas_limit: 0, + gas_used: 0, + timestamp: 0, + extra_data: Default::default(), + base_fee_per_gas: Default::default(), + block_hash, + transactions: Default::default(), + withdrawals: Default::default(), + blob_gas_used: 0, + excess_blob_gas: 0, + block_access_list: Default::default(), + slot_number: slot, + }; + SignedExecutionPayloadEnvelope { + message: ExecutionPayloadEnvelope { + payload, + execution_requests: Default::default(), + builder_index: 0, + beacon_block_root, + parent_beacon_block_root: Root::ZERO, + }, + signature: Default::default(), + } + } +} diff --git a/crates/net/p2p/src/beacon/envelope_checks.rs b/crates/net/p2p/src/beacon/envelope_checks.rs new file mode 100644 index 000000000..5dbbd43de --- /dev/null +++ b/crates/net/p2p/src/beacon/envelope_checks.rs @@ -0,0 +1,272 @@ +//! The gossip rules for an execution payload envelope, run on every envelope a +//! peer sent in answer to a request, before the chain actor gets it. +//! +//! The specification treats an envelope obtained by any other means as if it +//! had arrived on gossip, so a fetched one passes the same rules +//! (`gossip::envelope::{cheap_checks, stateful_checks}`) minus the seen cache, +//! which only means something on a topic: the chain actor already drops an +//! envelope for a block it holds one for. Verdicts map onto the actor like so: +//! +//! - `Accept`: forwarded. +//! - `Queue`: forwarded too. The envelope's block (or its post-state) is not +//! here yet, so the actor keeps it, but only briefly and in small numbers +//! (`ENVELOPE_AWAITING_BLOCK_TTL_SLOTS` by arrival slot, at most +//! `MAX_ENVELOPES_AWAITING_BLOCK_PER_SLOT` of them per slot), and the rest +//! are dropped. A range answer is therefore only safe to deliver while the +//! blocks it belongs to are imported in order, which is why range pacing +//! waits on it: see [`check_and_forward_then`]. +//! - `Ignore` and `Reject`: dropped. +//! +//! Envelopes go to the actor in the order given. The caller sorts them by +//! slot, and the checks run concurrently but are awaited in order, so a slower +//! signature check does not reorder the batch. + +use std::num::NonZeroUsize; + +use ethlambda_network_api::BlockArrival; +use ethlambda_state_transition::beacon::gossip::Outcome; +use ethlambda_state_transition::beacon::gossip::envelope::{ + SeenEnvelopes, cheap_checks, stateful_checks, +}; +use ethlambda_storage::Store; +use ethlambda_types::beacon::containers::gloas::SignedExecutionPayloadEnvelope; +use tracing::{debug, error, warn}; + +use crate::P2PServer; + +/// Whether a fetched envelope goes to the chain actor. +pub(crate) fn forwards(outcome: &Outcome) -> bool { + match outcome { + Outcome::Accept | Outcome::Queue(_) => true, + Outcome::Ignore(_) | Outcome::Reject(_) => false, + } +} + +/// Both halves of the gossip rule for one envelope. Blocking: the second half +/// verifies a signature against the block's post-state. +fn judge(store: &Store, envelope: &SignedExecutionPayloadEnvelope) -> Outcome { + // A fresh, empty cache: a fetched envelope is not a gossip duplicate of + // anything, so the cache's one rule never fires. + let seen = SeenEnvelopes::new(NonZeroUsize::MIN); + match cheap_checks(&seen, store, envelope) { + Ok(()) => stateful_checks(store, envelope), + Err(outcome) => outcome, + } +} + +/// Run the gossip rules on `envelopes` and send the chain actor the ones that +/// pass, in order. +/// +/// Returns at once: the checks run on a task of their own, bounded by +/// [`P2PServer::column_check_permits`] like the column checks are, since a +/// range answer can carry a full request's worth of signatures. +pub(crate) fn check_and_forward( + server: &P2PServer, + envelopes: Vec, +) { + check_and_forward_then(server, envelopes, None); +} + +/// A completion callback: runs once every envelope that passed has been handed +/// to the chain actor. +pub(crate) type Done = Box; + +/// [`check_and_forward`], then `done` once the last passing envelope has been +/// sent, or at once when there is nothing to check. +/// +/// This is how range sync paces itself on a gloas batch: the next batch's +/// blocks are requested only after `done`, so the actor never sees them ahead +/// of this batch's envelopes. All of the batch's envelopes are sent by one +/// task, in the order given, with nothing from another batch between them. +pub(crate) fn check_and_forward_then( + server: &P2PServer, + envelopes: Vec, + done: Option, +) { + let run_done = |done: Option| { + if let Some(done) = done { + done(); + } + }; + if envelopes.is_empty() { + run_done(done); + return; + } + let Some(blockchain) = server.blockchain.clone() else { + run_done(done); + return; + }; + let store = server.store.clone(); + let permits = server.column_check_permits.clone(); + tokio::spawn(async move { + let mut checks = Vec::with_capacity(envelopes.len()); + for envelope in envelopes { + // The semaphore is never closed, so this only returns once a + // permit is free. + let Ok(permit) = permits.clone().acquire_owned().await else { + if let Some(done) = done { + done(); + } + return; + }; + let store = store.clone(); + checks.push(tokio::task::spawn_blocking(move || { + let _permit = permit; + let outcome = judge(&store, &envelope); + (envelope, outcome) + })); + } + + for check in checks { + let (envelope, outcome) = match check.await { + Ok(checked) => checked, + Err(err) => { + error!(%err, "An envelope check panicked; dropping the envelope"); + continue; + } + }; + if !forwards(&outcome) { + let block_root = envelope.message.beacon_block_root; + let (_, reason) = outcome.labels(); + debug!( + block_root = %ethlambda_types::ShortRoot(&block_root.0), + reason, + "Dropping a fetched execution payload envelope" + ); + continue; + } + let _ = blockchain + .new_execution_payload_envelope(Box::new(envelope), BlockArrival::now()) + .inspect_err(|err| warn!(%err, "Failed to forward a fetched envelope")); + } + if let Some(done) = done { + done(); + } + }); +} + +#[cfg(test)] +pub(crate) mod tests { + use std::sync::Arc; + + use ethlambda_network_api::{ + AggregateArrival, BlockAnnouncement, BlockSource, P2PToBlockChain, + }; + use ethlambda_state_transition::beacon::gossip::{IgnoreReason, QueueReason, RejectReason}; + use ethlambda_types::attestation::{SignedAggregatedAttestation, SignedAttestation}; + use ethlambda_types::beacon::config::Config; + use ethlambda_types::beacon::containers::{ + DataColumnSidecar, SignedAggregateAndProof, SignedBeaconBlock, + }; + use ethlambda_types::beacon::primitives::ValidatorIndex; + use ethlambda_types::primitives::H256; + use spawned_concurrency::error::ActorError; + use tokio::sync::mpsc; + + use super::*; + use crate::test_support::unconnected_beacon_server; + + /// What a [`Recorder`] saw the chain actor be sent, in order. + #[derive(Debug, PartialEq)] + pub(crate) enum Seen { + Block(u64), + Envelope(H256, u64), + } + + /// A chain actor stand-in that reports every block and envelope it gets, + /// in one stream so their relative order is observable. + pub(crate) struct Recorder(pub mpsc::UnboundedSender); + + impl P2PToBlockChain for Recorder { + fn new_block( + &self, + block: SignedBeaconBlock, + _: BlockSource, + _: BlockArrival, + _: BlockAnnouncement, + ) -> Result<(), ActorError> { + let _ = self.0.send(Seen::Block(block.slot())); + Ok(()) + } + fn new_attestation(&self, _: SignedAttestation) -> Result<(), ActorError> { + Ok(()) + } + fn new_aggregated_attestation( + &self, + _: SignedAggregatedAttestation, + ) -> Result<(), ActorError> { + Ok(()) + } + fn new_data_column_sidecars(&self, _: Vec) -> Result<(), ActorError> { + Ok(()) + } + fn data_column_sidecars_awaiting_parent( + &self, + _: Vec, + ) -> Result<(), ActorError> { + Ok(()) + } + fn new_beacon_aggregate( + &self, + _: Box, + _: Vec, + _: AggregateArrival, + ) -> Result<(), ActorError> { + Ok(()) + } + fn new_execution_payload_envelope( + &self, + envelope: Box, + _: BlockArrival, + ) -> Result<(), ActorError> { + let _ = self.0.send(Seen::Envelope( + envelope.message.beacon_block_root, + envelope.message.payload.slot_number, + )); + Ok(()) + } + fn new_payload_attestation_message( + &self, + _: ethlambda_types::beacon::containers::gloas::PayloadAttestationMessage, + _: BlockArrival, + ) -> Result<(), ActorError> { + Ok(()) + } + } + + pub(crate) fn envelope(slot: u64, byte: u8) -> SignedExecutionPayloadEnvelope { + crate::beacon::encoding::test_support::envelope(H256::repeat_byte(byte), slot, H256::ZERO) + } + + #[test] + fn only_accept_and_queue_reach_the_chain_actor() { + assert!(forwards(&Outcome::Accept)); + assert!(forwards(&Outcome::Queue(QueueReason::BlockUnknown))); + assert!(forwards(&Outcome::Queue(QueueReason::StateNotCached))); + assert!(!forwards(&Outcome::Ignore(IgnoreReason::Finalized))); + assert!(!forwards(&Outcome::Reject(RejectReason::BadSignature))); + } + + /// An envelope whose block this node has not seen is queued, so it reaches + /// the actor, and a batch keeps the order it was given. + #[tokio::test] + async fn envelopes_for_unknown_blocks_are_forwarded_in_order() { + let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; + let (sender, mut received) = mpsc::unbounded_channel(); + server.blockchain = Some(Arc::new(Recorder(sender))); + + check_and_forward( + &server, + vec![envelope(7, 1), envelope(8, 2), envelope(9, 3)], + ); + + for (slot, byte) in [(7, 1), (8, 2), (9, 3)] { + assert_eq!( + received.recv().await, + Some(Seen::Envelope(H256::repeat_byte(byte), slot)) + ); + } + drop(server); + assert_eq!(received.recv().await, None); + } +} diff --git a/crates/net/p2p/src/beacon/fork_schedule.rs b/crates/net/p2p/src/beacon/fork_schedule.rs new file mode 100644 index 000000000..252f6e9bd --- /dev/null +++ b/crates/net/p2p/src/beacon/fork_schedule.rs @@ -0,0 +1,459 @@ +//! Which fork digests the chain will use, and when a running node must move +//! between them. +//! +//! Every fork activation and every blob-schedule entry (a blob-parameter-only +//! fork, BPO) changes the fork digest, and the digest is in every gossip topic +//! name, the `Status` handshake and the ENR `eth2` entry. A node that computed +//! its digest once at startup is stranded on topics nobody publishes to the +//! moment a boundary passes, so the node follows this schedule instead. +//! +//! The transition rules are the spec's ("Transitioning the gossip" in altair's +//! `p2p-interface.md`): "In advance of the fork, a node SHOULD subscribe to the +//! post-fork variants of the topics", and the pre-fork topics are dropped "two +//! epochs after the fork". A digest is therefore held from +//! [`SUBSCRIBE_LEAD_EPOCHS`] before it activates until +//! [`UNSUBSCRIBE_LAG_EPOCHS`] after the next one does, and the node is +//! subscribed to whichever digests that window covers at the current epoch. +//! Everything here is a pure function of the epoch, so applying it is +//! idempotent and a node started inside a window lands in the same state as one +//! that crossed into it. +//! +//! Pure on purpose: the P2P actor owns the clock and the swarm; this owns only +//! the arithmetic, so it is testable without either. + +use ethlambda_types::beacon::config::Config; +use ethlambda_types::beacon::constants::FAR_FUTURE_EPOCH; +use ethlambda_types::beacon::fork::ForkName; +use ethlambda_types::beacon::fork_digest::compute_fork_digest; +use ethlambda_types::beacon::primitives::{Epoch, ForkDigest, Root, Version}; +use ethlambda_types::enr::EnrForkId; + +/// Epochs before a digest activates at which its topics are joined. +pub const SUBSCRIBE_LEAD_EPOCHS: Epoch = 1; + +/// Epochs after the *next* digest activates at which a digest's topics are +/// left. +pub const UNSUBSCRIBE_LAG_EPOCHS: Epoch = 2; + +/// One digest and the epoch it becomes current. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ScheduledDigest { + /// The first epoch this digest is current at. + pub activation_epoch: Epoch, + pub digest: ForkDigest, + /// The fork version in effect at `activation_epoch`, for the `eth2` entry's + /// `next_fork_version` when this is the next boundary. + pub version: Version, + /// The fork in effect at `activation_epoch`, so the fork of everything + /// published under `digest`. + pub fork: ForkName, +} + +/// Every digest a chain will use, in activation order. +#[derive(Debug, Clone)] +pub struct ForkSchedule { + /// Never empty: the first entry is the genesis digest. Consecutive equal + /// digests are merged, so each entry is a real change on the wire. + entries: Vec, +} + +impl ForkSchedule { + /// Build the schedule from every fork epoch and blob-schedule epoch in + /// `config`. Unscheduled forks carry [`FAR_FUTURE_EPOCH`] and are skipped. + pub fn new(config: &Config, genesis_validators_root: Root) -> Self { + let mut epochs: Vec = ForkName::ALL + .into_iter() + .map(|fork| config.fork_epoch(fork)) + .chain(config.blob_schedule.iter().map(|entry| entry.epoch)) + .chain([0]) + .filter(|&epoch| epoch != FAR_FUTURE_EPOCH) + .collect(); + epochs.sort_unstable(); + epochs.dedup(); + + let mut entries: Vec = Vec::with_capacity(epochs.len()); + for epoch in epochs { + let digest = compute_fork_digest(config, genesis_validators_root, epoch); + // Only a blob-schedule entry before fulu, or one landing on an + // epoch another entry already covers, leaves the digest alone: + // from fulu on the digest hashes the entry's own epoch as well as + // its limit, so a later entry always moves it, even one restating + // the limit already in force. + if entries.last().is_some_and(|last| last.digest == digest) { + continue; + } + entries.push(ScheduledDigest { + activation_epoch: epoch, + digest, + version: config.fork_version(config.fork_at_epoch(epoch)), + fork: config.fork_at_epoch(epoch), + }); + } + Self { entries } + } + + /// Every scheduled digest, in activation order. + pub fn entries(&self) -> &[ScheduledDigest] { + &self.entries + } + + fn index_at(&self, epoch: Epoch) -> usize { + self.entries + .partition_point(|entry| entry.activation_epoch <= epoch) + .saturating_sub(1) + } + + /// The digest (and its fork) that is current at `epoch`. + pub fn current_at(&self, epoch: Epoch) -> ScheduledDigest { + self.entries[self.index_at(epoch)] + } + + /// The digest current at `epoch`. + pub fn digest_at(&self, epoch: Epoch) -> ForkDigest { + self.current_at(epoch).digest + } + + /// The fork a message published under `digest` is in, if any scheduled + /// digest is `digest`. + /// + /// This is how a gossip message's fork is taken from its own topic rather + /// than from whichever digest the node currently advertises. + pub fn fork_for_digest(&self, digest: ForkDigest) -> Option { + self.entries + .iter() + .find(|entry| entry.digest == digest) + .map(|entry| entry.fork) + } + + /// The digests whose topics the node holds at `epoch`, in activation order. + /// + /// Entry `i` is held from [`SUBSCRIBE_LEAD_EPOCHS`] before it activates + /// until [`UNSUBSCRIBE_LAG_EPOCHS`] after entry `i + 1` does; the first is + /// held from genesis and the last forever. Always contains the current + /// digest, since `epoch` lies inside its own window. + pub fn held_at(&self, epoch: Epoch) -> Vec { + self.entries + .iter() + .enumerate() + .filter(|&(index, entry)| { + let joined = index == 0 + || epoch.saturating_add(SUBSCRIBE_LEAD_EPOCHS) >= entry.activation_epoch; + let left = self.entries.get(index + 1).is_some_and(|next| { + epoch >= next.activation_epoch.saturating_add(UNSUBSCRIBE_LAG_EPOCHS) + }); + joined && !left + }) + .map(|(_, entry)| *entry) + .collect() + } + + /// The first epoch after `epoch` at which anything above changes: a join, a + /// switch or a leave. `None` once the schedule has run out. + pub fn next_change_after(&self, epoch: Epoch) -> Option { + self.entries + .iter() + .enumerate() + .flat_map(|(index, entry)| { + let join = entry.activation_epoch.saturating_sub(SUBSCRIBE_LEAD_EPOCHS); + let leave = self + .entries + .get(index + 1) + .map(|next| next.activation_epoch.saturating_add(UNSUBSCRIBE_LAG_EPOCHS)); + [Some(join), Some(entry.activation_epoch), leave] + }) + .flatten() + .filter(|&change| change > epoch) + .min() + } + + /// The next digest change after `epoch`, for the `eth2` entry's + /// `next_fork_*` and for the startup log. + pub fn next_boundary_after(&self, epoch: Epoch) -> Option { + self.entries + .iter() + .find(|entry| entry.activation_epoch > epoch) + .copied() + } + + /// The `eth2` ENR entry at `epoch`. + /// + /// `next_fork_*` name the next entry of this schedule, so never an epoch at + /// which the digest does not change. With none ahead, the spec says to + /// repeat the current version and name the far-future epoch. + pub fn enr_fork_id(&self, epoch: Epoch) -> EnrForkId { + let current = self.current_at(epoch); + match self.next_boundary_after(epoch) { + Some(next) => EnrForkId { + fork_digest: current.digest, + next_fork_version: next.version, + next_fork_epoch: next.activation_epoch, + }, + None => EnrForkId { + fork_digest: current.digest, + next_fork_version: current.version, + next_fork_epoch: FAR_FUTURE_EPOCH, + }, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use ethlambda_types::beacon::config::BlobScheduleEntry; + use libssz_types::SszList; + + const FULU: Epoch = 1_000; + const GLOAS: Epoch = 5_000; + + fn gvr() -> Root { + Root::repeat_byte(7) + } + + /// Sepolia's shape: fulu, then gloas, with no blob-schedule entry between. + fn fulu_then_gloas() -> Config { + let mut config = Config::mainnet() + .with_fork_epoch(ForkName::Altair, 1) + .with_fork_epoch(ForkName::Bellatrix, 2) + .with_fork_epoch(ForkName::Capella, 3) + .with_fork_epoch(ForkName::Deneb, 4) + .with_fork_epoch(ForkName::Electra, 5) + .with_fork_epoch(ForkName::Fulu, FULU) + .with_fork_epoch(ForkName::Gloas, GLOAS); + config.blob_schedule = SszList::new(); + config + } + + /// Fulu, two BPOs, then gloas. + fn with_bpos() -> Config { + let mut config = fulu_then_gloas(); + config.blob_schedule = SszList::try_from(vec![ + BlobScheduleEntry { + epoch: 2_000, + max_blobs_per_block: 15, + }, + BlobScheduleEntry { + epoch: 3_000, + max_blobs_per_block: 21, + }, + ]) + .expect("within capacity"); + config + } + + fn digests_held(schedule: &ForkSchedule, epoch: Epoch) -> Vec { + schedule + .held_at(epoch) + .into_iter() + .map(|entry| entry.digest) + .collect() + } + + #[test] + fn every_scheduled_digest_resolves_to_its_fork() { + let config = with_bpos(); + let schedule = ForkSchedule::new(&config, gvr()); + // Genesis, each fork through fulu, two BPOs, gloas. + let shape: Vec<(Epoch, ForkName)> = schedule + .entries() + .iter() + .map(|entry| (entry.activation_epoch, entry.fork)) + .collect(); + assert_eq!( + shape, + vec![ + (0, ForkName::Phase0), + (1, ForkName::Altair), + (2, ForkName::Bellatrix), + (3, ForkName::Capella), + (4, ForkName::Deneb), + (5, ForkName::Electra), + (FULU, ForkName::Fulu), + (2_000, ForkName::Fulu), + (3_000, ForkName::Fulu), + (GLOAS, ForkName::Gloas), + ] + ); + for entry in schedule.entries() { + assert_eq!(schedule.fork_for_digest(entry.digest), Some(entry.fork)); + assert_eq!( + entry.digest, + compute_fork_digest(&config, gvr(), entry.activation_epoch) + ); + } + assert_eq!(schedule.fork_for_digest([0xde, 0xad, 0xbe, 0xef]), None); + } + + #[test] + fn a_bpo_moves_the_digest_but_not_the_fork() { + let schedule = ForkSchedule::new(&with_bpos(), gvr()); + let before = schedule.current_at(1_999); + let after = schedule.current_at(2_000); + assert_ne!(before.digest, after.digest); + assert_eq!(before.fork, ForkName::Fulu); + assert_eq!(after.fork, ForkName::Fulu); + assert_eq!(schedule.fork_for_digest(after.digest), Some(ForkName::Fulu)); + } + + #[test] + fn the_gloas_boundary_resolves_both_sides() { + let schedule = ForkSchedule::new(&fulu_then_gloas(), gvr()); + assert_eq!(schedule.current_at(GLOAS - 1).fork, ForkName::Fulu); + assert_eq!(schedule.current_at(GLOAS).fork, ForkName::Gloas); + assert_eq!( + schedule.digest_at(GLOAS), + compute_fork_digest(&fulu_then_gloas(), gvr(), GLOAS) + ); + } + + #[test] + fn a_blob_entry_before_fulu_is_not_a_boundary() { + // The digest only hashes blob parameters from fulu on, so an entry + // placed before it cannot move anything on the wire. + let plain = ForkSchedule::new(&fulu_then_gloas(), gvr()); + let mut config = fulu_then_gloas(); + config.blob_schedule = SszList::try_from(vec![BlobScheduleEntry { + // An epoch no fork uses, so the entry reaches the digest + // comparison instead of being dropped as a duplicate epoch. + epoch: 500, + max_blobs_per_block: 9, + }]) + .expect("within capacity"); + let with_entry = ForkSchedule::new(&config, gvr()); + assert_eq!(with_entry.entries().len(), plain.entries().len()); + } + + #[test] + fn a_bpo_restating_the_previous_limit_is_still_a_boundary() { + // `compute_fork_digest` hashes the entry's epoch as well as its limit, + // so a later entry with the same limit moves the digest. + let mut config = fulu_then_gloas(); + config.blob_schedule = SszList::try_from(vec![ + BlobScheduleEntry { + epoch: 2_000, + max_blobs_per_block: 15, + }, + BlobScheduleEntry { + epoch: 3_000, + max_blobs_per_block: 15, + }, + ]) + .expect("within capacity"); + let schedule = ForkSchedule::new(&config, gvr()); + assert_ne!(schedule.digest_at(2_999), schedule.digest_at(3_000)); + assert!( + schedule + .entries() + .iter() + .any(|entry| entry.activation_epoch == 3_000) + ); + } + + #[test] + fn the_next_digest_is_joined_one_epoch_early_and_the_old_left_two_late() { + let schedule = ForkSchedule::new(&fulu_then_gloas(), gvr()); + let fulu = schedule.digest_at(GLOAS - 1); + let gloas = schedule.digest_at(GLOAS); + + // Two epochs ahead: only the pre-boundary digest. + assert_eq!(digests_held(&schedule, GLOAS - 2), vec![fulu]); + // B-1: both, and the old one is still the current. + assert_eq!(digests_held(&schedule, GLOAS - 1), vec![fulu, gloas]); + assert_eq!(schedule.digest_at(GLOAS - 1), fulu); + // B: both, and the new one is current. + assert_eq!(digests_held(&schedule, GLOAS), vec![fulu, gloas]); + assert_eq!(schedule.digest_at(GLOAS), gloas); + // B+1: still both. + assert_eq!(digests_held(&schedule, GLOAS + 1), vec![fulu, gloas]); + // B+2: the old topics go. + assert_eq!(digests_held(&schedule, GLOAS + 2), vec![gloas]); + } + + #[test] + fn a_node_started_inside_the_window_holds_both() { + // Start at B+1: nothing crossed the boundary, yet the state is the one + // a node that did cross it would be in. + let schedule = ForkSchedule::new(&fulu_then_gloas(), gvr()); + let held = digests_held(&schedule, GLOAS + 1); + assert_eq!(held.len(), 2); + assert!(held.contains(&schedule.digest_at(GLOAS + 1))); + } + + #[test] + fn the_current_digest_is_always_held() { + let schedule = ForkSchedule::new(&with_bpos(), gvr()); + for epoch in 0..6_000 { + assert!( + digests_held(&schedule, epoch).contains(&schedule.digest_at(epoch)), + "epoch {epoch}" + ); + } + } + + #[test] + fn changes_are_announced_at_every_join_switch_and_leave() { + let schedule = ForkSchedule::new(&fulu_then_gloas(), gvr()); + assert_eq!(schedule.next_change_after(GLOAS - 10), Some(GLOAS - 1)); + assert_eq!(schedule.next_change_after(GLOAS - 1), Some(GLOAS)); + assert_eq!(schedule.next_change_after(GLOAS), Some(GLOAS + 2)); + assert_eq!(schedule.next_change_after(GLOAS + 2), None); + } + + #[test] + fn the_held_set_only_changes_at_announced_epochs() { + let schedule = ForkSchedule::new(&with_bpos(), gvr()); + let mut previous = (digests_held(&schedule, 0), schedule.digest_at(0)); + let mut epoch = 0; + while epoch < 6_000 { + let next = schedule.next_change_after(epoch); + let until = next.unwrap_or(6_000); + for probe in epoch..until { + assert_eq!( + (digests_held(&schedule, probe), schedule.digest_at(probe)), + previous, + "epoch {probe} changed without being announced" + ); + } + epoch = until; + previous = (digests_held(&schedule, epoch), schedule.digest_at(epoch)); + } + } + + #[test] + fn the_next_boundary_names_the_digest_and_fork_ahead() { + let schedule = ForkSchedule::new(&fulu_then_gloas(), gvr()); + let next = schedule.next_boundary_after(FULU + 1).expect("gloas ahead"); + assert_eq!(next.activation_epoch, GLOAS); + assert_eq!(next.fork, ForkName::Gloas); + assert_eq!(schedule.next_boundary_after(GLOAS), None); + } + + #[test] + fn the_enr_entry_points_at_the_next_boundary() { + let schedule = ForkSchedule::new(&fulu_then_gloas(), gvr()); + let before = schedule.enr_fork_id(GLOAS - 1); + assert_eq!(before.next_fork_epoch, GLOAS); + let after = schedule.enr_fork_id(GLOAS); + assert_eq!(after.fork_digest, schedule.digest_at(GLOAS)); + assert_eq!(after.next_fork_epoch, FAR_FUTURE_EPOCH); + } + + #[test] + fn closely_spaced_boundaries_hold_each_digest_in_its_own_window() { + let mut config = fulu_then_gloas(); + config.blob_schedule = SszList::try_from(vec![ + BlobScheduleEntry { + epoch: 2_000, + max_blobs_per_block: 15, + }, + BlobScheduleEntry { + epoch: 2_001, + max_blobs_per_block: 21, + }, + ]) + .expect("within capacity"); + let schedule = ForkSchedule::new(&config, gvr()); + // Three digests are live at once around 2_001; none is dropped early. + assert_eq!(schedule.held_at(2_001).len(), 3); + } +} diff --git a/crates/net/p2p/src/beacon/handler.rs b/crates/net/p2p/src/beacon/handler.rs index c593815da..b8065a37d 100644 --- a/crates/net/p2p/src/beacon/handler.rs +++ b/crates/net/p2p/src/beacon/handler.rs @@ -231,7 +231,12 @@ mod tests { BeaconWire { fork_digest: [0x8c, 0x9f, 0x62, 0xfe], fork: ForkName::Fulu, + schedule: crate::beacon::fork_schedule::ForkSchedule::new( + &Config::mainnet(), + Root::ZERO, + ), topics: topics::BeaconTopics::new([0x8c, 0x9f, 0x62, 0xfe], &[], &[]), + window_topics: Vec::new(), config: Config::mainnet(), genesis_time: 1_606_824_023, genesis_validators_root: Root::ZERO, diff --git a/crates/net/p2p/src/beacon/messages.rs b/crates/net/p2p/src/beacon/messages.rs index e32d26256..2a89713ae 100644 --- a/crates/net/p2p/src/beacon/messages.rs +++ b/crates/net/p2p/src/beacon/messages.rs @@ -14,7 +14,7 @@ use libssz_derive::{SszDecode, SszEncode}; use libssz_types::{SszBitvector, SszList}; use super::constants::{ATTESTATION_SUBNET_COUNT, SYNC_COMMITTEE_SUBNET_COUNT}; -use super::protocols::MAX_REQUEST_BLOCKS_DENEB; +use super::protocols::{MAX_REQUEST_BLOCKS_DENEB, MAX_REQUEST_PAYLOADS}; /// `attnets`: which attestation subnets a node serves. pub type AttnetsBits = SszBitvector<{ ATTESTATION_SUBNET_COUNT as usize }>; @@ -206,6 +206,20 @@ pub struct DataColumnsByRangeRequest { pub columns: ColumnIndices, } +/// `ExecutionPayloadEnvelopeRoots`: `ExecutionPayloadEnvelopesByRoot` v1's +/// request body, the beacon block roots of the envelopes asked for, bare as an +/// SSZ field. +pub type ExecutionPayloadEnvelopeRoots = SszList; + +/// `ExecutionPayloadEnvelopesByRange` v1's request body. +/// +/// Fixed-size like the block range body, minus the deprecated `step`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, SszEncode, SszDecode)] +pub struct ExecutionPayloadEnvelopesByRangeRequest { + pub start_slot: Slot, + pub count: u64, +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/net/p2p/src/beacon/mod.rs b/crates/net/p2p/src/beacon/mod.rs index 3369ccba0..8ebc3721e 100644 --- a/crates/net/p2p/src/beacon/mod.rs +++ b/crates/net/p2p/src/beacon/mod.rs @@ -16,17 +16,20 @@ pub mod column_checks; pub mod decode; pub mod encoding; +pub mod envelope_checks; +pub mod fork_schedule; pub mod handler; pub mod messages; pub mod protocols; pub mod subnets; pub mod swarm; pub mod topics; +pub mod transition; pub mod verdict; use ethlambda_types::beacon::config::Config; use ethlambda_types::beacon::fork::ForkName; -use ethlambda_types::beacon::primitives::{ForkDigest, Root}; +use ethlambda_types::beacon::primitives::{ForkDigest, Root, Slot}; /// Everything the beacon wire needs after startup has computed it. /// @@ -34,15 +37,24 @@ use ethlambda_types::beacon::primitives::{ForkDigest, Root}; /// fork a gossip payload decodes under is derived from its slot, and that /// derivation must use the same schedule the fork digest was computed from. pub struct BeaconWire { + /// The digest this node publishes under and advertises in `Status`. Moves + /// at each boundary of [`Self::schedule`]; see [`transition`]. pub fork_digest: ForkDigest, - /// The fork `fork_digest` was computed at, so the fork of everything - /// that arrives on a subscribed topic. + /// The fork `fork_digest` was computed at. /// - /// Only `beacon_attestation_{subnet_id}` reads it. Every other - /// fork-dependent topic finds its fork from the slot inside the payload, - /// which this one cannot do; see [`decode::decode_attestation`]. + /// What a message published now is in. A received message's fork comes from + /// its own topic's digest through [`Self::schedule`] instead, since while a + /// boundary's window is open this node is subscribed under two digests. pub fork: ForkName, + /// Every digest the chain will use, and when to join and leave each. + pub schedule: fork_schedule::ForkSchedule, + /// The topics under `fork_digest`. pub topics: topics::BeaconTopics, + /// The topics of every other digest the subscription window holds: the + /// next one from [`fork_schedule::SUBSCRIBE_LEAD_EPOCHS`] ahead of its + /// boundary, the previous one until [`fork_schedule::UNSUBSCRIBE_LAG_EPOCHS`] + /// after. Empty outside a window. + pub window_topics: Vec, pub config: Config, pub genesis_time: u64, /// The chain every fork digest is bound to. @@ -52,8 +64,8 @@ pub struct BeaconWire { /// epoch, so serving history means computing digests this node never runs /// on. See [`encoding`]'s module docs for the rule. pub genesis_validators_root: Root, - /// Advertised in `Ping` responses and in `MetaData`. Never bumped today: - /// nothing this node advertises changes at runtime. + /// Advertised in `Ping` responses and in `MetaData`. Bumped when the + /// digest switches, the one advertised value that changes at runtime. pub metadata_seq_number: u64, /// The columns this node custodies, carried alongside `topics` so the /// gossip handler and the request handlers can check what this node @@ -67,6 +79,36 @@ pub struct BeaconWire { } impl BeaconWire { + /// Every topic set this node is subscribed to: the current digest's, then + /// the window's. + pub fn held_topics(&self) -> impl Iterator { + std::iter::once(&self.topics).chain(self.window_topics.iter()) + } + + /// The digest a message for `slot` is published under: the one the schedule + /// names for that slot's epoch, rather than whichever this node has + /// switched to so far, so a slot-B block goes out on the new topics even if + /// the switch has not run yet. + pub fn digest_for_slot(&self, slot: Slot) -> ForkDigest { + self.schedule + .digest_at(slot / ethlambda_types::beacon::preset::SLOTS_PER_EPOCH) + } + + /// [`Self::digest_for_slot`], but only if this node holds that digest's + /// topics: a slot far in the past or beyond the next boundary names a digest + /// nobody is listening to, and publishing there would only fan out to peers + /// that never subscribed. + pub fn publish_digest(&self, slot: Slot) -> Option { + Some(self.digest_for_slot(slot)).filter(|&digest| self.holds_digest(digest)) + } + + /// Whether this node is subscribed under `digest`, so whether a peer on it + /// is on a digest this node still speaks. + pub fn holds_digest(&self, digest: ForkDigest) -> bool { + self.held_topics() + .any(|topics| topics.fork_digest == digest) + } + /// The two values the codec needs to put a block chunk on or off the wire. pub fn codec_context(&self) -> BeaconContext { BeaconContext { diff --git a/crates/net/p2p/src/beacon/protocols.rs b/crates/net/p2p/src/beacon/protocols.rs index 3c376ba58..7498420de 100644 --- a/crates/net/p2p/src/beacon/protocols.rs +++ b/crates/net/p2p/src/beacon/protocols.rs @@ -9,6 +9,10 @@ //! into, and an unregistered protocol is still refused at stream negotiation //! rather than answered with a lie. //! +//! The two gloas execution payload envelope protocols are registered for the +//! same reason the column ones are: the server side answers out of +//! `Table::ExecutionPayloadEnvelopes`. Nothing requests them yet. +//! //! Only version 2 of the two block protocols is registered. Version 1 is //! deprecated by the spec, which lets a client answer it with an empty list, //! and its chunks carry no ``, so serving it would mean a second @@ -31,6 +35,14 @@ pub const DATA_COLUMN_SIDECARS_BY_RANGE_V1: &str = "/eth2/beacon_chain/req/data_column_sidecars_by_range/1/ssz_snappy"; pub const DATA_COLUMN_SIDECARS_BY_ROOT_V1: &str = "/eth2/beacon_chain/req/data_column_sidecars_by_root/1/ssz_snappy"; +pub const EXECUTION_PAYLOAD_ENVELOPES_BY_RANGE_V1: &str = + "/eth2/beacon_chain/req/execution_payload_envelopes_by_range/1/ssz_snappy"; +pub const EXECUTION_PAYLOAD_ENVELOPES_BY_ROOT_V1: &str = + "/eth2/beacon_chain/req/execution_payload_envelopes_by_root/1/ssz_snappy"; + +/// `MAX_REQUEST_PAYLOADS`: the ceiling on either gloas envelope request, and +/// on the list a response carries. +pub const MAX_REQUEST_PAYLOADS: u64 = 128; /// `MAX_REQUEST_BLOCKS`: the ceiling phase0 put on either block request. /// @@ -99,6 +111,14 @@ pub fn registrations() -> Vec<(StreamProtocol, ProtocolSupport)> { StreamProtocol::new(DATA_COLUMN_SIDECARS_BY_ROOT_V1), ProtocolSupport::Full, ), + ( + StreamProtocol::new(EXECUTION_PAYLOAD_ENVELOPES_BY_RANGE_V1), + ProtocolSupport::Full, + ), + ( + StreamProtocol::new(EXECUTION_PAYLOAD_ENVELOPES_BY_ROOT_V1), + ProtocolSupport::Full, + ), ] } @@ -116,6 +136,12 @@ pub fn label(protocol: &str) -> Option<&'static str> { BLOCKS_BY_ROOT_V2 => Some("beacon_blocks_by_root_v2"), DATA_COLUMN_SIDECARS_BY_RANGE_V1 => Some("beacon_data_column_sidecars_by_range"), DATA_COLUMN_SIDECARS_BY_ROOT_V1 => Some("beacon_data_column_sidecars_by_root"), + EXECUTION_PAYLOAD_ENVELOPES_BY_RANGE_V1 => { + Some("beacon_execution_payload_envelopes_by_range") + } + EXECUTION_PAYLOAD_ENVELOPES_BY_ROOT_V1 => { + Some("beacon_execution_payload_envelopes_by_root") + } _ => None, } } @@ -158,6 +184,18 @@ mod tests { ); } + #[test] + fn the_envelope_protocol_ids_are_the_spec_strings() { + assert_eq!( + EXECUTION_PAYLOAD_ENVELOPES_BY_RANGE_V1, + "/eth2/beacon_chain/req/execution_payload_envelopes_by_range/1/ssz_snappy" + ); + assert_eq!( + EXECUTION_PAYLOAD_ENVELOPES_BY_ROOT_V1, + "/eth2/beacon_chain/req/execution_payload_envelopes_by_root/1/ssz_snappy" + ); + } + #[test] fn a_sidecar_request_is_bounded_by_the_block_ceiling_times_the_columns() { assert_eq!( diff --git a/crates/net/p2p/src/beacon/topics.rs b/crates/net/p2p/src/beacon/topics.rs index b367a0536..ff4e6da60 100644 --- a/crates/net/p2p/src/beacon/topics.rs +++ b/crates/net/p2p/src/beacon/topics.rs @@ -28,6 +28,7 @@ use std::collections::BTreeMap; +use ethlambda_types::beacon::fork::ForkName; use ethlambda_types::beacon::primitives::ForkDigest; use libp2p::gossipsub::IdentTopic; @@ -46,7 +47,17 @@ pub const BLS_TO_EXECUTION_CHANGE: &str = "bls_to_execution_change"; /// Topic kind for aggregated sync committee contributions. pub const SYNC_COMMITTEE_CONTRIBUTION_AND_PROOF: &str = "sync_committee_contribution_and_proof"; -/// Every topic kind this node subscribes to, in the order they are subscribed. +/// Topic kind for gloas execution payload envelopes. +pub const EXECUTION_PAYLOAD: &str = "execution_payload"; +/// Topic kind for gloas payload timeliness committee votes. +pub const PAYLOAD_ATTESTATION_MESSAGE: &str = "payload_attestation_message"; + +/// The topic kinds gloas adds to [`SUBSCRIBED_TOPIC_KINDS`], subscribed from +/// the gloas digest on and never under an earlier one. +pub const GLOAS_TOPIC_KINDS: [&str; 2] = [EXECUTION_PAYLOAD, PAYLOAD_ATTESTATION_MESSAGE]; + +/// Every topic kind this node subscribes to at every fork, in the order they +/// are subscribed. [`GLOAS_TOPIC_KINDS`] follow from gloas. pub const SUBSCRIBED_TOPIC_KINDS: [&str; 7] = [ BEACON_BLOCK, BEACON_AGGREGATE_AND_PROOF, @@ -72,7 +83,11 @@ pub const BEACON_ATTESTATION_KIND: &str = "beacon_attestation"; /// `None` for anything else, lean kinds included, which is what tells the /// gossip handler a message needs no verdict. pub fn metric_kind(kind: &str) -> Option<&'static str> { - if let Some(&global) = SUBSCRIBED_TOPIC_KINDS.iter().find(|&&known| known == kind) { + if let Some(&global) = SUBSCRIBED_TOPIC_KINDS + .iter() + .chain(GLOAS_TOPIC_KINDS.iter()) + .find(|&&known| known == kind) + { return Some(global); } if data_column_subnet(kind).is_some() { @@ -99,6 +114,29 @@ pub fn topic_kind(topic: &str) -> Option<&str> { crate::gossipsub::topic_kind(topic) } +/// The fork digest embedded in a full topic name, or `None` if the name is not +/// shaped like a beacon topic or carries something other than four hex bytes. +/// +/// A message's fork is the fork its topic's digest names, which is not the +/// digest this node currently advertises while a fork boundary's subscription +/// window is open. +pub fn topic_digest(topic: &str) -> Option { + let digest = topic.split('/').nth(2)?; + hex::decode(digest).ok()?.try_into().ok() +} + +/// The column subnets a custody set maps onto, in custody order. +/// +/// A column's subnet is `column % DATA_COLUMN_SIDECAR_SUBNET_COUNT`, computed +/// rather than assumed so a network that ever separates the two counts still +/// subscribes to the right topic. `BeaconTopics::new` deduplicates. +pub fn column_subnets(custody_columns: &[u64]) -> Vec { + custody_columns + .iter() + .map(|column| column % ethlambda_types::beacon::constants::DATA_COLUMN_SIDECAR_SUBNET_COUNT) + .collect() +} + /// Topic family for unaggregated attestations, one topic per subnet. pub const BEACON_ATTESTATION_PREFIX: &str = "beacon_attestation_"; @@ -184,10 +222,55 @@ pub struct BeaconTopics { } impl BeaconTopics { + /// The topics to hold under `fork_digest`, for the fork that digest names. + /// + /// The one place a fork may change which topic kinds exist: a fork that adds + /// a kind extends its own arm, and the transition then joins those topics + /// ahead of the boundary as the spec asks. Every arm is named so a new fork + /// forces a decision here. + pub fn for_fork( + fork: ForkName, + fork_digest: ForkDigest, + column_subnets: &[u64], + attestation_subnets: &[u64], + ) -> Self { + match fork { + ForkName::Phase0 + | ForkName::Altair + | ForkName::Bellatrix + | ForkName::Capella + | ForkName::Deneb + | ForkName::Electra + | ForkName::Fulu => Self::new(fork_digest, column_subnets, attestation_subnets), + // Gloas adds the payload envelope and the payload attestation + // topics, which the follower consumes. + ForkName::Gloas => Self::with_extra_kinds( + fork_digest, + &GLOAS_TOPIC_KINDS, + column_subnets, + attestation_subnets, + ), + ForkName::Lean => { + unreachable!("a beacon topic's fork is never Lean: it is absent from ForkName::ALL") + } + } + } + pub fn new( fork_digest: ForkDigest, column_subnets: &[u64], attestation_subnets: &[u64], + ) -> Self { + Self::with_extra_kinds(fork_digest, &[], column_subnets, attestation_subnets) + } + + /// [`Self::new`] plus the global topic `extra_kinds` a fork adds, placed + /// after [`SUBSCRIBED_TOPIC_KINDS`] and ahead of the subnet families. + fn with_extra_kinds( + fork_digest: ForkDigest, + extra_kinds: &[&str], + column_subnets: &[u64], + attestation_subnets: &[u64], ) -> Self { // Built first so the map's own key semantics do the deduplication; // `topics` below only ever sees what survived that. @@ -207,6 +290,7 @@ impl BeaconTopics { let mut topics: Vec = SUBSCRIBED_TOPIC_KINDS .iter() + .chain(extra_kinds) .map(|kind| IdentTopic::new(topic_name(fork_digest, kind))) .collect(); topics.extend(column_topics.values().cloned()); @@ -259,6 +343,35 @@ mod tests { assert_eq!(topics.topics.len(), 7); } + #[test] + fn gloas_adds_the_envelope_and_payload_attestation_topics() { + let fork_topics = |fork| { + BeaconTopics::for_fork(fork, MAINNET, &[], &[]) + .topics + .iter() + .map(|topic| topic.to_string()) + .collect::>() + }; + let gloas = fork_topics(ForkName::Gloas); + let fulu = fork_topics(ForkName::Fulu); + assert_eq!(gloas.len(), SUBSCRIBED_TOPIC_KINDS.len() + 2); + assert_eq!(fulu.len(), SUBSCRIBED_TOPIC_KINDS.len()); + for kind in GLOAS_TOPIC_KINDS { + let name = topic_name(MAINNET, kind); + assert!(gloas.contains(&name), "gloas lacks {name}"); + assert!(!fulu.contains(&name), "fulu holds {name}"); + assert_eq!(metric_kind(kind), Some(kind)); + } + assert_eq!( + topic_name(MAINNET, EXECUTION_PAYLOAD), + "/eth2/8c9f62fe/execution_payload/ssz_snappy" + ); + assert_eq!( + topic_name(MAINNET, PAYLOAD_ATTESTATION_MESSAGE), + "/eth2/8c9f62fe/payload_attestation_message/ssz_snappy" + ); + } + #[test] fn no_subnet_family_is_subscribed() { // The narrow subscription set is a design decision, not an accident of @@ -287,6 +400,21 @@ mod tests { } } + #[test] + fn the_digest_reads_back_from_a_topic_name() { + assert_eq!( + topic_digest(&topic_name(MAINNET, BEACON_BLOCK)), + Some(MAINNET) + ); + assert_eq!( + topic_digest(&attestation_topic_name([0x0a, 0xbc, 0xde, 0xf0], 3)), + Some([0x0a, 0xbc, 0xde, 0xf0]) + ); + assert_eq!(topic_digest("/eth2/zz/beacon_block/ssz_snappy"), None); + assert_eq!(topic_digest("/eth2/8c9f62/beacon_block/ssz_snappy"), None); + assert_eq!(topic_digest("garbage"), None); + } + #[test] fn topic_kind_reads_the_name_back() { for kind in SUBSCRIBED_TOPIC_KINDS { diff --git a/crates/net/p2p/src/beacon/transition.rs b/crates/net/p2p/src/beacon/transition.rs new file mode 100644 index 000000000..59016d2e8 --- /dev/null +++ b/crates/net/p2p/src/beacon/transition.rs @@ -0,0 +1,504 @@ +//! Crossing fork-digest boundaries on a running node. +//! +//! [`ForkSchedule`](super::fork_schedule::ForkSchedule) says which digests the +//! node should hold at an epoch; this applies that to the swarm, the wire state +//! and discovery, and works out when to look again. Every fork activation and +//! every blob-parameter-only fork is a boundary, since both move the digest. +//! +//! [`apply`] is idempotent: it computes the target state from the epoch alone +//! and changes only what differs, so the first call at startup (which joins the +//! rest of a window the node started inside) and a call at a live crossing are +//! one code path, and a timer that fires twice does nothing the second time. + +use std::time::Duration; + +use ethlambda_types::beacon::config::Config; +use ethlambda_types::beacon::constants::MAXIMUM_GOSSIP_CLOCK_DISPARITY; +use ethlambda_types::beacon::preset; +use ethlambda_types::beacon::primitives::{Epoch, ForkDigest}; +use ethlambda_types::time::unix_now_ms; +use libp2p::gossipsub::IdentTopic; +use tracing::{error, info}; + +use super::BeaconWire; +use super::topics::{self, BeaconTopics}; +use crate::{P2PServer, Wire, discovery, metrics}; + +/// How far before a boundary a peer may already be on the next digest and still +/// count as current: clocks disagree by up to the gossip clock disparity. +const BOUNDARY_CLOCK_SKEW: Duration = Duration::from_millis(MAXIMUM_GOSSIP_CLOCK_DISPARITY); + +/// The topics [`apply`] joined and left, which it has also handed to the swarm. +/// Returned so the aggregator-subnet handling is observable in tests without a +/// second swarm. +#[derive(Debug, Default, PartialEq, Eq)] +pub(crate) struct Changes { + pub(crate) subscribed: Vec, + pub(crate) unsubscribed: Vec, +} + +/// Whether a peer whose `Status` carries `digest` is one to range-sync from. +/// +/// Answering a `Status` at any held digest is courtesy; syncing is a choice. +/// After a boundary a peer still on the previous digest has not upgraded, so +/// its chain is not the one being followed. Only the current digest qualifies, +/// or the next one within [`BOUNDARY_CLOCK_SKEW`] of its boundary, for a peer +/// whose clock is slightly ahead of ours. +pub(crate) fn is_syncable_digest(wire: &BeaconWire, digest: ForkDigest, now_ms: u64) -> bool { + let epoch = epoch_at_ms(&wire.config, wire.genesis_time, now_ms); + if digest == wire.schedule.digest_at(epoch) { + return true; + } + wire.schedule + .next_boundary_after(epoch) + .is_some_and(|next| { + next.digest == digest + && delay_until_epoch( + &wire.config, + wire.genesis_time, + next.activation_epoch, + now_ms, + ) <= BOUNDARY_CLOCK_SKEW + }) +} + +fn epoch_duration_ms(config: &Config) -> u64 { + config + .slot_duration_ms + .max(1) + .saturating_mul(preset::SLOTS_PER_EPOCH) +} + +/// The epoch the wall clock reads at `now_ms`; epoch 0 before genesis. +pub(crate) fn epoch_at_ms(config: &Config, genesis_time: u64, now_ms: u64) -> Epoch { + now_ms.saturating_sub(genesis_time.saturating_mul(1000)) / epoch_duration_ms(config) +} + +/// How long from `now_ms` until `epoch` begins; zero if it already has. +pub(crate) fn delay_until_epoch( + config: &Config, + genesis_time: u64, + epoch: Epoch, + now_ms: u64, +) -> Duration { + let start_ms = genesis_time + .saturating_mul(1000) + .saturating_add(epoch.saturating_mul(epoch_duration_ms(config))); + Duration::from_millis(start_ms.saturating_sub(now_ms)) +} + +/// Apply the schedule at the current wall-clock epoch, and return how long to +/// wait before the next join, switch or leave. `None` on a lean node or once the +/// schedule has run out. +pub(crate) fn advance(server: &mut P2PServer) -> Option { + let wire = server.wire.beacon()?; + let now_ms = unix_now_ms(); + let epoch = epoch_at_ms(&wire.config, wire.genesis_time, now_ms); + let next = wire.schedule.next_change_after(epoch).map(|next| { + // A timer that fires a hair early would read the same epoch and + // reschedule itself with no delay, spinning until the boundary; one + // millisecond floor keeps that to a handful of wakeups. + delay_until_epoch(&wire.config, wire.genesis_time, next, now_ms) + .max(Duration::from_millis(1)) + }); + apply(server, epoch); + next +} + +/// Move the node to the state the schedule prescribes at `epoch`. +/// +/// 1. Join the topics of every digest the window now covers, including the +/// aggregator-subnet topics already held for the current slot. +/// 2. Leave the topics of every digest the window no longer covers. +/// 3. If the current digest changed, switch what this node publishes and +/// advertises: digest, fork, metadata sequence number, metric, and the +/// `eth2` entry. +/// 4. Point the admission filter at the new entry, admitting every held digest. +pub(crate) fn apply(server: &mut P2PServer, epoch: Epoch) -> Changes { + let mut changes = Changes::default(); + let Wire::Beacon(wire) = &mut server.wire else { + return changes; + }; + let held = wire.schedule.held_at(epoch); + let current = wire.schedule.current_at(epoch); + // Checked before anything is joined, so a schedule bug cannot leave topics + // subscribed but untracked. `held_at` always contains the current digest. + if !held.iter().any(|entry| entry.digest == current.digest) { + error!( + epoch, + "The fork schedule holds no topics for the current digest" + ); + return changes; + } + let column_subnets = topics::column_subnets(&wire.custody_columns); + + let mut existing = std::mem::take(&mut wire.window_topics); + existing.push(wire.topics.clone()); + + let mut kept: Vec = Vec::with_capacity(held.len()); + for entry in &held { + if let Some(index) = existing + .iter() + .position(|topics| topics.fork_digest == entry.digest) + { + kept.push(existing.swap_remove(index)); + continue; + } + let joined = BeaconTopics::for_fork( + entry.fork, + entry.digest, + &column_subnets, + &wire.attestation_subnets, + ); + for topic in &joined.topics { + server.swarm_handle.subscribe(topic.clone()); + changes.subscribed.push(topic.to_string()); + } + for &subnet_id in server.aggregator_subnets.keys() { + let topic = attestation_topic(entry.digest, subnet_id); + changes.subscribed.push(topic.to_string()); + server.swarm_handle.subscribe(topic); + } + info!( + fork_digest = %hex::encode(entry.digest), + fork = entry.fork.as_str(), + activation_epoch = entry.activation_epoch, + epoch, + topics = joined.topics.len(), + "Joined the topics of an upcoming or live fork digest" + ); + kept.push(joined); + } + + for left in existing { + for topic in &left.topics { + server.swarm_handle.unsubscribe(topic.clone()); + changes.unsubscribed.push(topic.to_string()); + } + for &subnet_id in server.aggregator_subnets.keys() { + let topic = attestation_topic(left.fork_digest, subnet_id); + changes.unsubscribed.push(topic.to_string()); + server.swarm_handle.unsubscribe(topic); + } + info!( + fork_digest = %hex::encode(left.fork_digest), + epoch, + "Left the topics of a past fork digest" + ); + } + + let index = kept + .iter() + .position(|topics| topics.fork_digest == current.digest) + .expect("`kept` holds one entry per held digest, checked above to include the current"); + wire.topics = kept.remove(index); + wire.window_topics = kept; + + let switched = wire.fork_digest != current.digest; + if switched { + let previous = wire.fork_digest; + wire.fork_digest = current.digest; + wire.fork = current.fork; + wire.metadata_seq_number += 1; + let digest_hex = hex::encode(current.digest); + metrics::set_beacon_fork_digest(&digest_hex); + info!( + epoch, + previous_digest = %hex::encode(previous), + fork_digest = %digest_hex, + fork = current.fork.as_str(), + metadata_seq_number = wire.metadata_seq_number, + "Crossed a fork digest boundary" + ); + } + + let fork_id = wire.schedule.enr_fork_id(epoch); + let also_admitted: Vec = wire + .window_topics + .iter() + .map(|topics| topics.fork_digest) + .collect(); + // Without discovery there is no peer table whose admission to update. + if let Some(state) = &server.discovery { + state.filter().set_fork_id(fork_id, also_admitted); + } + if switched { + discovery::update_served_enr(&fork_id); + } + changes +} + +fn attestation_topic(digest: ForkDigest, subnet_id: u64) -> IdentTopic { + IdentTopic::new(topics::attestation_topic_name(digest, subnet_id)) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::test_support::unconnected_beacon_server; + use ethlambda_types::beacon::fork::ForkName; + use libssz_types::SszList; + + use ethlambda_types::beacon::config::BlobScheduleEntry; + + const FULU: Epoch = 1_000; + const BPO: Epoch = 2_000; + const GLOAS: Epoch = 5_000; + + /// Fulu, one BPO, then gloas, on a chain whose wire the test server builds + /// with `Root::ZERO` as its genesis validators root. + fn config() -> Config { + let mut config = Config::mainnet() + .with_fork_epoch(ForkName::Altair, 1) + .with_fork_epoch(ForkName::Bellatrix, 2) + .with_fork_epoch(ForkName::Capella, 3) + .with_fork_epoch(ForkName::Deneb, 4) + .with_fork_epoch(ForkName::Electra, 5) + .with_fork_epoch(ForkName::Fulu, FULU) + .with_fork_epoch(ForkName::Gloas, GLOAS); + config.blob_schedule = SszList::try_from(vec![BlobScheduleEntry { + epoch: BPO, + max_blobs_per_block: 15, + }]) + .expect("within capacity"); + config + } + + fn digests(server: &P2PServer) -> (ForkDigest, Vec) { + let wire = server.wire.beacon().expect("a beacon wire"); + ( + wire.fork_digest, + wire.window_topics + .iter() + .map(|topics| topics.fork_digest) + .collect(), + ) + } + + #[test] + fn the_epoch_is_read_off_the_slot_duration() { + let config = Config::mainnet(); + let epoch_ms = config.slot_duration_ms * preset::SLOTS_PER_EPOCH; + let genesis = 1_000; + assert_eq!(epoch_at_ms(&config, genesis, 0), 0); + assert_eq!( + epoch_at_ms(&config, genesis, genesis * 1000 + epoch_ms - 1), + 0 + ); + assert_eq!(epoch_at_ms(&config, genesis, genesis * 1000 + epoch_ms), 1); + assert_eq!( + delay_until_epoch(&config, genesis, 3, genesis * 1000 + epoch_ms), + Duration::from_millis(2 * epoch_ms) + ); + assert_eq!( + delay_until_epoch(&config, genesis, 1, genesis * 1000 + 2 * epoch_ms), + Duration::ZERO + ); + } + + #[tokio::test] + async fn a_live_crossing_joins_switches_and_leaves_in_order() { + let config = config(); + let mut server = unconnected_beacon_server(config.clone(), 0).await; + let schedule = server.wire.beacon().expect("beacon").schedule.clone(); + let bpo = schedule.digest_at(BPO); + let before = schedule.digest_at(BPO - 1); + + // Before the window: only the current digest. + apply(&mut server, BPO - 2); + assert_eq!(digests(&server), (before, vec![])); + let seq_before = server.wire.beacon().unwrap().metadata_seq_number; + + // B-1: the next digest is joined while the old one stays current. + apply(&mut server, BPO - 1); + assert_eq!(digests(&server), (before, vec![bpo])); + assert_eq!( + server.wire.beacon().unwrap().metadata_seq_number, + seq_before + ); + + // B: the switch. Publishing and Status move, both digests stay held, + // and the sequence number is bumped exactly once. + apply(&mut server, BPO); + assert_eq!(digests(&server), (bpo, vec![before])); + let wire = server.wire.beacon().unwrap(); + assert_eq!(wire.metadata_seq_number, seq_before + 1); + assert!(wire.holds_digest(before) && wire.holds_digest(bpo)); + + // Applying again at the same epoch changes nothing. + apply(&mut server, BPO); + assert_eq!(digests(&server), (bpo, vec![before])); + assert_eq!( + server.wire.beacon().unwrap().metadata_seq_number, + seq_before + 1 + ); + + // B+2: the old topics are left. + apply(&mut server, BPO + 2); + assert_eq!(digests(&server), (bpo, vec![])); + assert!(!server.wire.beacon().unwrap().holds_digest(before)); + } + + #[tokio::test] + async fn the_gloas_crossing_switches_the_fork_the_node_publishes_in() { + let config = config(); + let mut server = unconnected_beacon_server(config, 0).await; + apply(&mut server, GLOAS - 1); + assert_eq!(server.wire.beacon().unwrap().fork, ForkName::Fulu); + apply(&mut server, GLOAS); + assert_eq!(server.wire.beacon().unwrap().fork, ForkName::Gloas); + } + + #[tokio::test] + async fn a_node_started_inside_the_window_joins_both_digests() { + let config = config(); + let mut server = unconnected_beacon_server(config, 0).await; + let schedule = server.wire.beacon().expect("beacon").schedule.clone(); + + // The startup call, at B+1: the node was built on one digest and the + // window says two. + apply(&mut server, GLOAS + 1); + let (current, window) = digests(&server); + assert_eq!(current, schedule.digest_at(GLOAS)); + assert_eq!(window, vec![schedule.digest_at(GLOAS - 1)]); + } + + /// While the rollover window holds a fulu and a gloas digest, only the + /// gloas one carries the two topics gloas adds, whether the gloas digest + /// is joined ahead of the fork or is current. + #[tokio::test] + async fn only_the_gloas_digest_holds_the_gloas_topics() { + let has_gloas_kinds = |held: &BeaconTopics| { + topics::GLOAS_TOPIC_KINDS.iter().all(|kind| { + let name = topics::topic_name(held.fork_digest, kind); + held.topics.iter().any(|topic| topic.to_string() == name) + }) + }; + let has_no_gloas_kinds = |held: &BeaconTopics| { + topics::GLOAS_TOPIC_KINDS.iter().all(|kind| { + let name = topics::topic_name(held.fork_digest, kind); + held.topics.iter().all(|topic| topic.to_string() != name) + }) + }; + + let mut server = unconnected_beacon_server(config(), 0).await; + // Ahead of the fork: fulu is current, gloas is joined in the window. + apply(&mut server, GLOAS - 1); + let wire = server.wire.beacon().unwrap(); + assert!(has_no_gloas_kinds(&wire.topics)); + assert!(has_gloas_kinds(&wire.window_topics[0])); + + // After it: gloas is current, fulu is still held. + apply(&mut server, GLOAS); + let wire = server.wire.beacon().unwrap(); + assert!(has_gloas_kinds(&wire.topics)); + assert!(has_no_gloas_kinds(&wire.window_topics[0])); + } + + #[tokio::test] + async fn a_message_takes_its_fork_from_its_topic() { + let config = config(); + let mut server = unconnected_beacon_server(config, 0).await; + apply(&mut server, GLOAS); + let wire = server.wire.beacon().unwrap(); + // Gloas is current, yet a message on the still-held fulu topic is fulu. + let fulu = wire.schedule.digest_at(GLOAS - 1); + let topic = topics::topic_name(fulu, topics::BEACON_BLOCK); + let digest = topics::topic_digest(&topic).expect("a digest"); + assert_eq!(wire.schedule.fork_for_digest(digest), Some(ForkName::Fulu)); + assert_eq!(wire.fork, ForkName::Gloas); + } + + fn epoch_start_ms(wire: &BeaconWire, epoch: Epoch) -> u64 { + wire.genesis_time * 1000 + epoch * epoch_duration_ms(&wire.config) + } + + #[tokio::test] + async fn aggregator_subnets_follow_every_held_digest() { + let mut server = unconnected_beacon_server(config(), 0).await; + let schedule = server.wire.beacon().expect("beacon").schedule.clone(); + let fulu = schedule.digest_at(GLOAS - 2); + let gloas = schedule.digest_at(GLOAS); + server.aggregator_subnets.insert(9, u64::MAX); + let name = |digest| topics::attestation_topic_name(digest, 9); + + // Settle on the pre-window digest first. + apply(&mut server, GLOAS - 2); + + // Joining the next digest brings the aggregator subnet along. + let joined = apply(&mut server, GLOAS - 1); + assert!(joined.subscribed.contains(&name(gloas))); + assert!(joined.unsubscribed.is_empty()); + + // Leaving the old digest takes it away again. + let left = apply(&mut server, GLOAS + 2); + assert!(left.unsubscribed.contains(&name(fulu))); + assert!(left.subscribed.is_empty()); + + // Nothing to do the second time. + assert_eq!(apply(&mut server, GLOAS + 2), Changes::default()); + } + + #[tokio::test] + async fn publishing_follows_the_messages_own_slot() { + let mut server = unconnected_beacon_server(config(), 0).await; + apply(&mut server, GLOAS - 1); + let wire = server.wire.beacon().unwrap(); + let first_gloas_slot = GLOAS * preset::SLOTS_PER_EPOCH; + // Before the switch has run, a slot-B message still names the new digest. + assert_eq!(wire.fork, ForkName::Fulu); + assert_eq!( + wire.digest_for_slot(first_gloas_slot), + wire.schedule.digest_at(GLOAS) + ); + assert_eq!( + wire.digest_for_slot(first_gloas_slot - 1), + wire.schedule.digest_at(GLOAS - 1) + ); + } + + #[tokio::test] + async fn a_digest_that_is_not_held_is_not_published_to() { + let mut server = unconnected_beacon_server(config(), 0).await; + apply(&mut server, GLOAS - 2); + let wire = server.wire.beacon().unwrap(); + let slot_of = |epoch: Epoch| epoch * preset::SLOTS_PER_EPOCH; + // The current digest, and the next one once its window has opened. + assert!(wire.publish_digest(slot_of(GLOAS - 2)).is_some()); + assert!(wire.publish_digest(slot_of(GLOAS)).is_none()); + apply(&mut server, GLOAS - 1); + let wire = server.wire.beacon().unwrap(); + assert!(wire.publish_digest(slot_of(GLOAS)).is_some()); + // A slot long past names a digest that is no longer held. + assert!(wire.publish_digest(slot_of(FULU)).is_none()); + } + + #[tokio::test] + async fn sync_starts_only_from_a_peer_on_a_current_digest() { + let mut server = unconnected_beacon_server(config(), 0).await; + apply(&mut server, GLOAS); + let wire = server.wire.beacon().unwrap(); + let old = wire.schedule.digest_at(GLOAS - 1); + let new = wire.schedule.digest_at(GLOAS); + + // After the boundary: the current digest, not the superseded one, even + // though the latter is still held (and so still answered). + let after = epoch_start_ms(wire, GLOAS + 1); + assert!(wire.holds_digest(old)); + assert!(is_syncable_digest(wire, new, after)); + assert!(!is_syncable_digest(wire, old, after)); + + // Just before it, within the skew: a peer whose clock is ahead is fine. + let skew = BOUNDARY_CLOCK_SKEW.as_millis() as u64; + let almost = epoch_start_ms(wire, GLOAS) - skew; + assert!(is_syncable_digest(wire, old, almost)); + assert!(is_syncable_digest(wire, new, almost)); + + // Earlier than the skew allows, the next digest is not yet current. + let early = epoch_start_ms(wire, GLOAS) - skew - 1; + assert!(!is_syncable_digest(wire, new, early)); + + // A digest on no schedule never is. + assert!(!is_syncable_digest(wire, [0xde, 0xad, 0xbe, 0xef], after)); + } +} diff --git a/crates/net/p2p/src/beacon/verdict.rs b/crates/net/p2p/src/beacon/verdict.rs index 27f9dadfb..ab99e9525 100644 --- a/crates/net/p2p/src/beacon/verdict.rs +++ b/crates/net/p2p/src/beacon/verdict.rs @@ -19,8 +19,11 @@ use ethlambda_state_transition::beacon::gossip::{self, IgnoreReason, Outcome}; use ethlambda_state_transition::beacon::helpers::accessors::CommitteeCacheExt as _; use ethlambda_storage::{CacheKey, Store}; use ethlambda_types::beacon::containers::electra::SingleAttestation; +use ethlambda_types::beacon::containers::gloas::{ + PayloadAttestationMessage, SignedExecutionPayloadEnvelope, +}; use ethlambda_types::beacon::containers::{ - SignedAggregateAndProof, SignedBeaconBlock, fulu::DataColumnSidecar, + DataColumnSidecar, SignedAggregateAndProof, SignedBeaconBlock, }; use ethlambda_types::beacon::primitives::{Root, ValidatorIndex}; use libp2p::PeerId; @@ -72,6 +75,11 @@ pub(crate) enum Validated { attestation: Box, subnet_id: u64, }, + /// A gloas `execution_payload`. Boxed for the reason `Block` is: it + /// carries a whole execution payload. + Envelope(Box), + /// A gloas `payload_attestation_message`. + PayloadAttestation(PayloadAttestationMessage), } impl Validated { @@ -86,7 +94,12 @@ impl Validated { Self::Block { block, block_root } => { gossip::block::stateful_checks(store, block, *block_root) } - Self::Column(sidecar) => gossip::column::stateful_checks(store, sidecar), + Self::Column(sidecar) => match &**sidecar { + DataColumnSidecar::Fulu(sidecar) => gossip::column::stateful_checks(store, sidecar), + DataColumnSidecar::Gloas(sidecar) => { + gossip::column::stateful_checks_gloas(store, sidecar) + } + }, Self::Aggregate { aggregate, attesting_indices, @@ -101,6 +114,10 @@ impl Validated { attestation, subnet_id, } => gossip::attestation::stateful_checks(store, attestation, *subnet_id), + Self::Envelope(envelope) => gossip::envelope::stateful_checks(store, envelope), + Self::PayloadAttestation(message) => { + gossip::payload_attestation::stateful_checks(store, message) + } } } @@ -113,14 +130,26 @@ impl Validated { .seen_blocks .record(block.slot(), block.proposer_index(), *block_root) } - Self::Column(sidecar) => { - let header = &sidecar.signed_block_header.message; - server - .seen_columns - .record(header.slot, header.proposer_index, sidecar.index) - } + Self::Column(sidecar) => match &**sidecar { + DataColumnSidecar::Fulu(sidecar) => { + let header = &sidecar.signed_block_header.message; + server + .seen_columns + .record(header.slot, header.proposer_index, sidecar.index) + } + DataColumnSidecar::Gloas(sidecar) => server + .seen_block_columns + .record(sidecar.beacon_block_root, sidecar.index), + }, Self::Aggregate { aggregate, .. } => server.seen_aggregates.record(aggregate), Self::Attestation { attestation, .. } => server.seen_attestations.record(attestation), + Self::Envelope(envelope) => server.seen_envelopes.record( + envelope.message.beacon_block_root, + envelope.message.builder_index, + ), + Self::PayloadAttestation(message) => server + .seen_payload_attestations + .record(message.data.slot, message.validator_index), } } @@ -141,6 +170,14 @@ impl Validated { /// validators, which verifies and relays its own backbone subnets but /// never calls `apply_attestation_to_fork_choice` for them either. /// + /// An envelope goes on when accepted, and also when queued: the chain + /// actor holds an envelope whose block it has not imported yet, which is + /// exactly what `Queue` means, and its own checks judge it again once the + /// block is there; a queued one has not had its signature checked, so the + /// actor checks it before it parks one. Any other outcome keeps it away. A + /// payload attestation goes on only when accepted: it is only valid within + /// its own slot, and the specification makes queueing one optional. + /// /// What an accepted subnet attestation does feed is the attestation pool, /// when its subnet is one a validator client's aggregator had this node /// join: that aggregator will ask for exactly these votes. See @@ -212,7 +249,30 @@ impl Validated { .new_beacon_aggregate(aggregate, attesting_indices, arrival) .inspect_err(|err| warn!(%err, "Failed to forward a gossip aggregate")); } - Self::Aggregate { .. } | Self::Attestation { .. } => {} + Self::Envelope(envelope) if matches!(outcome, Outcome::Accept | Outcome::Queue(_)) => { + let arrival = BlockArrival { + decode_start: Some(received_at), + handed_off: Instant::now(), + deferred_from: None, + }; + let _ = blockchain + .new_execution_payload_envelope(envelope, arrival) + .inspect_err(|err| warn!(%err, "Failed to forward a gossip envelope")); + } + Self::PayloadAttestation(message) if outcome == Outcome::Accept => { + let arrival = BlockArrival { + decode_start: Some(received_at), + handed_off: Instant::now(), + deferred_from: None, + }; + let _ = blockchain + .new_payload_attestation_message(message, arrival) + .inspect_err(|err| warn!(%err, "Failed to forward a payload attestation")); + } + Self::Aggregate { .. } + | Self::Attestation { .. } + | Self::Envelope(_) + | Self::PayloadAttestation(_) => {} } } } @@ -240,7 +300,10 @@ fn record_liveness(server: &P2PServer, object: &Validated) { Validated::Attestation { attestation, .. } => { observed.record(attestation.data.target.epoch, attestation.attester_index); } - Validated::Block { .. } | Validated::Column(_) => {} + Validated::Block { .. } + | Validated::Column(_) + | Validated::Envelope(_) + | Validated::PayloadAttestation(_) => {} } } @@ -395,6 +458,23 @@ pub(crate) fn report(server: &P2PServer, id: GossipId, outcome: Outcome) -> bool forward } +/// The permit pool `object`'s stateful checks draw from: blocks, columns and +/// envelopes from the gossip pool, everything the attesters send from the +/// attestation pool, so neither burst starves the other. +fn permits_for<'a>( + server: &'a P2PServer, + object: &Validated, +) -> &'a std::sync::Arc { + match object { + Validated::Block { .. } | Validated::Column(_) | Validated::Envelope(_) => { + &server.gossip_validation_permits + } + Validated::Aggregate { .. } + | Validated::Attestation { .. } + | Validated::PayloadAttestation(_) => &server.attestation_validation_permits, + } +} + /// Run `object`'s stateful checks on a blocking thread. The verdict comes back /// to the actor as a [`GossipVerdict`]. /// @@ -420,13 +500,7 @@ pub(crate) fn spawn_stateful_checks( id: GossipId, object: Validated, ) { - let permits = match &object { - Validated::Block { .. } | Validated::Column(_) => &server.gossip_validation_permits, - Validated::Aggregate { .. } | Validated::Attestation { .. } => { - &server.attestation_validation_permits - } - }; - let Ok(permit) = permits.clone().try_acquire_owned() else { + let Ok(permit) = permits_for(server, &object).clone().try_acquire_owned() else { let received_at = id.received_at; let outcome = Outcome::Ignore(IgnoreReason::Overloaded); report(server, id, outcome); @@ -474,12 +548,17 @@ pub(crate) fn guarded(checks: impl FnOnce() -> Outcome) -> Outcome { mod tests { use std::sync::Arc; + use ethlambda_network_api::P2PToBlockChain; use ethlambda_state_transition::beacon::gossip::{QueueReason, RejectReason}; + use ethlambda_types::attestation::{SignedAggregatedAttestation, SignedAttestation}; use ethlambda_types::beacon::config::Config; use ethlambda_types::beacon::containers::{AttestationData, Checkpoint, electra, phase0}; + use spawned_concurrency::error::ActorError; use super::*; - use crate::test_support::{RecordingChain, unconnected_beacon_server, valid_shaped_sidecar}; + use crate::test_support::{ + RecordingChain, envelope, unconnected_beacon_server, valid_shaped_sidecar, + }; /// A minimal fulu block for a given `(slot, proposer)`: `settle` and /// `record_seen` only ever read those two fields plus the root passed @@ -630,7 +709,9 @@ mod tests { #[tokio::test] async fn the_first_accept_for_a_column_key_stands_and_the_second_is_marked_seen() { let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; - let object = Validated::Column(Box::new(valid_shaped_sidecar(5, 0))); + let object = Validated::Column(Box::new(DataColumnSidecar::Fulu(valid_shaped_sidecar( + 5, 0, + )))); assert_eq!( settle(&mut server, Outcome::Accept, &object), @@ -856,15 +937,196 @@ mod tests { assert!(chain.aggregates.lock().unwrap().is_empty()); } + /// What `forward` handed the chain actor, by method. + #[derive(Debug, PartialEq)] + enum Sent { + Envelope, + PayloadAttestation, + } + + /// A chain actor stand-in that reports only the two gloas deliveries. + struct GloasRecorder(tokio::sync::mpsc::UnboundedSender); + + impl P2PToBlockChain for GloasRecorder { + fn new_block( + &self, + _block: SignedBeaconBlock, + _source: BlockSource, + _arrival: BlockArrival, + _announcement: BlockAnnouncement, + ) -> Result<(), ActorError> { + Ok(()) + } + fn new_attestation(&self, _attestation: SignedAttestation) -> Result<(), ActorError> { + Ok(()) + } + fn new_aggregated_attestation( + &self, + _attestation: SignedAggregatedAttestation, + ) -> Result<(), ActorError> { + Ok(()) + } + fn new_data_column_sidecars( + &self, + _sidecars: Vec, + ) -> Result<(), ActorError> { + Ok(()) + } + fn data_column_sidecars_awaiting_parent( + &self, + _sidecars: Vec, + ) -> Result<(), ActorError> { + Ok(()) + } + fn new_beacon_aggregate( + &self, + _aggregate: Box, + _attesting_indices: Vec, + _arrival: AggregateArrival, + ) -> Result<(), ActorError> { + Ok(()) + } + fn new_execution_payload_envelope( + &self, + _envelope: Box, + _arrival: BlockArrival, + ) -> Result<(), ActorError> { + let _ = self.0.send(Sent::Envelope); + Ok(()) + } + fn new_payload_attestation_message( + &self, + _message: PayloadAttestationMessage, + _arrival: BlockArrival, + ) -> Result<(), ActorError> { + let _ = self.0.send(Sent::PayloadAttestation); + Ok(()) + } + } + + pub(crate) fn payload_attestation(slot: u64, validator: u64) -> PayloadAttestationMessage { + let mut message = PayloadAttestationMessage { + validator_index: validator, + data: Default::default(), + signature: Default::default(), + }; + message.data.slot = slot; + message + } + + #[tokio::test] + async fn the_first_accept_for_an_envelope_key_stands_and_the_second_is_marked_seen() { + let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; + let object = Validated::Envelope(Box::new(envelope(1, 3))); + + assert_eq!( + settle(&mut server, Outcome::Accept, &object), + Outcome::Accept + ); + assert_eq!( + settle(&mut server, Outcome::Accept, &object), + Outcome::Ignore(IgnoreReason::AlreadySeen) + ); + // A queued envelope records nothing, so a later accept still stands. + let other = Validated::Envelope(Box::new(envelope(2, 3))); + assert_eq!( + settle( + &mut server, + Outcome::Queue(QueueReason::BlockUnknown), + &other + ), + Outcome::Queue(QueueReason::BlockUnknown) + ); + assert_eq!( + settle(&mut server, Outcome::Accept, &other), + Outcome::Accept + ); + } + + #[tokio::test] + async fn the_first_accept_for_a_payload_attestation_key_stands_and_the_second_is_marked_seen() { + let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; + let object = Validated::PayloadAttestation(payload_attestation(5, 1)); + + assert_eq!( + settle(&mut server, Outcome::Accept, &object), + Outcome::Accept + ); + assert_eq!( + settle(&mut server, Outcome::Accept, &object), + Outcome::Ignore(IgnoreReason::AlreadySeen) + ); + } + + /// An envelope reaches the chain actor on `Accept` and on `Queue` (the + /// actor holds it for its block), and on nothing else. + #[tokio::test] + async fn an_envelope_is_forwarded_on_accept_and_queue_only() { + let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; + let (sender, mut received) = tokio::sync::mpsc::unbounded_channel(); + server.blockchain = Some(Arc::new(GloasRecorder(sender))); + + let forward = |outcome| { + Validated::Envelope(Box::new(envelope(1, 3))).forward(&server, Instant::now(), outcome) + }; + forward(Outcome::Accept); + assert_eq!(received.try_recv(), Ok(Sent::Envelope)); + forward(Outcome::Queue(QueueReason::BlockUnknown)); + assert_eq!(received.try_recv(), Ok(Sent::Envelope)); + forward(Outcome::Ignore(IgnoreReason::Overloaded)); + forward(Outcome::Reject(RejectReason::BadSignature)); + assert!(received.try_recv().is_err()); + } + + /// A payload attestation reaches the chain actor on `Accept` alone. + #[tokio::test] + async fn a_payload_attestation_is_forwarded_on_accept_only() { + let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; + let (sender, mut received) = tokio::sync::mpsc::unbounded_channel(); + server.blockchain = Some(Arc::new(GloasRecorder(sender))); + + let forward = |outcome| { + Validated::PayloadAttestation(payload_attestation(5, 1)).forward( + &server, + Instant::now(), + outcome, + ) + }; + forward(Outcome::Ignore(IgnoreReason::UnknownBlock)); + forward(Outcome::Ignore(IgnoreReason::Overloaded)); + forward(Outcome::Reject(RejectReason::NotInPtc)); + assert!(received.try_recv().is_err()); + forward(Outcome::Accept); + assert_eq!(received.try_recv(), Ok(Sent::PayloadAttestation)); + } + /// The pool an aggregate or a subnet attestation draws its stateful-check /// permit from is not the pool a block or a column draws from: exhausting /// one must leave the other untouched, or a burst on this topic could /// make a block or a column answer `Ignore(Overloaded)` too. + /// + /// An envelope shares the block pool and a payload attestation the + /// attestation pool, for the same reason: a slot's committee votes arrive + /// in a burst that must not starve block validation. #[tokio::test] async fn the_block_column_and_attestation_permit_pools_are_independent() { let server = unconnected_beacon_server(Config::mainnet(), 0).await; let attestation_permits_before = server.attestation_validation_permits.available_permits(); + let same_pool = |a: &std::sync::Arc, b: &std::sync::Arc<_>| { + std::sync::Arc::ptr_eq(a, b) + }; + let envelope = Validated::Envelope(Box::new(envelope(1, 3))); + let vote = Validated::PayloadAttestation(payload_attestation(5, 1)); + assert!(same_pool( + permits_for(&server, &envelope), + &server.gossip_validation_permits + )); + assert!(same_pool( + permits_for(&server, &vote), + &server.attestation_validation_permits + )); + let mut held = Vec::new(); while let Ok(permit) = server.gossip_validation_permits.clone().try_acquire_owned() { held.push(permit); diff --git a/crates/net/p2p/src/discovery/admission.rs b/crates/net/p2p/src/discovery/admission.rs index c4068ba60..6a1c6adf5 100644 --- a/crates/net/p2p/src/discovery/admission.rs +++ b/crates/net/p2p/src/discovery/admission.rs @@ -18,6 +18,7 @@ //! ([`rank_candidates`]). use std::collections::HashSet; +use std::sync::{Arc, RwLock}; use ethrex_p2p::peer_filter::PeerFilter; use ethrex_p2p::types::NodeRecord; @@ -27,6 +28,7 @@ use tracing::debug; use ethlambda_state_transition::beacon::das; use ethlambda_types::beacon::constants; +use ethlambda_types::beacon::primitives::ForkDigest; use super::enr::{ ATTNETS_ENR_KEY, CGC_ENR_KEY, ETH2_ENR_KEY, EnrForkId, node_id_from_peer_id, read_ip, @@ -80,23 +82,51 @@ pub(crate) enum RejectReason { /// Holds what [`admit`] needs to judge a record, so the dial loop no longer /// carries the local fork id and committee count around: the value handed to /// [`PeerTableServer::spawn_with_filter`](ethrex_p2p::peer_table::PeerTableServer::spawn_with_filter) -/// judges by the same rules the dial loop later asks for dial targets. It is +/// judges by the same rules the dial loop later asks for dial targets. +/// /// [`Clone`] because the peer table takes ownership of the filter it runs, and -/// both fields are plain data. +/// every clone shares one [`Policy`], so [`Self::set_fork_id`] at a fork +/// boundary changes what the table's copy admits as well as the dial loop's. #[derive(Clone)] pub struct LeanFilter { - fork_id: EnrForkId, + policy: Arc>, attestation_committee_count: u64, } +/// The part of admission that moves at a fork boundary. +#[derive(Debug, Clone)] +struct Policy { + /// This node's own `eth2` entry; its digest is always admitted. + fork_id: EnrForkId, + /// Further digests admitted alongside it: while a boundary's subscription + /// window is open, peers are still legitimately on the other side of it. + also_admitted: Vec, +} + impl LeanFilter { pub(crate) fn new(fork_id: EnrForkId, attestation_committee_count: u64) -> Self { Self { - fork_id, + policy: Arc::new(RwLock::new(Policy { + fork_id, + also_admitted: Vec::new(), + })), attestation_committee_count, } } + /// Replace the local `eth2` entry and the further digests admitted beside + /// it. + /// + /// The peer table re-judges a record only when its `seq` rises, so a peer + /// rejected under the old policy stays rejected until it republishes; + /// peers cross a boundary by doing exactly that. + pub(crate) fn set_fork_id(&self, fork_id: EnrForkId, also_admitted: Vec) { + *self.policy.write().expect("admission policy lock poisoned") = Policy { + fork_id, + also_admitted, + }; + } + /// What to dial for a record that has already been admitted, or `None` if it /// would not be. /// @@ -106,13 +136,23 @@ impl LeanFilter { /// unchanged, and the peer table hands out clones: a caller that reaches /// this with an arbitrary record should get nothing to dial, not a panic. pub(crate) fn dial_target(&self, record: &NodeRecord) -> Option { - admit(record, &self.fork_id, self.attestation_committee_count).ok() + self.judge(record).ok() + } + + fn judge(&self, record: &NodeRecord) -> Result { + let policy = self.policy.read().expect("admission policy lock poisoned"); + admit( + record, + &policy.fork_id, + &policy.also_admitted, + self.attestation_committee_count, + ) } } impl PeerFilter for LeanFilter { fn accepts(&self, record: &NodeRecord) -> bool { - admit(record, &self.fork_id, self.attestation_committee_count) + self.judge(record) // The only place a rejection is visible: the peer table records // that the record failed the filter but says nothing about why. .inspect_err(|reason| { @@ -136,6 +176,7 @@ impl PeerFilter for LeanFilter { fn admit( record: &NodeRecord, local: &EnrForkId, + also_admitted: &[ForkDigest], attestation_committee_count: u64, ) -> Result { let pairs = record.pairs(); @@ -144,7 +185,7 @@ fn admit( .ok_or(RejectReason::MissingForkId)?; let remote = EnrForkId::from_ssz_bytes(&raw).map_err(|_| RejectReason::MissingForkId)?; - if remote.fork_digest != local.fork_digest { + if remote.fork_digest != local.fork_digest && !also_admitted.contains(&remote.fork_digest) { return Err(RejectReason::ForkDigestMismatch); } if remote.next_fork_version != local.next_fork_version @@ -333,7 +374,7 @@ mod tests { } fn admit_record(record: &NodeRecord) -> Result { - admit(record, &EnrForkId::local(), TEST_COMMITTEE_COUNT) + admit(record, &EnrForkId::local(), &[], TEST_COMMITTEE_COUNT) } /// Build a record whose pairs are fully controlled, bypassing @@ -575,6 +616,36 @@ mod tests { assert!(filter().dial_target(&record).is_none()); } + #[test] + fn a_fork_boundary_moves_what_every_clone_admits() { + // The peer table owns a clone of the filter, so a boundary has to reach + // it through the shared policy rather than through the dial loop's copy. + let table_copy = filter(); + let dial_copy = table_copy.clone(); + + let mut next = EnrForkId::local(); + next.fork_digest = [0xde, 0xad, 0xbe, 0xef]; + let on_next = record_with(|pairs| { + set_eth2(pairs, next); + set_quic(pairs, 9001); + }); + let on_local = record_with(|pairs| { + set_eth2(pairs, EnrForkId::local()); + set_quic(pairs, 9001); + }); + assert!(!table_copy.accepts(&on_next)); + + // Window open: both sides of the boundary are admitted. + dial_copy.set_fork_id(EnrForkId::local(), vec![next.fork_digest]); + assert!(table_copy.accepts(&on_next)); + assert!(table_copy.accepts(&on_local)); + + // Switched and window closed: only the new digest. + dial_copy.set_fork_id(next, Vec::new()); + assert!(table_copy.accepts(&on_next)); + assert!(!table_copy.accepts(&on_local)); + } + #[test] fn a_missing_quic_port_is_rejected() { // Discoverable, but not over the only transport we speak. The peer can diff --git a/crates/net/p2p/src/discovery/dial.rs b/crates/net/p2p/src/discovery/dial.rs index bd1ef0c4c..229573c26 100644 --- a/crates/net/p2p/src/discovery/dial.rs +++ b/crates/net/p2p/src/discovery/dial.rs @@ -40,6 +40,9 @@ pub(crate) struct DiscoveryState { local_peer_id: PeerId, /// Connected-peer count above which the loop stops dialing. target_peers: usize, + /// A handle on the admission policy the peer table also runs, so a fork + /// boundary can move both with one call. See [`LeanFilter::set_fork_id`]. + filter: LeanFilter, } impl DiscoveryState { @@ -47,14 +50,20 @@ impl DiscoveryState { /// runtime. Every call site builds a `P2PServer` from an async context. pub(crate) fn new(handle: DiscoveryHandle, local_peer_id: PeerId) -> Self { Self { - contacts: spawn_contact_poll(handle.peer_table, handle.filter), + contacts: spawn_contact_poll(handle.peer_table, handle.filter.clone()), candidates: VecDeque::new(), peer_attnets: HashMap::new(), local_peer_id, target_peers: handle.target_peers, + filter: handle.filter, } } + /// The admission policy shared with the peer table. + pub(crate) fn filter(&self) -> &LeanFilter { + &self.filter + } + /// Connected-peer count above which the loop stops dialing. pub(crate) fn target_peers(&self) -> usize { self.target_peers diff --git a/crates/net/p2p/src/discovery/mod.rs b/crates/net/p2p/src/discovery/mod.rs index 6a6e3a09b..4020dc27b 100644 --- a/crates/net/p2p/src/discovery/mod.rs +++ b/crates/net/p2p/src/discovery/mod.rs @@ -259,6 +259,33 @@ pub async fn spawn_discovery( }) } +/// Publish a new `eth2` entry in the ENR this node serves over discv5. +/// +/// Not possible with the discovery stack this node is built on, so this only +/// says so. [`DiscoveryServer::spawn`] takes the local record by value, moves it +/// into an actor it starts itself, and returns `()`: no actor handle comes back, +/// and `DiscoveryServerProtocol` (`raw_packet`, `lookup_*`, `revalidate_*`, +/// `enr_lookup`, `prune`, `shutdown`) has no message that replaces the record. +/// The record is re-signed only internally, by discv5 IP voting +/// (`update_local_ip`), which edits the address and nothing else. What would +/// close the gap is a protocol message on `DiscoveryServer` that edits the +/// local record and bumps its `seq`, or `spawn` returning the actor's handle. +/// +/// Until then the record served to discv5 queries keeps the `eth2` entry it +/// started with. Peers already connected are unaffected (they handshake on +/// `Status`, which does follow the switch) and the admission filter does move, +/// but a peer that discovers this node after a boundary reads the stale digest +/// and rejects the record, so a long-running node becomes harder to find until +/// it is restarted. +pub(crate) fn update_served_enr(fork_id: &EnrForkId) { + warn!( + fork_digest = %hex::encode(fork_id.fork_digest), + next_fork_epoch = fork_id.next_fork_epoch, + "The served ENR keeps its startup eth2 entry: the discovery server offers no way to \ + replace its record at runtime" + ); +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/net/p2p/src/gossipsub/handler.rs b/crates/net/p2p/src/gossipsub/handler.rs index b68422b85..5134ae782 100644 --- a/crates/net/p2p/src/gossipsub/handler.rs +++ b/crates/net/p2p/src/gossipsub/handler.rs @@ -12,7 +12,8 @@ use ethlambda_state_transition::beacon::gossip::{self, IgnoreReason, Outcome, Re use ethlambda_types::{ ShortRoot, attestation::{SignedAggregatedAttestation, SignedAttestation}, - beacon::containers::{SignedBeaconBlock, electra::SingleAttestation}, + beacon::containers::{DataColumnSidecar, SignedBeaconBlock, electra::SingleAttestation}, + beacon::fork::ForkName, beacon::primitives::ValidatorIndex, block::SignedBlock, primitives::HashTreeRoot as _, @@ -99,7 +100,9 @@ pub async fn handle_gossip_message( handle_lean_attestation(server, &payload, compressed_len).await } _ => match beacon_id { - Some(id) => handle_beacon_gossip(server, ctx, id, kind, &payload), + Some(id) => { + handle_beacon_gossip(server, ctx, id, message.topic.as_str(), kind, &payload) + } None => trace!(topic = %message.topic, "Gossip on an unhandled topic"), }, } @@ -224,6 +227,7 @@ fn handle_beacon_gossip( server: &P2PServer, ctx: &Context, id: GossipId, + topic: &str, kind: &str, payload: &[u8], ) { @@ -233,14 +237,24 @@ fn handle_beacon_gossip( debug!(kind, "Beacon gossip arrived on a lean node"); return; }; + // The fork comes from the topic's own digest: while a boundary's window is + // open the node is subscribed under two, and `wire.fork` is only the one it + // publishes under. + let topic_fork = beacon_topics::topic_digest(topic) + .and_then(|digest| wire.schedule.fork_for_digest(digest)) + .unwrap_or(wire.fork); let dispatch = if kind == beacon_topics::BEACON_BLOCK { triage_block(server, wire, payload) } else if let Some(subnet_id) = beacon_topics::data_column_subnet(kind) { - triage_data_column(server, payload, subnet_id) + triage_data_column(server, topic_fork, payload, subnet_id) } else if kind == beacon_topics::BEACON_AGGREGATE_AND_PROOF { triage_aggregate(server, wire, payload, id.received_at) } else if let Some(subnet_id) = beacon_topics::attestation_subnet(kind) { - triage_attestation(server, wire, payload, subnet_id) + triage_attestation(server, topic_fork, payload, subnet_id) + } else if kind == beacon_topics::EXECUTION_PAYLOAD { + triage_envelope(server, payload) + } else if kind == beacon_topics::PAYLOAD_ATTESTATION_MESSAGE { + triage_payload_attestation(server, payload) } else { triage_other(wire, kind, payload) }; @@ -299,32 +313,27 @@ fn triage_block(server: &P2PServer, wire: &BeaconWire, payload: &[u8]) -> Dispat /// Decode a data column sidecar and run its cheap gossip checks. Same shape as /// [`triage_block`]. /// -/// Decodes first, unlike the clock-based gate this used to be: gossipsub -/// topic subscriptions are frozen at startup (`build_swarm` does not -/// resubscribe as a fork boundary is crossed), so a node running across the -/// gloas boundary stays on its fulu-digest topic the whole time, where a -/// late but perfectly legitimate fulu sidecar can still legally arrive. A -/// clock check ahead of the decode would drop that one too, mistaking it for -/// gloas-shaped just because the clock has moved on. Only on a decode -/// failure does the clock matter, and only as an approximation: this node's -/// *topic* fork (whichever one gossip actually subscribed under at startup) -/// is not threaded down to this handler today, so [`beacon_decode::current_fork`] -/// (the wall clock) stands in for it. That is exactly backwards for a node -/// stuck on stale fulu topics past the boundary, the same case this doc -/// opens with: a genuinely malformed fulu sidecar arriving there reads as -/// `Ignore` instead of `Reject`, since the clock alone cannot tell "stale -/// topic, bad bytes" apart from "current topic, gloas-shaped bytes". Safe -/// either way, since `Ignore` never down-scores a peer; a future change that -/// carries the topic's own fork into `BeaconWire` (or wherever else carries -/// the fork digest to this handler) can make this exact instead of merely -/// safe. -fn triage_data_column(server: &P2PServer, payload: &[u8], subnet_id: u64) -> Dispatch { +/// Decodes first rather than gating on the clock: around a fork boundary the +/// node holds topics under two digests (see `beacon::transition`), so a late +/// but perfectly legitimate sidecar can still arrive on the old digest's +/// topic, and a clock check ahead of the decode would drop it, mistaking it for +/// the new fork's shape just because the clock has moved on. `fork` is the one +/// the message's own topic digest names, and it picks the container to decode +/// (fulu's and gloas's differ) and the cheap rules to run. On a decode failure +/// it also decides the verdict: a failure under a fork this build does not +/// follow is its own gap, so `Ignore`; under a followed fork it is the +/// sender's fault, so `Reject`. +fn triage_data_column( + server: &P2PServer, + fork: ForkName, + payload: &[u8], + subnet_id: u64, +) -> Dispatch { const KIND: &str = beacon_topics::DATA_COLUMN_SIDECAR_KIND; - let sidecar = match beacon_decode::decode_data_column_sidecar(payload) { + let sidecar = match beacon_decode::decode_data_column_sidecar(fork, payload) { Ok(sidecar) => sidecar, Err(err) => { - let config = server.store.config(); - if !beacon_decode::current_fork(&config).is_followed() { + if !fork.is_followed() { metrics::inc_beacon_gossip(KIND, "unsupported_fork"); return Dispatch::Report(Outcome::Ignore(IgnoreReason::UnsupportedFork)); } @@ -335,13 +344,23 @@ fn triage_data_column(server: &P2PServer, payload: &[u8], subnet_id: u64) -> Dis }; metrics::inc_beacon_gossip(KIND, "decoded"); let now_ms = unix_now_ms(); - if let Err(outcome) = gossip::column::cheap_checks( - &server.seen_columns, - &server.store, - &sidecar, - subnet_id, - now_ms, - ) { + let cheap = match &sidecar { + DataColumnSidecar::Fulu(sidecar) => gossip::column::cheap_checks( + &server.seen_columns, + &server.store, + sidecar, + subnet_id, + now_ms, + ), + DataColumnSidecar::Gloas(sidecar) => gossip::column::cheap_checks_gloas( + &server.seen_block_columns, + &server.store, + sidecar, + subnet_id, + now_ms, + ), + }; + if let Err(outcome) = cheap { return Dispatch::Report(outcome); } Dispatch::Validate(Validated::Column(Box::new(sidecar))) @@ -359,13 +378,6 @@ fn triage_aggregate( const KIND: &str = beacon_topics::BEACON_AGGREGATE_AND_PROOF; let aggregate = match beacon_decode::decode_aggregate_and_proof(&wire.config, payload) { Ok(aggregate) => aggregate, - // `UnsupportedFork` is not the sender's fault (an honest gloas peer - // sends exactly this once this node's own clock reaches gloas), so it - // must not score like every other decode failure does. - Err(beacon_decode::DecodeError::UnsupportedFork) => { - metrics::inc_beacon_gossip(KIND, "unsupported_fork"); - return Dispatch::Report(Outcome::Ignore(IgnoreReason::UnsupportedFork)); - } Err(err) => { metrics::inc_beacon_gossip(KIND, "decode_failed"); debug!(kind = KIND, %err, bytes = payload.len(), "Beacon gossip decode failed"); @@ -429,19 +441,13 @@ fn triage_aggregate( /// already carries in full. fn triage_attestation( server: &P2PServer, - wire: &BeaconWire, + fork: ForkName, payload: &[u8], subnet_id: u64, ) -> Dispatch { const KIND: &str = beacon_topics::BEACON_ATTESTATION_KIND; - let attestation = match beacon_decode::decode_attestation(wire.fork, payload) { + let attestation = match beacon_decode::decode_attestation(fork, payload) { Ok(attestation) => attestation, - // As on the aggregate topic: an honest peer on a fork this node has no - // attestation rules for must not be scored as a bad decoder. - Err(beacon_decode::DecodeError::UnsupportedFork) => { - metrics::inc_beacon_gossip(KIND, "unsupported_fork"); - return Dispatch::Report(Outcome::Ignore(IgnoreReason::UnsupportedFork)); - } Err(err) => { metrics::inc_beacon_gossip(KIND, "decode_failed"); debug!(kind = KIND, %err, bytes = payload.len(), "Beacon gossip decode failed"); @@ -479,6 +485,65 @@ fn triage_attestation( }) } +/// Decode a gloas execution payload envelope and run its cheap gossip checks. +/// Same shape as [`triage_block`]. +fn triage_envelope(server: &P2PServer, payload: &[u8]) -> Dispatch { + const KIND: &str = beacon_topics::EXECUTION_PAYLOAD; + let envelope = match beacon_decode::decode_execution_payload_envelope(payload) { + Ok(envelope) => envelope, + Err(err) => { + metrics::inc_beacon_gossip(KIND, "decode_failed"); + debug!(kind = KIND, %err, bytes = payload.len(), "Beacon gossip decode failed"); + return Dispatch::Report(Outcome::Reject(RejectReason::Decode)); + } + }; + metrics::inc_beacon_gossip(KIND, "decoded"); + debug!( + slot = envelope.message.payload.slot_number, + builder_index = envelope.message.builder_index, + block_root = %ShortRoot(&envelope.message.beacon_block_root.0), + bytes = payload.len(), + "Beacon execution payload envelope decoded" + ); + if let Err(outcome) = + gossip::envelope::cheap_checks(&server.seen_envelopes, &server.store, &envelope) + { + return Dispatch::Report(outcome); + } + Dispatch::Validate(Validated::Envelope(Box::new(envelope))) +} + +/// Decode a gloas payload attestation message and run its cheap gossip +/// checks. Same shape as [`triage_block`]. +fn triage_payload_attestation(server: &P2PServer, payload: &[u8]) -> Dispatch { + const KIND: &str = beacon_topics::PAYLOAD_ATTESTATION_MESSAGE; + let message = match beacon_decode::decode_payload_attestation_message(payload) { + Ok(message) => message, + Err(err) => { + metrics::inc_beacon_gossip(KIND, "decode_failed"); + debug!(kind = KIND, %err, bytes = payload.len(), "Beacon gossip decode failed"); + return Dispatch::Report(Outcome::Reject(RejectReason::Decode)); + } + }; + metrics::inc_beacon_gossip(KIND, "decoded"); + // `trace` rather than `debug`: the committee votes in one burst per slot. + trace!( + slot = message.data.slot, + validator = message.validator_index, + block_root = %ShortRoot(&message.data.beacon_block_root.0), + "Beacon payload attestation message decoded" + ); + if let Err(outcome) = gossip::payload_attestation::cheap_checks( + &server.seen_payload_attestations, + &server.store, + &message, + unix_now_ms(), + ) { + return Dispatch::Report(outcome); + } + Dispatch::Validate(Validated::PayloadAttestation(message)) +} + /// Decode one of the five beacon topics with nothing particular to report, /// and count it. Ignored rather than validated: none of the five has a /// consumer, so this always answers `Dispatch::Report`. @@ -632,10 +697,11 @@ pub async fn publish_beacon_attestation( error!(%slot, validator, subnet_id, "Attestation subnet out of range; dropping it"); return; } - let topic = IdentTopic::new(beacon_topics::attestation_topic_name( - beacon.fork_digest, - subnet_id, - )); + let Some(digest) = beacon.publish_digest(slot) else { + warn!(%slot, validator, "No held fork digest covers this attestation's slot; not publishing"); + return; + }; + let topic = IdentTopic::new(beacon_topics::attestation_topic_name(digest, subnet_id)); let compressed = compress_message(&attestation.to_ssz()); server.swarm_handle.publish(topic, compressed); debug!( @@ -666,8 +732,12 @@ pub async fn publish_beacon_aggregate( error!(%slot, aggregator, "A beacon aggregate reached a lean node; dropping it"); return; }; + let Some(digest) = beacon.publish_digest(slot) else { + warn!(%slot, aggregator, "No held fork digest covers this aggregate's slot; not publishing"); + return; + }; let topic = IdentTopic::new(beacon_topics::topic_name( - beacon.fork_digest, + digest, beacon_topics::BEACON_AGGREGATE_AND_PROOF, )); // Each fork's container encodes as itself on the wire; the enum is only @@ -679,6 +749,9 @@ pub async fn publish_beacon_aggregate( ethlambda_types::beacon::containers::SignedAggregateAndProof::Electra(signed) => { signed.to_ssz() } + ethlambda_types::beacon::containers::SignedAggregateAndProof::Gloas(signed) => { + signed.to_ssz() + } }; server.swarm_handle.publish(topic, compress_message(&ssz)); debug!(%slot, aggregator, "Published aggregate to gossipsub"); @@ -703,8 +776,15 @@ pub async fn publish_beacon_block(server: &mut P2PServer, block: SignedBeaconBlo error!(slot, "A beacon block reached a lean node; dropping it"); return; }; + let Some(digest) = beacon.publish_digest(slot) else { + warn!( + slot, + "No held fork digest covers this block's slot; not publishing" + ); + return; + }; let topic = IdentTopic::new(beacon_topics::topic_name( - beacon.fork_digest, + digest, beacon_topics::BEACON_BLOCK, )); server @@ -760,9 +840,13 @@ pub fn join_aggregator_subnets(server: &mut P2PServer, subnets: Vec<(u64, u64)>) }); *until = (*until).max(slot); } + // Under every digest the node holds: while a boundary's window is open, + // attestations are published on either side of it. for &subnet_id in &joined { - let topic = beacon_topics::attestation_topic_name(wire.fork_digest, subnet_id); - server.swarm_handle.subscribe(IdentTopic::new(topic)); + for held in wire.held_topics() { + let topic = beacon_topics::attestation_topic_name(held.fork_digest, subnet_id); + server.swarm_handle.subscribe(IdentTopic::new(topic)); + } } if !joined.is_empty() { info!(?joined, "Joined attestation subnets for aggregation"); @@ -799,8 +883,10 @@ pub fn leave_expired_aggregator_subnets(server: &mut P2PServer) { .collect(); for subnet_id in &expired { server.aggregator_subnets.remove(subnet_id); - let topic = beacon_topics::attestation_topic_name(wire.fork_digest, *subnet_id); - server.swarm_handle.unsubscribe(IdentTopic::new(topic)); + for held in wire.held_topics() { + let topic = beacon_topics::attestation_topic_name(held.fork_digest, *subnet_id); + server.swarm_handle.unsubscribe(IdentTopic::new(topic)); + } } if !expired.is_empty() { debug!( @@ -813,7 +899,7 @@ pub fn leave_expired_aggregator_subnets(server: &mut P2PServer) { #[cfg(test)] mod tests { use ethlambda_types::beacon::config::Config; - use ethlambda_types::beacon::containers::{AttestationData, electra, phase0, shared}; + use ethlambda_types::beacon::containers::{AttestationData, electra, gloas, phase0, shared}; use ethlambda_types::beacon::fork::ForkName; use ethlambda_types::beacon::preset; use ethlambda_types::beacon::primitives::Slot; @@ -925,14 +1011,14 @@ mod tests { } #[tokio::test] - async fn an_undecodable_block_at_a_gloas_slot_is_ignored() { + async fn an_undecodable_block_at_a_gloas_slot_is_rejected() { let mut config = Config::mainnet(); config.gloas_fork_epoch = config.fulu_fork_epoch + 1; let slot = config.gloas_fork_epoch * preset::SLOTS_PER_EPOCH; assert!(matches!( triage_block_under(config, &mismatched_block_bytes(slot)).await, - Dispatch::Report(Outcome::Ignore(IgnoreReason::UnsupportedFork)) + Dispatch::Report(Outcome::Reject(RejectReason::Decode)) )); } @@ -963,7 +1049,7 @@ mod tests { let server = unconnected_beacon_server(Config::mainnet(), 0).await; assert!(matches!( - triage_data_column(&server, &[0xff; 3], 0), + triage_data_column(&server, ForkName::Fulu, &[0xff; 3], 0), Dispatch::Report(Outcome::Reject(RejectReason::Decode)) )); } @@ -978,7 +1064,7 @@ mod tests { let payload = sidecar.to_ssz(); assert!(matches!( - triage_data_column(&server, &payload, 0), + triage_data_column(&server, ForkName::Fulu, &payload, 0), Dispatch::Report(Outcome::Ignore(IgnoreReason::FutureSlot)) )); } @@ -999,11 +1085,44 @@ mod tests { let payload = sidecar.to_ssz(); assert!(matches!( - triage_data_column(&server, &payload, 0), + triage_data_column(&server, ForkName::Fulu, &payload, 0), Dispatch::Validate(Validated::Column(_)) )); } + #[tokio::test] + async fn a_gloas_sidecar_decodes_by_its_topics_fork_and_is_judged_by_gloas_rules() { + let now_ms = unix_now_ms(); + let config = Config { + genesis_time: now_ms / 1_000 - 5, + slot_duration_ms: 1_000, + ..Config::mainnet() + }; + let server = unconnected_beacon_server(config, 0).await; + let sidecar = ethlambda_types::beacon::containers::gloas::DataColumnSidecar { + index: 0, + slot: 4, + ..Default::default() + }; + let payload = sidecar.to_ssz(); + + assert!(matches!( + triage_data_column(&server, ForkName::Gloas, &payload, 0), + Dispatch::Validate(Validated::Column(_)) + )); + // Gloas's rule has no header to read a proposer from, but it does have + // a subnet rule. + assert!(matches!( + triage_data_column(&server, ForkName::Gloas, &payload, 1), + Dispatch::Report(Outcome::Reject(RejectReason::WrongSubnet)) + )); + // The same bytes under fulu's topic are not a fulu sidecar. + assert!(matches!( + triage_data_column(&server, ForkName::Fulu, &payload, 0), + Dispatch::Report(Outcome::Reject(RejectReason::Decode)) + )); + } + #[tokio::test] async fn the_same_sidecar_asked_for_on_the_wrong_subnet_is_rejected() { let now_ms = unix_now_ms(); @@ -1017,7 +1136,7 @@ mod tests { let payload = sidecar.to_ssz(); assert!(matches!( - triage_data_column(&server, &payload, 1), + triage_data_column(&server, ForkName::Fulu, &payload, 1), Dispatch::Report(Outcome::Reject(RejectReason::WrongSubnet)) )); } @@ -1054,11 +1173,11 @@ mod tests { )); } - /// Gloas's aggregate has no modeled container here, and an honest gloas - /// peer sends exactly this once the clock reaches the fork, so the decode - /// error must not become a `Reject`, which would penalize it. + /// Gloas repurposes `data.index` as the payload flag, so an honest `1` + /// passes the index rule electra would reject it on; it is turned away + /// later, here by the empty `committee_bits` of the test message. #[tokio::test] - async fn a_gloas_aggregate_is_ignored_as_an_unsupported_fork() { + async fn a_gloas_aggregate_with_a_payload_flag_passes_the_index_rule() { let config = Config::mainnet().with_fork_epoch(ForkName::Gloas, 0); let server = unconnected_beacon_server(config, 0).await; let wire = server @@ -1069,7 +1188,23 @@ mod tests { assert!(matches!( triage_aggregate(&server, wire, &payload, Instant::now()), - Dispatch::Report(Outcome::Ignore(IgnoreReason::UnsupportedFork)) + Dispatch::Report(Outcome::Reject(RejectReason::CommitteeBits)) + )); + } + + #[tokio::test] + async fn a_gloas_aggregate_with_a_data_index_above_one_is_rejected() { + let config = Config::mainnet().with_fork_epoch(ForkName::Gloas, 0); + let server = unconnected_beacon_server(config, 0).await; + let wire = server + .wire + .beacon() + .expect("a beacon server has a beacon wire"); + let payload = electra_aggregate(4, 2).to_ssz(); + + assert!(matches!( + triage_aggregate(&server, wire, &payload, Instant::now()), + Dispatch::Report(Outcome::Reject(RejectReason::DataIndexOutOfRange)) )); } @@ -1082,7 +1217,7 @@ mod tests { .expect("a beacon server has a beacon wire"); assert!(matches!( - triage_attestation(&server, wire, &[0xff; 3], 0), + triage_attestation(&server, wire.fork, &[0xff; 3], 0), Dispatch::Report(Outcome::Reject(RejectReason::Decode)) )); } @@ -1103,7 +1238,7 @@ mod tests { let payload = phase0_attestation(4).to_ssz(); assert!(matches!( - triage_attestation(&server, wire, &payload, 0), + triage_attestation(&server, wire.fork, &payload, 0), Dispatch::Report(Outcome::Ignore(IgnoreReason::NoConsumer)) )); } @@ -1120,27 +1255,34 @@ mod tests { let payload = electra_single_attestation(4, 1).to_ssz(); assert!(matches!( - triage_attestation(&server, wire, &payload, 0), + triage_attestation(&server, wire.fork, &payload, 0), Dispatch::Report(Outcome::Reject(RejectReason::NonZeroDataIndex)) )); } /// A gloas `SingleAttestation` has electra's bytes, but its `data.index` - /// is the payload flag, so an honest `1` must not reach electra's rules - /// and be rejected as a nonzero index. + /// is the payload flag: an honest `1` must not be rejected as a nonzero + /// index, while a value past the flag's range is. #[tokio::test] - async fn a_gloas_attestation_is_ignored_as_an_unsupported_fork() { + async fn a_gloas_attestation_is_judged_on_the_payload_flag_range() { let config = Config::mainnet().with_fork_epoch(ForkName::Gloas, 0); let server = unconnected_beacon_server(config, 0).await; let wire = server .wire .beacon() .expect("a beacon server has a beacon wire"); - let payload = electra_single_attestation(4, 1).to_ssz(); + let flagged = electra_single_attestation(4, 1).to_ssz(); + // Past the index rule, and turned away by the clock instead: slot 4 is + // long outside the epoch window of a mainnet genesis. + assert!(matches!( + triage_attestation(&server, wire.fork, &flagged, 0), + Dispatch::Report(Outcome::Ignore(IgnoreReason::OutsideEpochWindow)) + )); + let out_of_range = electra_single_attestation(4, 2).to_ssz(); assert!(matches!( - triage_attestation(&server, wire, &payload, 0), - Dispatch::Report(Outcome::Ignore(IgnoreReason::UnsupportedFork)) + triage_attestation(&server, wire.fork, &out_of_range, 0), + Dispatch::Report(Outcome::Reject(RejectReason::DataIndexOutOfRange)) )); } @@ -1186,6 +1328,12 @@ mod tests { #[tokio::test] async fn a_published_aggregate_is_handed_to_the_chain_with_its_indices() { let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; + // The test server's digest is a placeholder, and an aggregate whose + // slot no held digest covers is neither published nor handed on. + let crate::Wire::Beacon(beacon) = &mut server.wire else { + unreachable!("unconnected_beacon_server builds a beacon wire"); + }; + beacon.topics.fork_digest = beacon.digest_for_slot(5); let chain = std::sync::Arc::new(RecordingChain::default()); server.blockchain = Some(chain.clone()); let aggregate = ethlambda_types::beacon::containers::SignedAggregateAndProof::Electra( @@ -1196,4 +1344,78 @@ mod tests { assert_eq!(*chain.aggregates.lock().unwrap(), [(aggregate, vec![3, 4])]); } + + /// A payload that does not decode as a gloas envelope is the sender's + /// fault, on either gloas topic. + #[tokio::test] + async fn an_undecodable_gloas_message_is_rejected() { + let server = unconnected_beacon_server(Config::mainnet(), 0).await; + + assert!(matches!( + triage_envelope(&server, &[0xff; 3]), + Dispatch::Report(Outcome::Reject(RejectReason::Decode)) + )); + assert!(matches!( + triage_payload_attestation(&server, &[0xff; 3]), + Dispatch::Report(Outcome::Reject(RejectReason::Decode)) + )); + } + + /// An envelope that clears the cheap checks goes on to the stateful ones, + /// and a second one for a key the seen cache holds is ignored before them. + #[tokio::test] + async fn an_envelope_passes_triage_unless_its_key_was_seen() { + let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; + let envelope = crate::test_support::envelope(1, 3); + let payload = envelope.to_ssz(); + + assert!(matches!( + triage_envelope(&server, &payload), + Dispatch::Validate(Validated::Envelope(decoded)) if *decoded == envelope + )); + + server.seen_envelopes.record( + envelope.message.beacon_block_root, + envelope.message.builder_index, + ); + assert!(matches!( + triage_envelope(&server, &payload), + Dispatch::Report(Outcome::Ignore(IgnoreReason::AlreadySeen)) + )); + } + + /// A payload attestation for a slot before gloas is rejected by the cheap + /// checks; mainnet's gloas epoch is not zero, so slot zero is before it. + #[tokio::test] + async fn a_payload_attestation_before_gloas_is_rejected() { + let server = unconnected_beacon_server(Config::mainnet(), 0).await; + let message = gloas::PayloadAttestationMessage { + validator_index: 1, + data: Default::default(), + signature: Default::default(), + }; + + assert!(matches!( + triage_payload_attestation(&server, &message.to_ssz()), + Dispatch::Report(Outcome::Reject(RejectReason::PreGloasSlot)) + )); + } + + /// A payload attestation inside gloas but outside the current slot is + /// ignored, and one in the current slot goes on to the stateful checks. + #[tokio::test] + async fn a_payload_attestation_is_judged_on_the_clock() { + let config = Config::mainnet().with_fork_epoch(ForkName::Gloas, 0); + let server = unconnected_beacon_server(config, 0).await; + let mut message = gloas::PayloadAttestationMessage { + validator_index: 1, + data: Default::default(), + signature: Default::default(), + }; + message.data.slot = 1_000_000; + assert!(matches!( + triage_payload_attestation(&server, &message.to_ssz()), + Dispatch::Report(Outcome::Ignore(IgnoreReason::NotCurrentSlot)) + )); + } } diff --git a/crates/net/p2p/src/lib.rs b/crates/net/p2p/src/lib.rs index c2d79514e..75922bbc7 100644 --- a/crates/net/p2p/src/lib.rs +++ b/crates/net/p2p/src/lib.rs @@ -51,7 +51,9 @@ use ethlambda_network_api::{ use ethlambda_state_transition::beacon::aggregate::MAX_AGGREGATES_PER_SLOT; use ethlambda_state_transition::beacon::attestation_pool::SharedAttestationPool; use ethlambda_state_transition::beacon::gossip::{ - SeenBlocks, SeenColumns, aggregate::SeenAggregates, attestation::SeenAttestations, + SeenBlockColumns, SeenBlocks, SeenColumns, aggregate::SeenAggregates, + attestation::SeenAttestations, envelope::SeenEnvelopes, + payload_attestation::SeenPayloadAttestations, }; use ethlambda_storage::{Chain, Store}; use ethlambda_types::beacon::preset::{MAX_VALIDATORS_PER_COMMITTEE, SLOTS_PER_EPOCH}; @@ -92,7 +94,7 @@ use crate::{ lean::protocols::MAX_REQUEST_BLOCKS, req_resp::{ Codec, MAX_COMPRESSED_PAYLOAD_SIZE, ReqResp, ReqRespEvent, Request, build_status, - fetch_block_from_peer, fetch_data_columns_from_peer, + fetch_block_from_peer, fetch_data_columns_from_peer, fetch_envelope_from_peer, handlers::{columns_custodied_by, resume_range_batch_held_for_custody}, }, swarm_adapter::SwarmHandle, @@ -250,6 +252,17 @@ const SEEN_BLOCKS_CAPACITY: NonZeroUsize = NonZeroUsize::new(1024).expect("non-z /// Capacity of the first-valid-sidecar cache, keyed by `(slot, proposer, index)`. const SEEN_COLUMNS_CAPACITY: NonZeroUsize = NonZeroUsize::new(4096).expect("non-zero"); +/// Capacity of the first-valid-envelope cache, keyed by `(block root, +/// builder index)`. One block a slot carries one envelope, so this is sized +/// like [`SEEN_BLOCKS_CAPACITY`]. +const SEEN_ENVELOPES_CAPACITY: NonZeroUsize = NonZeroUsize::new(1024).expect("non-zero"); + +/// Capacity of the first-valid-payload-attestation cache, keyed by `(slot, +/// validator index)`. The rule only asks about the current slot, so one +/// committee's worth of votes for a few slots is generous headroom, in the +/// spirit of [`SEEN_COLUMNS_CAPACITY`]. +const SEEN_PAYLOAD_ATTESTATIONS_CAPACITY: NonZeroUsize = NonZeroUsize::new(4096).expect("non-zero"); + /// How many `(target_epoch, aggregator_index)` pairs the accepted-aggregate /// cache remembers, and how many `(hash_tree_root(data), committee_index)` /// bitfields alongside it (`SeenAggregates::new`'s two capacities, both sized @@ -335,10 +348,12 @@ pub(crate) enum PendingRequestKind { end_slot: u64, }, /// A `DataColumnsByRoot` lookup for this block's missing columns. Carries - /// only the root: the columns, attempts and failed-peer set live in - /// `pending_column_requests`, keyed the same way, so this is enough to - /// route the response and nothing this map needs to duplicate. - Columns(H256), + /// the root and the columns this one request asked its peer for: the + /// attempts and failed-peer set live in `pending_column_requests`, keyed + /// by the root, but one lookup fans out across peers with a subset each and + /// retires its entry on the first answer, so what a later answer may + /// contain has to travel with the request itself. + Columns(H256, Vec), /// A `DataColumnsByRange` sweep for a range sync batch, covering the same /// slots the matching `BlocksByRange` asked for. /// @@ -351,6 +366,20 @@ pub(crate) enum PendingRequestKind { start_slot: u64, end_slot: u64, }, + /// An `ExecutionPayloadEnvelopesByRoot` lookup for this block root's + /// envelope. Its attempts and failed peers live in + /// `pending_envelope_requests`, keyed by the same root. + EnvelopeRoot(H256), + /// An `ExecutionPayloadEnvelopesByRange` sweep for a range sync batch, + /// asked for the slots a `BeaconBlocksByRange` answer just delivered. + /// + /// Like [`Self::ColumnRange`] it carries no per-root bookkeeping: a short + /// or empty answer is not a failure to retry, since the chain actor asks + /// again by root for every parent still missing its envelope. + EnvelopeRange { + start_slot: u64, + end_slot: u64, + }, } /// Which single-protocol field of [`ReqResp`] an outbound request travels @@ -389,6 +418,8 @@ pub enum ReqRespProtocol { BeaconBlocksByRoot, DataColumnSidecarsByRange, DataColumnSidecarsByRoot, + ExecutionPayloadEnvelopesByRange, + ExecutionPayloadEnvelopesByRoot, } /// An outbound request id, namespaced by the protocol it was sent on. @@ -408,6 +439,14 @@ pub(crate) struct RangeSyncState { /// Latest advertised head slot for each peer. pub(crate) peer_set: HashMap, pub(crate) in_flight: bool, + /// Set while the envelopes of the batch just delivered are being fetched + /// and checked, which only gloas spans have. + /// + /// The chain actor holds an envelope whose block has not imported only + /// briefly and in small numbers, and imports a batch in order, so the next + /// batch's blocks must not reach it before this batch's envelopes have. + /// Holding the next request back on the answer is what guarantees that. + pub(crate) envelopes_pending: bool, /// When the next batch was first held back for custody, while it still is. /// Beacon-only; see [`RANGE_BATCH_CUSTODY_WAIT`]. pub(crate) custody_wait_since: Option, @@ -432,6 +471,7 @@ impl RangeSyncState { current_range, peer_set: HashMap::from([(peer, peer_head)]), in_flight: false, + envelopes_pending: false, custody_wait_since: None, } } @@ -476,7 +516,7 @@ impl RangeSyncState { } pub(crate) fn next_batch(&self) -> Option<(PeerId, Range)> { - if self.in_flight || self.current_range.is_empty() { + if self.in_flight || self.envelopes_pending || self.current_range.is_empty() { return None; } @@ -972,14 +1012,9 @@ pub fn build_swarm(config: SwarmConfig) -> Result { // `column % DATA_COLUMN_SIDECAR_SUBNET_COUNT`, computed rather than // assumed so a network that ever separates the two counts still // subscribes to the right topic. - let column_subnets: Vec = beacon - .custody_columns - .iter() - .map(|column| { - column % ethlambda_types::beacon::constants::DATA_COLUMN_SIDECAR_SUBNET_COUNT - }) - .collect(); - let topics = beacon::topics::BeaconTopics::new( + let column_subnets = beacon::topics::column_subnets(&beacon.custody_columns); + let topics = beacon::topics::BeaconTopics::for_fork( + beacon.fork, beacon.fork_digest, &column_subnets, &beacon.attestation_subnets, @@ -998,10 +1033,21 @@ pub fn build_swarm(config: SwarmConfig) -> Result { "Beacon P2P node started" ); + // Only the current digest is joined here. The rest of the window + // around a boundary (the next digest's topics, or the previous + // one's if the node started just after one) is joined by the first + // `AdvanceForkSchedule`, which `P2P::spawn` sends at once, so that + // startup and a live crossing run one code path. + let schedule = beacon::fork_schedule::ForkSchedule::new( + &beacon.config, + beacon.genesis_validators_root, + ); Wire::Beacon(Box::new(beacon::BeaconWire { fork_digest: beacon.fork_digest, fork: beacon.fork, + schedule, topics, + window_topics: Vec::new(), config: beacon.config, genesis_time: beacon.genesis_time, genesis_validators_root: beacon.genesis_validators_root, @@ -1080,6 +1126,7 @@ impl P2P { connected_peers: HashMap::new(), peer_custody: HashMap::new(), pending_root_requests: HashMap::new(), + pending_envelope_requests: HashMap::new(), pending_column_requests: HashMap::new(), outbound_requests: HashMap::new(), range_sync_state: None, @@ -1089,6 +1136,11 @@ impl P2P { discovery: discovery.map(|handle| DiscoveryState::new(handle, built.local_peer_id)), seen_blocks: SeenBlocks::new(SEEN_BLOCKS_CAPACITY), seen_columns: SeenColumns::new(SEEN_COLUMNS_CAPACITY), + seen_block_columns: SeenBlockColumns::new(SEEN_COLUMNS_CAPACITY), + seen_envelopes: SeenEnvelopes::new(SEEN_ENVELOPES_CAPACITY), + seen_payload_attestations: SeenPayloadAttestations::new( + SEEN_PAYLOAD_ATTESTATIONS_CAPACITY, + ), seen_aggregates: SeenAggregates::new( SEEN_AGGREGATES_CAPACITY, SEEN_AGGREGATES_CAPACITY, @@ -1122,6 +1174,12 @@ impl P2P { p2p_protocol::DiscoverPeers, ); } + // Immediate on a beacon wire: joins the rest of a window the node + // started inside, then keeps itself scheduled. A no-op on lean. + let _ = handle + .context() + .send(p2p_protocol::AdvanceForkSchedule) + .inspect_err(|err| warn!(%err, "Could not schedule the fork schedule")); spawn_listener(handle.context(), swarm_stream.map(WrappedSwarmEvent)); Ok(P2P { handle }) } @@ -1174,6 +1232,11 @@ pub struct P2PServer { pub(crate) peer_custody: HashMap>, pub(crate) pending_root_requests: HashMap, /// One entry per block root with an in-flight or backed-off + /// `ExecutionPayloadEnvelopesByRoot` lookup: `pending_root_requests`'s + /// counterpart for envelopes, deduplicating a repeated ask and carrying + /// the retry ladder. + pub(crate) pending_envelope_requests: HashMap, + /// One entry per block root with an in-flight or backed-off /// `DataColumnsByRoot` lookup. Mirrors `pending_root_requests`'s role for /// the block path: `fetch_missing_columns` dedupes against it, and /// `handle_column_fetch_failure` is the only place an entry is retired. @@ -1197,6 +1260,15 @@ pub struct P2PServer { /// The first valid sidecar per `(slot, proposer, index)` accepted from /// gossip. Bounded by capacity, so a fabricated slot cannot grow it. pub(crate) seen_columns: SeenColumns, + /// Gloas's counterpart of [`Self::seen_columns`]: the first valid sidecar + /// per `(block root, index)`, since a gloas sidecar names no proposer. + pub(crate) seen_block_columns: SeenBlockColumns, + /// The first valid `execution_payload` per `(block root, builder index)` + /// accepted from gossip. + pub(crate) seen_envelopes: SeenEnvelopes, + /// The first valid `payload_attestation_message` per `(slot, validator + /// index)` accepted from gossip. + pub(crate) seen_payload_attestations: SeenPayloadAttestations, /// Accepted `beacon_aggregate_and_proof`s, by `(target_epoch, /// aggregator_index)` and by `(hash_tree_root(data), committee_index)`. pub(crate) seen_aggregates: SeenAggregates, @@ -1303,6 +1375,10 @@ pub(crate) trait P2PProtocol: Send + Sync { #[allow(dead_code)] // invoked via send_after, not called directly fn retry_data_column_fetch(&self, block_root: H256) -> Result<(), ActorError>; #[allow(dead_code)] // invoked via send_after, not called directly + fn retry_envelope_fetch(&self, block_root: H256) -> Result<(), ActorError>; + #[allow(dead_code)] // invoked via send_after, not called directly + fn resume_range_after_envelopes(&self) -> Result<(), ActorError>; + #[allow(dead_code)] // invoked via send_after, not called directly fn retry_peer_redial(&self, peer_id: PeerId) -> Result<(), ActorError>; #[allow(dead_code)] // invoked via send_after, not called directly fn discover_peers(&self) -> Result<(), ActorError>; @@ -1310,6 +1386,10 @@ pub(crate) trait P2PProtocol: Send + Sync { fn leave_expired_aggregator_subnets(&self) -> Result<(), ActorError>; #[allow(dead_code)] // invoked via send_after, not called directly fn retry_beacon_range_batch(&self) -> Result<(), ActorError>; + /// Apply the fork schedule at the wall-clock epoch. Sent once at startup and + /// then by its own timer for the next join, switch or leave. + #[allow(dead_code)] // invoked via send_after, not called directly + fn advance_fork_schedule(&self) -> Result<(), ActorError>; } #[actor(protocol = P2PProtocol)] @@ -1335,6 +1415,35 @@ impl P2PServer { } } + #[send_handler] + async fn handle_resume_range_after_envelopes( + &mut self, + _msg: p2p_protocol::ResumeRangeAfterEnvelopes, + ctx: &Context, + ) { + req_resp::handlers::release_envelope_gate(self, ctx).await; + } + + #[send_handler] + async fn handle_retry_envelope_fetch( + &mut self, + msg: p2p_protocol::RetryEnvelopeFetch, + _ctx: &Context, + ) { + let block_root = msg.block_root; + // Same "might have completed during backoff" guard as + // `handle_retry_block_fetch`. + if !self.pending_envelope_requests.contains_key(&block_root) { + trace!(%block_root, "Envelope fetch completed during backoff, skipping retry"); + return; + } + trace!(%block_root, "Retrying envelope fetch after backoff"); + if !fetch_envelope_from_peer(self, block_root).await { + tracing::error!(%block_root, "Failed to retry envelope fetch, giving up"); + self.pending_envelope_requests.remove(&block_root); + } + } + #[send_handler] async fn handle_retry_data_column_fetch( &mut self, @@ -1394,6 +1503,17 @@ impl P2PServer { gossipsub::prune_attestation_pool(self); } + #[send_handler] + async fn handle_advance_fork_schedule( + &mut self, + _msg: p2p_protocol::AdvanceForkSchedule, + ctx: &Context, + ) { + if let Some(delay) = beacon::transition::advance(self) { + send_after(delay, ctx.clone(), p2p_protocol::AdvanceForkSchedule); + } + } + #[send_handler] async fn handle_discover_peers( &mut self, @@ -1512,8 +1632,12 @@ async fn fetch_missing(server: &mut P2PServer, request: FetchRequest) { let FetchRequest { block_root, needs_block, + needs_envelope, columns, } = request; + if needs_envelope { + fetch_missing_envelope(server, block_root).await; + } if needs_block { fetch_missing_block(server, block_root).await; } @@ -1532,6 +1656,17 @@ async fn fetch_missing_block(server: &mut P2PServer, root: H256) { fetch_block_from_peer(server, root).await; } +/// The by-root envelope half of a [`FetchRequest`]. +async fn fetch_missing_envelope(server: &mut P2PServer, block_root: H256) { + // The chain actor re-asks once per slot for every parent still missing its + // envelope, so a root already in flight is the normal case, not a fault. + if server.pending_envelope_requests.contains_key(&block_root) { + trace!(%block_root, "Envelope fetch already in progress, ignoring duplicate"); + return; + } + fetch_envelope_from_peer(server, block_root).await; +} + /// The by-root column half of a [`FetchRequest`]. async fn fetch_missing_columns(server: &mut P2PServer, block_root: H256, columns: Vec) { // Same one-lookup-per-root rule as the block half, but merged rather @@ -1876,6 +2011,12 @@ async fn handle_behaviour_event( ReqRespEvent::DataColumnSidecarsByRoot(e) => { (ReqRespProtocol::DataColumnSidecarsByRoot, e) } + ReqRespEvent::ExecutionPayloadEnvelopesByRange(e) => { + (ReqRespProtocol::ExecutionPayloadEnvelopesByRange, e) + } + ReqRespEvent::ExecutionPayloadEnvelopesByRoot(e) => { + (ReqRespProtocol::ExecutionPayloadEnvelopesByRoot, e) + } }, }; @@ -2417,6 +2558,21 @@ fn compute_message_id(message: &libp2p::gossipsub::Message) -> libp2p::gossipsub /// keeps its own lean-flavored copy, since that one builds a different wire. #[cfg(test)] pub(crate) mod test_support { + /// Keep a test swarm's event stream consumed. The swarm task exits on + /// its first event once nothing receives them, which takes the command + /// channel down with it and leaves `send_request` racing a dead task. + pub(crate) fn drain_swarm_events( + stream: impl futures::Stream> + + Send + + 'static, + ) { + use futures::StreamExt as _; + tokio::spawn(async move { + let mut stream = Box::pin(stream); + while stream.next().await.is_some() {} + }); + } + use std::collections::{HashMap, HashSet}; use std::net::{IpAddr, Ipv4Addr}; use std::sync::Arc; @@ -2474,13 +2630,13 @@ pub(crate) mod test_support { } fn new_data_column_sidecars( &self, - _sidecars: Vec, + _sidecars: Vec, ) -> Result<(), spawned_concurrency::error::ActorError> { Ok(()) } fn data_column_sidecars_awaiting_parent( &self, - _sidecars: Vec, + _sidecars: Vec, ) -> Result<(), spawned_concurrency::error::ActorError> { Ok(()) } @@ -2496,6 +2652,67 @@ pub(crate) mod test_support { .push((*aggregate, attesting_indices)); Ok(()) } + fn new_execution_payload_envelope( + &self, + _envelope: Box< + ethlambda_types::beacon::containers::gloas::SignedExecutionPayloadEnvelope, + >, + _arrival: ethlambda_network_api::BlockArrival, + ) -> Result<(), spawned_concurrency::error::ActorError> { + Ok(()) + } + fn new_payload_attestation_message( + &self, + _message: ethlambda_types::beacon::containers::gloas::PayloadAttestationMessage, + _arrival: ethlambda_network_api::BlockArrival, + ) -> Result<(), spawned_concurrency::error::ActorError> { + Ok(()) + } + } + + /// A gloas envelope for `(root_byte, builder)` at slot `slot_number` + /// zero. Only what the cheap gossip checks and the seen cache read is + /// meaningful; nothing here is signature-valid. + pub(crate) fn envelope( + root_byte: u8, + builder_index: u64, + ) -> ethlambda_types::beacon::containers::gloas::SignedExecutionPayloadEnvelope { + use ethlambda_types::beacon::containers::{bellatrix, gloas}; + use ethlambda_types::beacon::primitives::{ + Bytes32, ExecutionAddress, ExecutionBlockHash, Uint256, + }; + let payload = gloas::ExecutionPayload { + parent_hash: ExecutionBlockHash::ZERO, + fee_recipient: ExecutionAddress::ZERO, + state_root: Bytes32::ZERO, + receipts_root: Bytes32::ZERO, + logs_bloom: bellatrix::LogsBloom::try_from(vec![0u8; preset::BYTES_PER_LOGS_BLOOM]) + .expect("built at exactly BYTES_PER_LOGS_BLOOM"), + prev_randao: Bytes32::ZERO, + block_number: 0, + gas_limit: 0, + gas_used: 0, + timestamp: 0, + extra_data: Default::default(), + base_fee_per_gas: Uint256::ZERO, + block_hash: ExecutionBlockHash::ZERO, + transactions: Default::default(), + withdrawals: Default::default(), + blob_gas_used: 0, + excess_blob_gas: 0, + block_access_list: Default::default(), + slot_number: 0, + }; + gloas::SignedExecutionPayloadEnvelope { + message: gloas::ExecutionPayloadEnvelope { + payload, + execution_requests: Default::default(), + builder_index, + beacon_block_root: Root::repeat_byte(root_byte), + parent_beacon_block_root: Root::ZERO, + }, + signature: Default::default(), + } } /// A real, unconnected beacon `P2PServer`, built the same way @@ -2528,8 +2745,9 @@ pub(crate) mod test_support { }) .expect("swarm builds"); - let (_swarm_stream, swarm_handle) = + let (swarm_stream, swarm_handle) = crate::swarm_adapter::start_swarm_adapter(built.swarm, HashMap::new()); + drain_swarm_events(swarm_stream); let discovery = crate::discovery::spawn_discovery(crate::discovery::DiscoverySpawnConfig { node_key: secp256k1::SecretKey::new(&mut rand::rngs::OsRng) @@ -2582,6 +2800,7 @@ pub(crate) mod test_support { connected_peers: HashMap::new(), peer_custody: HashMap::new(), pending_root_requests: HashMap::new(), + pending_envelope_requests: HashMap::new(), pending_column_requests: HashMap::new(), outbound_requests: HashMap::new(), range_sync_state: None, @@ -2598,6 +2817,16 @@ pub(crate) mod test_support { seen_columns: ethlambda_state_transition::beacon::gossip::SeenColumns::new( crate::SEEN_COLUMNS_CAPACITY, ), + seen_block_columns: ethlambda_state_transition::beacon::gossip::SeenBlockColumns::new( + crate::SEEN_COLUMNS_CAPACITY, + ), + seen_envelopes: ethlambda_state_transition::beacon::gossip::envelope::SeenEnvelopes::new( + crate::SEEN_ENVELOPES_CAPACITY, + ), + seen_payload_attestations: + ethlambda_state_transition::beacon::gossip::payload_attestation::SeenPayloadAttestations::new( + crate::SEEN_PAYLOAD_ATTESTATIONS_CAPACITY, + ), seen_aggregates: ethlambda_state_transition::beacon::gossip::aggregate::SeenAggregates::new( crate::SEEN_AGGREGATES_CAPACITY, @@ -3008,6 +3237,12 @@ mod tests { ReqRespEvent::DataColumnSidecarsByRoot(e) => { (ReqRespProtocol::DataColumnSidecarsByRoot, e) } + ReqRespEvent::ExecutionPayloadEnvelopesByRange(e) => { + (ReqRespProtocol::ExecutionPayloadEnvelopesByRange, e) + } + ReqRespEvent::ExecutionPayloadEnvelopesByRoot(e) => { + (ReqRespProtocol::ExecutionPayloadEnvelopesByRoot, e) + } }) } diff --git a/crates/net/p2p/src/metrics.rs b/crates/net/p2p/src/metrics.rs index c56f21bbc..a9779c211 100644 --- a/crates/net/p2p/src/metrics.rs +++ b/crates/net/p2p/src/metrics.rs @@ -362,7 +362,7 @@ static LEAN_BEACON_STATUS_DIGEST_MISMATCH_TOTAL: LazyLock = LazyLock static LEAN_BEACON_FORK_DIGEST: LazyLock = LazyLock::new(|| { register_int_gauge_vec!( "lean_beacon_fork_digest", - "The fork digest this node computed at startup, as a label", + "The fork digest this node currently publishes under, as a label", &["digest"] ) .unwrap() @@ -478,10 +478,19 @@ pub fn inc_beacon_status_digest_mismatch() { LEAN_BEACON_STATUS_DIGEST_MISMATCH_TOTAL.inc(); } -/// Publish the computed fork digest as a label, so a dashboard can tell at a +/// Publish the current fork digest as a label, so a dashboard can tell at a /// glance whether a node is stranded on a boundary it failed to cross. +/// +/// Resets the vector first: the digest changes at runtime, and a label left +/// behind at 1 would read as two current digests. pub fn set_beacon_fork_digest(digest: &str) { - LEAN_BEACON_FORK_DIGEST.with_label_values(&[digest]).set(1); + set_only_label(&LEAN_BEACON_FORK_DIGEST, digest); +} + +/// Make `label` the one series of `gauge` that reads 1. +fn set_only_label(gauge: &IntGaugeVec, label: &str) { + gauge.reset(); + gauge.with_label_values(&[label]).set(1); } static LEAN_DATA_COLUMN_FETCH_FAILURES_TOTAL: LazyLock = LazyLock::new(|| { @@ -615,3 +624,20 @@ pub fn observe_beacon_aggregate_decode(duration: std::time::Duration) { }); LEAN_BEACON_AGGREGATE_DECODE_SECONDS.observe(duration.as_secs_f64()); } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn only_the_newest_label_reads_one() { + // A local vector: the global gauge is also written by every transition + // test running in parallel in this binary. + let gauge = register_int_gauge_vec!("lean_test_only_label_gauge", "test", &["digest"]) + .expect("a unique name"); + set_only_label(&gauge, "aaaaaaaa"); + set_only_label(&gauge, "bbbbbbbb"); + assert_eq!(gauge.with_label_values(&["bbbbbbbb"]).get(), 1); + assert_eq!(gauge.with_label_values(&["aaaaaaaa"]).get(), 0); + } +} diff --git a/crates/net/p2p/src/req_resp/behaviour.rs b/crates/net/p2p/src/req_resp/behaviour.rs index cae1002b8..8242337e7 100644 --- a/crates/net/p2p/src/req_resp/behaviour.rs +++ b/crates/net/p2p/src/req_resp/behaviour.rs @@ -61,6 +61,8 @@ pub(crate) struct ReqResp { pub(crate) beacon_blocks_by_root: request_response::Behaviour, pub(crate) data_column_sidecars_by_range: request_response::Behaviour, pub(crate) data_column_sidecars_by_root: request_response::Behaviour, + pub(crate) execution_payload_envelopes_by_range: request_response::Behaviour, + pub(crate) execution_payload_envelopes_by_root: request_response::Behaviour, } impl ReqResp { @@ -197,7 +199,7 @@ impl ReqResp { fetch_protocol_config(), ), data_column_sidecars_by_root: request_response::Behaviour::with_codec( - codec, + codec.clone(), one_protocol( is_beacon, beacon::protocols::DATA_COLUMN_SIDECARS_BY_ROOT_V1, @@ -205,6 +207,24 @@ impl ReqResp { ), fetch_protocol_config(), ), + execution_payload_envelopes_by_range: request_response::Behaviour::with_codec( + codec.clone(), + one_protocol( + is_beacon, + beacon::protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_RANGE_V1, + request_response::ProtocolSupport::Full, + ), + fetch_protocol_config(), + ), + execution_payload_envelopes_by_root: request_response::Behaviour::with_codec( + codec, + one_protocol( + is_beacon, + beacon::protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_ROOT_V1, + request_response::ProtocolSupport::Full, + ), + fetch_protocol_config(), + ), } } } diff --git a/crates/net/p2p/src/req_resp/codec.rs b/crates/net/p2p/src/req_resp/codec.rs index 5aea1084e..9bd77f103 100644 --- a/crates/net/p2p/src/req_resp/codec.rs +++ b/crates/net/p2p/src/req_resp/codec.rs @@ -11,8 +11,8 @@ use super::{ }; use crate::beacon::messages::{ - BeaconBlocksByRangeRequest, DataColumnsByRangeRequest, DataColumnsByRootIdentifiers, Goodbye, - Ping, + BeaconBlocksByRangeRequest, DataColumnsByRangeRequest, DataColumnsByRootIdentifiers, + ExecutionPayloadEnvelopeRoots, ExecutionPayloadEnvelopesByRangeRequest, Goodbye, Ping, }; use crate::beacon::{BeaconContext, encoding as beacon_encoding, protocols}; use crate::lean::messages::{BlocksByRootRequest, RequestedBlockRoots}; @@ -20,8 +20,8 @@ use crate::lean::{encoding as lean_encoding, protocols as lean_protocols}; use crate::metrics; /// Protocols whose response payload has one shape forever write no context -/// bytes, which is every protocol here except the two beacon block ones and -/// the two beacon data column sidecar ones. +/// bytes, which is every protocol here except the two beacon block ones, the +/// two beacon data column sidecar ones and the two gloas envelope ones. const NO_CONTEXT: &[u8] = &[]; /// Short label extracted from a libp2p protocol id, used as the `protocol` @@ -197,6 +197,17 @@ impl libp2p::request_response::Codec for Codec { DataColumnsByRangeRequest::from_ssz_bytes(&payload) .map_err(|err| invalid(format!("{err:?}")))?, )), + protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_RANGE_V1 => { + Ok(Request::ExecutionPayloadEnvelopesByRange( + ExecutionPayloadEnvelopesByRangeRequest::from_ssz_bytes(&payload) + .map_err(|err| invalid(format!("{err:?}")))?, + )) + } + protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_ROOT_V1 => { + let roots = ExecutionPayloadEnvelopeRoots::from_ssz_bytes(&payload) + .map_err(|err| invalid(format!("{err:?}")))?; + Ok(Request::ExecutionPayloadEnvelopesByRoot(roots.into_inner())) + } _ => Err(invalid(format!("unknown protocol: {}", protocol.as_ref()))), } } @@ -266,6 +277,20 @@ impl libp2p::request_response::Codec for Codec { sidecars, ))) } + protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_RANGE_V1 + | protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_ROOT_V1 => { + let context = self.beacon_context(protocol.as_ref())?; + let envelopes = beacon_encoding::decode_execution_payload_envelopes_response( + io, + label, + &context.config, + context.genesis_validators_root, + ) + .await?; + Ok(Response::success( + ResponsePayload::ExecutionPayloadEnvelopes(envelopes), + )) + } _ => Err(invalid(format!("unknown protocol: {}", protocol.as_ref()))), } } @@ -333,6 +358,12 @@ impl libp2p::request_response::Codec for Codec { .to_ssz() } Request::DataColumnsByRange(request) => request.to_ssz(), + Request::ExecutionPayloadEnvelopesByRange(request) => request.to_ssz(), + Request::ExecutionPayloadEnvelopesByRoot(roots) => { + ExecutionPayloadEnvelopeRoots::try_from(roots.clone()) + .map_err(|err| invalid(format!("{err:?}")))? + .to_ssz() + } }; let compressed_size = write_payload(io, &encoded).await?; @@ -398,6 +429,17 @@ impl libp2p::request_response::Codec for Codec { ) .await } + ResponsePayload::ExecutionPayloadEnvelopes(envelopes) => { + let context = self.beacon_context(protocol.as_ref())?; + beacon_encoding::write_execution_payload_envelopes_response( + io, + label, + &context.config, + context.genesis_validators_root, + envelopes, + ) + .await + } }, Response::Error { code, message } => { // Send error code @@ -470,6 +512,8 @@ mod tests { self, ColumnIndices, DataColumnsByRootIdentifier, }; use ethlambda_types::beacon::containers::shared; + use ethlambda_types::beacon::containers::{DataColumnSidecar, gloas}; + use ethlambda_types::beacon::fork_digest::compute_fork_digest; use ethlambda_types::beacon::preset; use ethlambda_types::beacon::primitives::Root; use futures::io::Cursor; @@ -724,34 +768,75 @@ mod tests { } } + /// A config scheduling gloas shortly after mainnet's fulu, so fulu and gloas + /// sidecars get different digests. + fn gloas_config() -> Config { + let base = Config::mainnet(); + Config { + gloas_fork_epoch: base.fulu_fork_epoch + 64, + ..base + } + } + + fn codec_for(config: Config) -> Codec { + Codec::beacon(BeaconContext { + config, + genesis_validators_root: mainnet_gvr(), + }) + } + + fn fulu_slot() -> u64 { + Config::mainnet().fulu_fork_epoch * preset::SLOTS_PER_EPOCH + 1 + } + + fn gloas_slot() -> u64 { + gloas_config().gloas_fork_epoch * preset::SLOTS_PER_EPOCH + 1 + } + + fn gloas_data_column_sidecar(slot: u64, index: u64) -> gloas::DataColumnSidecar { + gloas::DataColumnSidecar { + index, + slot, + ..Default::default() + } + } + /// Exercises `write_data_column_sidecars_response` into /// `decode_data_column_sidecars_response` directly, unlike /// [`a_by_range_column_request_round_trips`] and /// [`a_by_root_column_request_round_trips`] above, which only round-trip /// the thin SSZ-derive request wrappers and never touch this pair. /// - /// The two sidecars straddle mainnet's altair fork boundary on purpose, - /// so their digests actually differ: each chunk's `` is - /// derived from *that sidecar's own* slot - /// (`write_data_column_sidecars_response`'s doc explains why — a - /// backfill answer labels each chunk with its own fork), so an - /// implementation that computed one digest for the whole response - /// (from, say, the first sidecar's epoch) would still round-trip a - /// batch that never crosses a fork boundary but fail this one. + /// The two sidecars straddle the fulu and gloas boundary on purpose, so + /// their digests actually differ and so do their shapes: each chunk's + /// `` is derived from *that sidecar's own* slot + /// (`write_data_column_sidecars_response`'s doc explains why: a backfill + /// answer labels each chunk with its own fork), and the context is what + /// selects the decoder. An implementation that computed one digest for the + /// whole response would fail this batch. #[tokio::test] async fn a_data_column_sidecars_response_round_trips_with_a_per_item_context() { - let sidecar_a = data_column_sidecar(3, 0); - let post_altair_slot = Config::mainnet().altair_fork_epoch * preset::SLOTS_PER_EPOCH + 1; - let sidecar_b = data_column_sidecar(post_altair_slot, 7); + let sidecar_a = DataColumnSidecar::Fulu(data_column_sidecar(fulu_slot(), 0)); + let sidecar_b = DataColumnSidecar::Gloas(gloas_data_column_sidecar(gloas_slot(), 7)); + let stream_protocol = StreamProtocol::new(protocols::DATA_COLUMN_SIDECARS_BY_RANGE_V1); - let decoded = response_round_trip( - protocols::DATA_COLUMN_SIDECARS_BY_RANGE_V1, - Response::success(ResponsePayload::DataColumnSidecars(vec![ - sidecar_a.clone(), - sidecar_b.clone(), - ])), - ) - .await; + let mut buffer = Cursor::new(Vec::new()); + codec_for(gloas_config()) + .write_response( + &stream_protocol, + &mut buffer, + Response::success(ResponsePayload::DataColumnSidecars(vec![ + sidecar_a.clone(), + sidecar_b.clone(), + ])), + ) + .await + .expect("writes"); + let mut buffer = Cursor::new(buffer.into_inner()); + let decoded = codec_for(gloas_config()) + .read_response(&stream_protocol, &mut buffer) + .await + .expect("reads"); match decoded { Response::Success { @@ -763,17 +848,78 @@ mod tests { } } + /// A gloas sidecar under a fulu chunk's context bytes is a shape the + /// context does not name, so the stream aborts rather than handing back a + /// sidecar decoded as the wrong fork. + #[tokio::test] + async fn a_gloas_sidecar_under_a_fulu_context_aborts_the_stream() { + let config = gloas_config(); + let root = mainnet_gvr(); + let fulu_digest = compute_fork_digest(&config, root, config.fulu_fork_epoch); + let payload = gloas_data_column_sidecar(gloas_slot(), 1).to_ssz(); + let stream_protocol = StreamProtocol::new(protocols::DATA_COLUMN_SIDECARS_BY_RANGE_V1); + + let mut buffer = Cursor::new(Vec::new()); + write_success_chunk(&mut buffer, "test", &fulu_digest, payload) + .await + .expect("writes"); + let mut buffer = Cursor::new(buffer.into_inner()); + let result = codec_for(config) + .read_response(&stream_protocol, &mut buffer) + .await; + + assert!(result.is_err()); + } + + /// A chunk's context must also be the digest of its sidecar's own slot: a + /// fulu-shaped sidecar carrying a gloas slot, sent under the fulu digest, + /// decodes cleanly as fulu and is still refused. + #[tokio::test] + async fn a_fulu_shaped_sidecar_at_a_gloas_slot_under_the_fulu_digest_aborts_the_stream() { + let config = gloas_config(); + let root = mainnet_gvr(); + let fulu_digest = compute_fork_digest(&config, root, config.fulu_fork_epoch); + let payload = data_column_sidecar(gloas_slot(), 1).to_ssz(); + let stream_protocol = StreamProtocol::new(protocols::DATA_COLUMN_SIDECARS_BY_RANGE_V1); + + let mut buffer = Cursor::new(Vec::new()); + write_success_chunk(&mut buffer, "test", &fulu_digest, payload) + .await + .expect("writes"); + let mut buffer = Cursor::new(buffer.into_inner()); + let result = codec_for(config) + .read_response(&stream_protocol, &mut buffer) + .await; + + assert!(result.is_err()); + } + + /// A digest no scheduled fork uses names no shape at all. + #[tokio::test] + async fn a_data_column_chunk_under_an_unscheduled_digest_aborts_the_stream() { + let payload = data_column_sidecar(fulu_slot(), 1).to_ssz(); + let stream_protocol = StreamProtocol::new(protocols::DATA_COLUMN_SIDECARS_BY_RANGE_V1); + + let mut buffer = Cursor::new(Vec::new()); + write_success_chunk(&mut buffer, "test", &[0xde, 0xad, 0xbe, 0xef], payload) + .await + .expect("writes"); + let mut buffer = Cursor::new(buffer.into_inner()); + let result = codec().read_response(&stream_protocol, &mut buffer).await; + + assert!(result.is_err()); + } + /// A peer's `genesis_validators_root` differing from ours means every /// digest it labels a chunk with is for the wrong chain, even though the - /// chunk decodes cleanly on its own. `decode_data_column_sidecars_response` - /// checks the digest against what *this* node's own root implies, so - /// reading the same bytes back through a codec built with a different - /// root must abort the stream rather than hand back a sidecar under the - /// wrong context — mirroring the block response's own fork-mismatch - /// check, which nothing here exercised before. + /// chunk decodes cleanly on its own. The digest matches no scheduled one + /// under our root, so reading the same bytes back through a codec built + /// with a different root must abort the stream rather than hand back a + /// sidecar under the wrong context, mirroring the block response's own + /// fork-mismatch check. #[tokio::test] async fn a_data_column_sidecars_response_aborts_on_a_fork_digest_mismatch() { - let sidecar = data_column_sidecar(3, 0); + let sidecar = DataColumnSidecar::Fulu(data_column_sidecar(fulu_slot(), 0)); let stream_protocol = StreamProtocol::new(protocols::DATA_COLUMN_SIDECARS_BY_RANGE_V1); let mut buffer = Cursor::new(Vec::new()); @@ -800,4 +946,137 @@ mod tests { "a fork digest mismatch must abort the stream rather than decode successfully" ); } + + fn envelope_at(slot: u64, byte: u8) -> gloas::SignedExecutionPayloadEnvelope { + crate::beacon::encoding::test_support::envelope( + Root::repeat_byte(byte), + slot, + Root::repeat_byte(byte), + ) + } + + #[tokio::test] + async fn an_envelope_by_range_request_round_trips() { + let request = Request::ExecutionPayloadEnvelopesByRange( + crate::beacon::messages::ExecutionPayloadEnvelopesByRangeRequest { + start_slot: 9, + count: 128, + }, + ); + let decoded = + request_round_trip(protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_RANGE_V1, request).await; + match decoded { + Request::ExecutionPayloadEnvelopesByRange(wire) => { + assert_eq!((wire.start_slot, wire.count), (9, 128)); + } + other => panic!("decoded as {other:?}"), + } + } + + #[tokio::test] + async fn an_envelope_by_root_request_round_trips_as_a_bare_list() { + let roots = vec![Root::repeat_byte(1), Root::repeat_byte(2)]; + let decoded = request_round_trip( + protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_ROOT_V1, + Request::ExecutionPayloadEnvelopesByRoot(roots.clone()), + ) + .await; + match decoded { + Request::ExecutionPayloadEnvelopesByRoot(decoded) => assert_eq!(decoded, roots), + other => panic!("decoded as {other:?}"), + } + } + + #[tokio::test] + async fn an_envelope_by_root_request_over_the_limit_is_refused_on_write() { + let roots = vec![Root::ZERO; protocols::MAX_REQUEST_PAYLOADS as usize + 1]; + let stream_protocol = + StreamProtocol::new(protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_ROOT_V1); + let mut buffer = Cursor::new(Vec::new()); + let result = codec() + .write_request( + &stream_protocol, + &mut buffer, + Request::ExecutionPayloadEnvelopesByRoot(roots), + ) + .await; + assert!(result.is_err()); + } + + /// Each chunk's context is the digest of its own envelope's epoch; for + /// gloas envelopes that is the gloas digest. + #[tokio::test] + async fn an_envelope_response_round_trips_under_the_gloas_digest() { + let envelopes = vec![ + envelope_at(gloas_slot(), 1), + envelope_at(gloas_slot() + 1, 2), + ]; + let stream_protocol = + StreamProtocol::new(protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_RANGE_V1); + + let mut buffer = Cursor::new(Vec::new()); + codec_for(gloas_config()) + .write_response( + &stream_protocol, + &mut buffer, + Response::success(ResponsePayload::ExecutionPayloadEnvelopes( + envelopes.clone(), + )), + ) + .await + .expect("writes"); + let bytes = buffer.into_inner(); + // The result code, then the four context bytes. + let config = gloas_config(); + let expected = compute_fork_digest(&config, mainnet_gvr(), config.gloas_fork_epoch); + assert_eq!(bytes[0], 0); + assert_eq!(bytes[1..5], expected); + + let decoded = codec_for(gloas_config()) + .read_response(&stream_protocol, &mut Cursor::new(bytes)) + .await + .expect("reads"); + match decoded { + Response::Success { + payload: ResponsePayload::ExecutionPayloadEnvelopes(decoded), + } => assert_eq!(decoded, envelopes), + other => panic!("decoded as {other:?}"), + } + } + + #[tokio::test] + async fn an_envelope_chunk_under_a_fulu_digest_aborts_the_stream() { + let config = gloas_config(); + let fulu_digest = compute_fork_digest(&config, mainnet_gvr(), config.fulu_fork_epoch); + let payload = envelope_at(gloas_slot(), 1).to_ssz(); + let stream_protocol = + StreamProtocol::new(protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_ROOT_V1); + + let mut buffer = Cursor::new(Vec::new()); + write_success_chunk(&mut buffer, "test", &fulu_digest, payload) + .await + .expect("writes"); + let result = codec_for(config) + .read_response(&stream_protocol, &mut Cursor::new(buffer.into_inner())) + .await; + + assert!(result.is_err()); + } + + #[tokio::test] + async fn an_envelope_chunk_under_an_unscheduled_digest_aborts_the_stream() { + let payload = envelope_at(gloas_slot(), 1).to_ssz(); + let stream_protocol = + StreamProtocol::new(protocols::EXECUTION_PAYLOAD_ENVELOPES_BY_ROOT_V1); + + let mut buffer = Cursor::new(Vec::new()); + write_success_chunk(&mut buffer, "test", &[0xde, 0xad, 0xbe, 0xef], payload) + .await + .expect("writes"); + let result = codec_for(gloas_config()) + .read_response(&stream_protocol, &mut Cursor::new(buffer.into_inner())) + .await; + + assert!(result.is_err()); + } } diff --git a/crates/net/p2p/src/req_resp/envelope_client.rs b/crates/net/p2p/src/req_resp/envelope_client.rs new file mode 100644 index 000000000..1714ce3dc --- /dev/null +++ b/crates/net/p2p/src/req_resp/envelope_client.rs @@ -0,0 +1,734 @@ +//! The client side of the gloas execution payload envelope protocols: asking +//! peers for envelopes by root, and alongside range sync batches by range. +//! +//! The server side is in [`super::envelopes`]. Everything fetched here goes +//! through `beacon::envelope_checks` before the chain actor sees it. + +use std::collections::HashSet; +use std::time::Duration; + +use ethlambda_types::beacon::config::Config; +use ethlambda_types::beacon::containers::gloas::SignedExecutionPayloadEnvelope; +use ethlambda_types::primitives::H256; +use libp2p::PeerId; +use spawned_concurrency::tasks::{Context, send_after}; +use tracing::{debug, error, trace}; + +use super::Request; +use super::handlers::{choose_fetch_peer, request_next_beacon_range_batch, retire_root_attempt}; +use crate::beacon::decode::fork_at_slot; +use crate::beacon::envelope_checks; +use crate::beacon::messages::ExecutionPayloadEnvelopesByRangeRequest; +use crate::beacon::protocols::{MAX_REQUEST_BLOCKS_DENEB, MAX_REQUEST_PAYLOADS}; +use crate::{P2PServer, PendingRequest, PendingRequestKind, ReqRespProtocol, p2p_protocol}; + +// A range request is clamped to the smaller of the two ceilings, so a block +// batch is never wider than its envelope request can answer for. +const _: () = assert!(MAX_REQUEST_PAYLOADS >= MAX_REQUEST_BLOCKS_DENEB); + +/// What one failed attempt at an envelope lookup leaves to do. +#[derive(Debug, PartialEq, Eq)] +enum FailedAttempt { + /// Ask again after the backoff. + Retry(Duration), + /// The ladder is exhausted and the lookup is retired. + GaveUp, + /// Nothing is waiting on this root: a late or duplicate failure. + Stale, +} + +/// Charge `peer` with a failed attempt at `block_root`'s envelope, retiring the +/// lookup when the ladder runs out. +fn fail_envelope_attempt(server: &mut P2PServer, block_root: H256, peer: PeerId) -> FailedAttempt { + let Some(pending) = server.pending_envelope_requests.get_mut(&block_root) else { + return FailedAttempt::Stale; + }; + let attempts = pending.attempts; + match retire_root_attempt(pending, peer) { + Some(backoff) => { + debug!(%block_root, %peer, attempts, ?backoff, "Envelope fetch failed, scheduling retry"); + FailedAttempt::Retry(backoff) + } + None => { + error!(%block_root, %peer, attempts, + "Envelope fetch failed after max retries, giving up"); + server.pending_envelope_requests.remove(&block_root); + FailedAttempt::GaveUp + } + } +} + +/// Retire one failed attempt at fetching `block_root`'s envelope. +/// +/// [`handle_fetch_failure`]'s counterpart, with the same invariant: every path +/// that ends an attempt comes through here, or the root stays in +/// `pending_envelope_requests` and deduplicates every later ask for it. A +/// lookup that gives up is not lost for good: the chain actor asks again each +/// slot for every parent still missing its envelope. +pub(super) async fn handle_envelope_fetch_failure( + server: &mut P2PServer, + block_root: H256, + peer: PeerId, + ctx: &Context, +) { + if let FailedAttempt::Retry(backoff) = fail_envelope_attempt(server, block_root, peer) { + send_after( + backoff, + ctx.clone(), + p2p_protocol::RetryEnvelopeFetch { block_root }, + ); + } +} + +/// Let range sync go on after a gloas batch's envelopes: whether they were +/// delivered, refused, or never came. Does nothing when no batch is waiting. +pub(crate) async fn release_envelope_gate(server: &mut P2PServer, ctx: &Context) { + let Some(state) = &mut server.range_sync_state else { + return; + }; + if !std::mem::take(&mut state.envelopes_pending) { + return; + } + request_next_beacon_range_batch(server, ctx).await; +} + +/// Fetch the envelope of `block_root` from a random connected peer. +/// +/// [`fetch_block_from_peer`]'s counterpart for envelopes: same peer choice, and +/// the `pending_envelope_requests` entry is what dedupes a repeated ask and +/// what the retry path reads. Beacon-only, since the lean wire has no +/// envelopes. Returns whether a request went out. +pub async fn fetch_envelope_from_peer(server: &mut P2PServer, block_root: H256) -> bool { + if !server.wire.is_beacon() { + debug!(%block_root, "Cannot fetch an envelope on the lean wire"); + return false; + } + if server.connected_peers.is_empty() { + debug!(%block_root, "Cannot fetch envelope: no connected peers"); + return false; + } + + let Some((peer, excluded)) = choose_fetch_peer( + &server.connected_peers, + server + .pending_envelope_requests + .get_mut(&block_root) + .map(|pending| &mut pending.failed_peers), + &block_root, + ) else { + debug!(%block_root, "Failed to select random peer"); + return false; + }; + + trace!(%peer, %block_root, excluded, "Sending ExecutionPayloadEnvelopesByRoot request"); + let Some(request_id) = server + .swarm_handle + .send_request( + peer, + Request::ExecutionPayloadEnvelopesByRoot(vec![block_root]), + ReqRespProtocol::ExecutionPayloadEnvelopesByRoot, + ) + .await + else { + debug!(%block_root, "Failed to send ExecutionPayloadEnvelopesByRoot request (swarm adapter closed)"); + return false; + }; + server + .outbound_requests + .insert(request_id, PendingRequestKind::EnvelopeRoot(block_root)); + server + .pending_envelope_requests + .entry(block_root) + .or_insert(PendingRequest { + attempts: 1, + failed_peers: HashSet::new(), + }); + true +} + +/// Keep the envelopes of `envelopes` that answer a by-root request for +/// `block_root`. Returns them with how many were dropped. +/// +/// Mirrors [`retain_requested_columns`]: nothing downstream asks whether an +/// envelope was wanted, so an unsolicited one would otherwise be checked and +/// handed to the chain actor on a peer's say-so. +fn retain_requested_envelopes( + envelopes: Vec, + block_root: H256, +) -> (Vec, usize) { + let received = envelopes.len(); + let kept: Vec<_> = envelopes + .into_iter() + .filter(|envelope| envelope.message.beacon_block_root == block_root) + .collect(); + let dropped = received - kept.len(); + (kept, dropped) +} + +/// Take delivery of an `ExecutionPayloadEnvelopesByRoot` answer. +/// +/// An answer with no envelope for the requested root, an empty one included, is +/// a failed attempt and goes through [`handle_envelope_fetch_failure`], the +/// same way a block-by-root answer does. What does match goes through the +/// envelope checks (`beacon::envelope_checks`), which hand the chain actor the +/// ones that pass. +pub(super) async fn handle_envelopes_by_root_response( + server: &mut P2PServer, + peer: PeerId, + block_root: H256, + envelopes: Vec, + ctx: &Context, +) { + match settle_by_root_answer(server, peer, block_root, envelopes) { + ByRootAnswer::Forward(envelopes) => { + if server.blockchain.is_none() { + debug!(%peer, %block_root, "No blockchain handler available"); + return; + } + envelope_checks::check_and_forward(server, envelopes); + } + ByRootAnswer::Failed(FailedAttempt::Retry(backoff)) => { + send_after( + backoff, + ctx.clone(), + p2p_protocol::RetryEnvelopeFetch { block_root }, + ); + } + ByRootAnswer::Failed(FailedAttempt::GaveUp | FailedAttempt::Stale) => {} + } +} + +/// What a by-root answer comes to. +#[derive(Debug, PartialEq)] +enum ByRootAnswer { + /// The envelopes that answer the request; the lookup is retired. + Forward(Vec), + /// Nothing answered it: an attempt is spent. + Failed(FailedAttempt), +} + +fn settle_by_root_answer( + server: &mut P2PServer, + peer: PeerId, + block_root: H256, + envelopes: Vec, +) -> ByRootAnswer { + let received = envelopes.len(); + trace!(%peer, %block_root, received, "Received ExecutionPayloadEnvelopesByRoot response"); + + let (envelopes, unrequested) = retain_requested_envelopes(envelopes, block_root); + if unrequested > 0 { + debug!( + %peer, + %block_root, + unrequested, + "Dropping envelopes the ExecutionPayloadEnvelopesByRoot request did not ask for" + ); + } + + if envelopes.is_empty() { + debug!(%peer, %block_root, "ExecutionPayloadEnvelopesByRoot response carried no matching envelope"); + return ByRootAnswer::Failed(fail_envelope_attempt(server, block_root, peer)); + } + + server.pending_envelope_requests.remove(&block_root); + ByRootAnswer::Forward(envelopes) +} + +/// Whether a range sync batch over `span` can contain a gloas block, and so an +/// envelope worth asking for. Earlier forks have no envelopes, and the server +/// answers such a request empty at best. +fn range_reaches_gloas(config: &Config, span: &std::ops::RangeInclusive) -> bool { + fork_at_slot(config, *span.end()).has_payload_envelopes() +} + +/// Ask `peer` for the envelopes in `[start_slot, start_slot + count)`, the span +/// of a `BeaconBlocksByRange` answer that was just handed to the chain actor. +/// +/// Sent after that answer rather than beside the blocks request, so the +/// ordering the chain actor needs does not depend on which of two streams +/// finishes first: the blocks are already in its mailbox when this request +/// goes out, and its answer is forwarded by a task spawned later still, so +/// the actor sees every envelope after the blocks of the batch (its mailbox is +/// FIFO). It never holds the batch's block delivery on this answer. +/// +/// The chain actor keeps an envelope whose block has not imported only +/// briefly and in small numbers, so range delivery relies on the batches +/// reaching it in order. A batch's envelopes must therefore land before the +/// next batch's blocks do, and the next request is held back until the answer +/// has been checked and forwarded: this sets +/// `RangeSyncState::envelopes_pending`, and [`release_envelope_gate`] clears +/// it. An envelope the answer lacks is asked for by root once its block is a +/// parent missing one. +/// +/// Returns whether a request went out, which is also whether range sync is now +/// waiting on it. Does nothing for a span that ends before gloas. +pub(crate) async fn request_beacon_envelopes_by_range( + server: &mut P2PServer, + peer: PeerId, + start_slot: u64, + count: u64, +) -> bool { + let Some(wire) = server.wire.beacon() else { + return false; + }; + // The ceiling the server truncates to, so the span recorded below is the + // one the peer actually answers for. + let count = count.min(MAX_REQUEST_PAYLOADS); + if count == 0 { + return false; + } + let end_slot = start_slot.saturating_add(count - 1); + if !range_reaches_gloas(&wire.config, &(start_slot..=end_slot)) { + return false; + } + + trace!(%peer, start_slot, count, "Sending ExecutionPayloadEnvelopesByRange request"); + let request = ExecutionPayloadEnvelopesByRangeRequest { start_slot, count }; + let Some(request_id) = server + .swarm_handle + .send_request( + peer, + Request::ExecutionPayloadEnvelopesByRange(request), + ReqRespProtocol::ExecutionPayloadEnvelopesByRange, + ) + .await + else { + debug!(%peer, "Failed to send ExecutionPayloadEnvelopesByRange request (swarm adapter closed)"); + return false; + }; + server.outbound_requests.insert( + request_id, + PendingRequestKind::EnvelopeRange { + start_slot, + end_slot, + }, + ); + if let Some(state) = &mut server.range_sync_state { + state.envelopes_pending = true; + } + true +} + +/// Keep the envelopes of a range answer that fall in `[start_slot, end_slot]`, +/// in slot order, with how many were dropped. An envelope outside the span is +/// dropped on its own rather than failing the answer, the way a block outside +/// its range is. The sort is what lets the chain actor see a parent's envelope +/// before its child's. +fn retain_envelopes_in_range( + envelopes: Vec, + start_slot: u64, + end_slot: u64, +) -> (Vec, usize) { + let received = envelopes.len(); + let mut kept: Vec<_> = envelopes + .into_iter() + .filter(|envelope| (start_slot..=end_slot).contains(&envelope.message.payload.slot_number)) + .collect(); + kept.sort_by_key(|envelope| envelope.message.payload.slot_number); + let dropped = received - kept.len(); + (kept, dropped) +} + +/// Take delivery of an `ExecutionPayloadEnvelopesByRange` answer. +/// +/// No retry, unlike the by-root path: a short or empty answer is not an +/// attempt spent, and the envelopes it lacks are asked for by root, since the +/// chain actor re-asks every slot for each parent still missing one. Range +/// sync is released once the answer has been forwarded, or at once when there +/// is nothing to forward. +pub(super) async fn handle_envelopes_by_range_response( + server: &mut P2PServer, + peer: PeerId, + start_slot: u64, + end_slot: u64, + envelopes: Vec, + ctx: &Context, +) { + let received = envelopes.len(); + trace!(%peer, start_slot, end_slot, received, "Received ExecutionPayloadEnvelopesByRange response"); + + if server.blockchain.is_none() { + debug!(%peer, "No blockchain handler available"); + release_envelope_gate(server, ctx).await; + return; + } + + let (envelopes, dropped) = retain_envelopes_in_range(envelopes, start_slot, end_slot); + if dropped > 0 { + debug!(%peer, start_slot, end_slot, dropped, "Dropping out-of-range envelopes"); + } + let resume = ctx.clone(); + let done: envelope_checks::Done = Box::new(move || { + send_after( + Duration::ZERO, + resume, + p2p_protocol::ResumeRangeAfterEnvelopes, + ); + }); + envelope_checks::check_and_forward_then(server, envelopes, Some(done)); +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + use std::sync::Arc; + use std::time::Duration; + + use ethlambda_network_api::{BlockArrival, BlockSource}; + use ethlambda_types::beacon::containers::SignedBeaconBlock; + use ethlambda_types::beacon::fork::ForkName; + + use super::*; + use crate::req_resp::handlers::tests::unconnected_server; + use crate::{BACKOFF_MULTIPLIER, ConnectionDirection, INITIAL_BACKOFF_MS, MAX_FETCH_RETRIES}; + + fn envelope_for(slot: u64, block_root: H256) -> SignedExecutionPayloadEnvelope { + crate::beacon::encoding::test_support::envelope(block_root, slot, H256::ZERO) + } + + fn fulu_block_at(slot: u64) -> SignedBeaconBlock { + use ethlambda_types::beacon::containers::electra; + SignedBeaconBlock::Fulu(electra::SignedBeaconBlock { + message: electra::BeaconBlock { + slot, + proposer_index: 0, + parent_root: H256::ZERO, + state_root: H256::ZERO, + body: electra::BeaconBlockBody::empty(), + }, + signature: Default::default(), + }) + } + + #[test] + fn a_by_root_envelope_answer_keeps_only_the_requested_root() { + let asked = H256::repeat_byte(1); + let other = H256::repeat_byte(2); + let answer = vec![envelope_for(5, other), envelope_for(5, asked)]; + + let (kept, dropped) = retain_requested_envelopes(answer, asked); + + assert_eq!(kept, vec![envelope_for(5, asked)]); + assert_eq!(dropped, 1); + } + + #[test] + fn a_range_envelope_answer_keeps_the_span_in_slot_order() { + let root = H256::repeat_byte(1); + let answer = vec![ + envelope_for(12, root), + envelope_for(9, root), + envelope_for(10, root), + envelope_for(11, root), + ]; + + let (kept, dropped) = retain_envelopes_in_range(answer, 10, 11); + + let slots: Vec = kept + .iter() + .map(|envelope| envelope.message.payload.slot_number) + .collect(); + assert_eq!(slots, vec![10, 11]); + assert_eq!(dropped, 2); + } + + #[test] + fn only_a_span_reaching_gloas_asks_for_envelopes() { + let config = Config::mainnet().with_fork_epoch(ForkName::Gloas, 10); + let first_gloas_slot = 10 * ethlambda_types::beacon::preset::SLOTS_PER_EPOCH; + + assert!(!range_reaches_gloas(&config, &(0..=first_gloas_slot - 1))); + // A span that starts before gloas and ends inside it overlaps. + assert!(range_reaches_gloas( + &config, + &(first_gloas_slot - 3..=first_gloas_slot) + )); + assert!(range_reaches_gloas( + &config, + &(first_gloas_slot + 5..=first_gloas_slot + 9) + )); + // A chain with no gloas scheduled never does. + assert!(!range_reaches_gloas( + &Config::mainnet(), + &(0..=u64::MAX / 2) + )); + } + + #[test] + fn an_envelope_lookup_backs_off_by_doubling_then_gives_up() { + let mut pending = PendingRequest { + attempts: 1, + failed_peers: HashSet::new(), + }; + let peer = PeerId::random(); + + let first = retire_root_attempt(&mut pending, peer).expect("first retry"); + let second = retire_root_attempt(&mut pending, peer).expect("second retry"); + + assert_eq!(first, Duration::from_millis(INITIAL_BACKOFF_MS)); + assert_eq!(second, first * BACKOFF_MULTIPLIER as u32); + assert!(pending.failed_peers.contains(&peer)); + assert_eq!(pending.attempts, 3); + + pending.attempts = MAX_FETCH_RETRIES; + assert_eq!(retire_root_attempt(&mut pending, peer), None); + } + + #[test] + fn a_failed_peer_is_skipped_until_every_peer_has_failed() { + let root = H256::repeat_byte(1); + let (a, b) = (PeerId::random(), PeerId::random()); + let connected: HashMap = [(a, ()), (b, ())].into(); + let mut failed: HashSet = [a].into(); + + for _ in 0..16 { + assert_eq!( + choose_fetch_peer(&connected, Some(&mut failed), &root), + Some((b, 1)) + ); + } + + // With every peer failed the set is cleared, so a new round starts. + failed.insert(b); + let (_, excluded) = choose_fetch_peer(&connected, Some(&mut failed), &root).unwrap(); + assert_eq!(excluded, 0); + assert!(failed.is_empty()); + + assert_eq!(choose_fetch_peer::<()>(&HashMap::new(), None, &root), None); + } + + #[tokio::test] + async fn an_envelope_fetch_with_nobody_to_ask_leaves_nothing_pending() { + let mut server = crate::test_support::unconnected_beacon_server(Config::mainnet(), 0).await; + + let sent = fetch_envelope_from_peer(&mut server, H256::repeat_byte(1)).await; + + assert!(!sent); + assert!(server.pending_envelope_requests.is_empty()); + assert!(server.outbound_requests.is_empty()); + } + + #[tokio::test] + async fn an_envelope_fetch_on_the_lean_wire_sends_nothing() { + let mut server = unconnected_server().await; + server + .connected_peers + .insert(PeerId::random(), ConnectionDirection::Inbound); + + assert!(!fetch_envelope_from_peer(&mut server, H256::repeat_byte(1)).await); + assert!(server.pending_envelope_requests.is_empty()); + } + + /// One request goes out per root however often the chain actor asks, and + /// it names the root alone. + #[tokio::test] + async fn an_envelope_fetch_is_tracked_and_deduplicated() { + let mut server = crate::test_support::unconnected_beacon_server(Config::mainnet(), 0).await; + server + .connected_peers + .insert(PeerId::random(), ConnectionDirection::Outbound); + let root = H256::repeat_byte(7); + + crate::fetch_missing_envelope(&mut server, root).await; + crate::fetch_missing_envelope(&mut server, root).await; + + assert_eq!(server.pending_envelope_requests[&root].attempts, 1); + let asked: Vec<_> = server.outbound_requests.iter().collect(); + assert_eq!( + asked.len(), + 1, + "a duplicate ask must not send a second request" + ); + let (id, kind) = asked[0]; + assert_eq!( + id.protocol, + ReqRespProtocol::ExecutionPayloadEnvelopesByRoot + ); + assert!(matches!(kind, PendingRequestKind::EnvelopeRoot(r) if *r == root)); + } + + #[tokio::test] + async fn a_range_batch_asks_for_envelopes_only_where_gloas_overlaps() { + let config = Config::mainnet().with_fork_epoch(ForkName::Gloas, 10); + let first_gloas_slot = 10 * ethlambda_types::beacon::preset::SLOTS_PER_EPOCH; + let mut server = crate::test_support::unconnected_beacon_server(config, 0).await; + let peer = PeerId::random(); + server + .connected_peers + .insert(peer, ConnectionDirection::Outbound); + + server.range_sync_state = Some(crate::RangeSyncState::new(0..100_000, peer, 99_999)); + + assert!(!request_beacon_envelopes_by_range(&mut server, peer, 0, 64).await); + assert!(server.outbound_requests.is_empty()); + // Pre-gloas pacing is unchanged: the next batch is not held back. + let state = server.range_sync_state.as_ref().unwrap(); + assert!(!state.envelopes_pending); + assert!(state.next_batch().is_some()); + + // The span is clamped to what the peer will answer for. + assert!( + request_beacon_envelopes_by_range(&mut server, peer, first_gloas_slot, 1_000).await + ); + let (id, kind) = server.outbound_requests.iter().next().unwrap(); + assert_eq!( + id.protocol, + ReqRespProtocol::ExecutionPayloadEnvelopesByRange + ); + assert!(matches!( + kind, + PendingRequestKind::EnvelopeRange { start_slot, end_slot } + if *start_slot == first_gloas_slot + && *end_slot == first_gloas_slot + MAX_REQUEST_PAYLOADS - 1 + )); + // A gloas span holds the next batch back until its answer is delivered. + let state = server.range_sync_state.as_mut().unwrap(); + assert!(state.envelopes_pending); + assert!(state.next_batch().is_none()); + state.envelopes_pending = false; + assert!(state.next_batch().is_some()); + } + + /// The release runs the completion callback even with nothing to forward, + /// or no chain to forward to, so range sync cannot wait forever. + #[tokio::test] + async fn the_range_release_fires_when_there_is_nothing_to_forward() { + use crate::beacon::envelope_checks::tests::Recorder; + + let mut server = crate::test_support::unconnected_beacon_server(Config::mainnet(), 0).await; + let (done_tx, mut done_rx) = tokio::sync::mpsc::unbounded_channel(); + let callback = |tx: tokio::sync::mpsc::UnboundedSender<()>| -> envelope_checks::Done { + Box::new(move || { + let _ = tx.send(()); + }) + }; + + // No chain actor. + envelope_checks::check_and_forward_then( + &server, + vec![envelope_for(7, H256::repeat_byte(1))], + Some(callback(done_tx.clone())), + ); + assert_eq!(done_rx.recv().await, Some(())); + + // An empty answer. + let (sender, _received) = tokio::sync::mpsc::unbounded_channel(); + server.blockchain = Some(Arc::new(Recorder(sender))); + envelope_checks::check_and_forward_then(&server, Vec::new(), Some(callback(done_tx))); + assert_eq!(done_rx.recv().await, Some(())); + } + + fn pending_lookup(server: &mut P2PServer, root: H256, attempts: u32) { + server.pending_envelope_requests.insert( + root, + PendingRequest { + attempts, + failed_peers: HashSet::new(), + }, + ); + } + + /// An empty or non-matching by-root answer spends an attempt and schedules + /// a retry after the first rung of the backoff; a good one retires the + /// lookup and is forwarded. + #[tokio::test] + async fn a_by_root_answer_without_the_envelope_schedules_a_retry() { + let mut server = crate::test_support::unconnected_beacon_server(Config::mainnet(), 0).await; + let root = H256::repeat_byte(4); + let peer = PeerId::random(); + pending_lookup(&mut server, root, 1); + + let empty = settle_by_root_answer(&mut server, peer, root, Vec::new()); + assert_eq!( + empty, + ByRootAnswer::Failed(FailedAttempt::Retry(Duration::from_millis( + INITIAL_BACKOFF_MS + ))) + ); + assert_eq!(server.pending_envelope_requests[&root].attempts, 2); + assert!( + server.pending_envelope_requests[&root] + .failed_peers + .contains(&peer) + ); + + // An envelope for another root is as good as none. + let wrong = settle_by_root_answer( + &mut server, + peer, + root, + vec![envelope_for(9, H256::repeat_byte(5))], + ); + assert!(matches!( + wrong, + ByRootAnswer::Failed(FailedAttempt::Retry(_)) + )); + + let good = settle_by_root_answer(&mut server, peer, root, vec![envelope_for(9, root)]); + assert_eq!(good, ByRootAnswer::Forward(vec![envelope_for(9, root)])); + assert!(server.pending_envelope_requests.is_empty()); + } + + #[tokio::test] + async fn a_by_root_lookup_gives_up_when_the_ladder_is_spent() { + let mut server = crate::test_support::unconnected_beacon_server(Config::mainnet(), 0).await; + let root = H256::repeat_byte(4); + pending_lookup(&mut server, root, MAX_FETCH_RETRIES); + + let outcome = fail_envelope_attempt(&mut server, root, PeerId::random()); + + assert_eq!(outcome, FailedAttempt::GaveUp); + assert!(server.pending_envelope_requests.is_empty()); + // A late failure for a retired lookup is ignored. + assert_eq!( + fail_envelope_attempt(&mut server, root, PeerId::random()), + FailedAttempt::Stale + ); + } + + /// The range answer reaches the chain actor after the blocks the batch + /// handed it, and in slot order, whichever order the peer listed them. + #[tokio::test] + async fn range_envelopes_are_forwarded_after_the_batch_blocks_in_slot_order() { + use crate::beacon::envelope_checks::tests::{Recorder, Seen}; + + let mut server = crate::test_support::unconnected_beacon_server(Config::mainnet(), 0).await; + let (sender, mut received) = tokio::sync::mpsc::unbounded_channel(); + let chain = Arc::new(Recorder(sender)); + server.blockchain = Some(chain.clone()); + let root = H256::repeat_byte(3); + + // What `handle_beacon_blocks_by_range_response` does before it asks + // for the span's envelopes. + for slot in [20, 21] { + let _ = ethlambda_network_api::P2PToBlockChain::new_block( + chain.as_ref(), + fulu_block_at(slot), + BlockSource::Sync, + BlockArrival::now(), + ethlambda_network_api::BlockAnnouncement::Silent, + ); + } + // The handler's own steps, less the actor `Context` it also needs for + // the release message: filter to the span in slot order, then check + // and forward with a completion callback. + let (answer, _) = + retain_envelopes_in_range(vec![envelope_for(21, root), envelope_for(20, root)], 20, 21); + let (done_tx, mut done_rx) = tokio::sync::mpsc::unbounded_channel(); + envelope_checks::check_and_forward_then( + &server, + answer, + Some(Box::new(move || { + let _ = done_tx.send(()); + })), + ); + + assert_eq!(received.recv().await, Some(Seen::Block(20))); + assert_eq!(received.recv().await, Some(Seen::Block(21))); + assert_eq!(received.recv().await, Some(Seen::Envelope(root, 20))); + assert_eq!(received.recv().await, Some(Seen::Envelope(root, 21))); + // The release comes only after the last envelope was sent. + assert_eq!(done_rx.recv().await, Some(())); + } +} diff --git a/crates/net/p2p/src/req_resp/envelopes.rs b/crates/net/p2p/src/req_resp/envelopes.rs new file mode 100644 index 000000000..c7f2249c9 --- /dev/null +++ b/crates/net/p2p/src/req_resp/envelopes.rs @@ -0,0 +1,431 @@ +//! The gloas execution payload envelope protocols +//! (`execution_payload_envelopes_by_range/1` and `_by_root/1`): the server side, +//! answering peers out of `Table::ExecutionPayloadEnvelopes`. + +use ethlambda_storage::Store; +use ethlambda_types::beacon::containers::gloas; +use ethlambda_types::primitives::H256; +use libp2p::PeerId; +use libp2p::request_response::ResponseChannel; +use tracing::{debug, error, trace, warn}; + +use super::handlers::{beacon_block_store_or_refuse, refuse, respond}; +use super::messages::ResponseCode; +use super::{Response, ResponsePayload}; +use crate::P2PServer; +use crate::beacon::messages::ExecutionPayloadEnvelopesByRangeRequest; +use crate::beacon::protocols::MAX_REQUEST_PAYLOADS; + +/// Answer `execution_payload_envelopes_by_range/1` off the canonical chain. +/// +/// The spec makes it equivalent to `BeaconBlocksByRange` v2 with another +/// response type. An empty window is `INVALID_REQUEST` and a `start_slot` +/// below the anchor is `RESOURCE_UNAVAILABLE`, as for blocks. A `count` above +/// `MAX_REQUEST_PAYLOADS` is truncated to it rather than refused, which the +/// specification's "Clients MAY limit the number of payload envelopes in the +/// response" allows; the block handler refuses a `count` past its phase0 +/// ceiling instead, because a request that wide is a protocol violation there. +/// A window before gloas simply holds no envelopes, so it is an empty answer. Which envelopes count as +/// canonical is [`Store::canonical_execution_payload_envelopes`]'s business. +pub(super) async fn handle_execution_payload_envelopes_by_range_request( + server: &mut P2PServer, + peer: PeerId, + request: ExecutionPayloadEnvelopesByRangeRequest, + channel: ResponseChannel, +) { + let Some(channel) = beacon_block_store_or_refuse(server, peer, channel) else { + return; + }; + + if request.count == 0 { + refuse( + server, + channel, + ResponseCode::INVALID_REQUEST, + "invalid ExecutionPayloadEnvelopesByRange request", + ); + return; + } + let envelopes = match envelopes_by_range(&server.store, request.start_slot, request.count) { + Ok(envelopes) => envelopes, + Err(reason) => { + debug!( + %peer, + start_slot = request.start_slot, + "ExecutionPayloadEnvelopesByRange request starts before this node's chain" + ); + refuse(server, channel, ResponseCode::RESOURCE_UNAVAILABLE, reason); + return; + } + }; + + trace!( + %peer, + start_slot = request.start_slot, + count = request.count, + found = envelopes.len(), + "Responding to ExecutionPayloadEnvelopesByRange request" + ); + + respond( + server, + channel, + ResponsePayload::ExecutionPayloadEnvelopes(envelopes), + ); +} + +/// The envelopes `ExecutionPayloadEnvelopesByRange` answers a non-empty window +/// with: the canonical ones in `[start_slot, start_slot + count)`, truncated +/// to `MAX_REQUEST_PAYLOADS` slots. +/// +/// `Err` carries the `RESOURCE_UNAVAILABLE` reason for a `start_slot` below +/// the store's anchor, which this chain does not reach back to. +fn envelopes_by_range( + store: &Store, + start_slot: u64, + count: u64, +) -> Result, &'static str> { + if start_slot < store.anchor_slot() { + return Err("requested range starts before this node's earliest available slot"); + } + let count = count.min(MAX_REQUEST_PAYLOADS); + // An overflowing window is attacker-supplied and answered empty, as is a + // zero `count`, which has no last offset to add. + let Some(end_slot) = count + .checked_sub(1) + .and_then(|last_offset| start_slot.checked_add(last_offset)) + else { + return Ok(Vec::new()); + }; + Ok(store + .canonical_execution_payload_envelopes(start_slot, end_slot) + .inspect_err(|err| { + warn!( + start_slot, + end_slot, + ?err, + "Failed to get execution payload envelopes by slot range" + ) + }) + .unwrap_or_default()) +} + +/// Answer `execution_payload_envelopes_by_root/1` with whichever of the named +/// envelopes this node holds. +/// +/// The counterpart of [`handle_beacon_blocks_by_root_request`]: a root with no +/// stored envelope is left out rather than answered with an error, and the +/// answer follows the order asked in. Roots past `MAX_REQUEST_PAYLOADS` are +/// ignored (the wire list type already bounds a decoded request to it). +pub(super) async fn handle_execution_payload_envelopes_by_root_request( + server: &mut P2PServer, + peer: PeerId, + roots: Vec, + channel: ResponseChannel, +) { + let Some(channel) = beacon_block_store_or_refuse(server, peer, channel) else { + return; + }; + + let requested = roots.len(); + let envelopes = envelopes_by_root(&server.store, &roots); + + trace!( + %peer, + requested, + found = envelopes.len(), + "Responding to ExecutionPayloadEnvelopesByRoot request" + ); + + respond( + server, + channel, + ResponsePayload::ExecutionPayloadEnvelopes(envelopes), + ); +} + +/// The held envelopes among `roots`, in the order asked, looking at no more +/// than `MAX_REQUEST_PAYLOADS` of them. +fn envelopes_by_root(store: &Store, roots: &[H256]) -> Vec { + roots + .iter() + .take(MAX_REQUEST_PAYLOADS as usize) + .filter_map(|root| { + store + .get_execution_payload_envelope(root) + .inspect_err(|err| { + error!( + root = %ethlambda_types::ShortRoot(&root.0), + %err, + "Stored execution payload envelope failed to read" + ) + }) + .ok() + .flatten() + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::beacon::encoding::test_support; + use ethlambda_storage::{ForkCheckpoints, backend::InMemoryBackend}; + use ethlambda_types::beacon::config::Config; + use ethlambda_types::beacon::containers::SignedBeaconBlock; + use ethlambda_types::beacon::fork_choice::PayloadStatus; + use ethlambda_types::checkpoint::Checkpoint; + use std::sync::Arc; + + /// A gloas block whose bid builds on the payload `parent_block_hash`. + fn gloas_block(slot: u64, parent_root: H256, parent_block_hash: H256) -> SignedBeaconBlock { + let mut block = gloas::SignedBeaconBlock { + message: gloas::BeaconBlock { + slot, + proposer_index: 0, + parent_root, + state_root: H256::ZERO, + body: Default::default(), + }, + signature: Default::default(), + }; + block + .message + .body + .signed_execution_payload_bid + .message + .parent_block_hash = parent_block_hash; + SignedBeaconBlock::Gloas(block) + } + + fn beacon_store() -> Store { + let mut store = Store::init_beacon( + Arc::new(InMemoryBackend::new()), + 0, + Config::mainnet(), + H256::ZERO, + Checkpoint::default(), + 0, + ); + store + .insert_signed_block(H256::ZERO, gloas_block(0, H256::ZERO, H256::ZERO)) + .expect("insert anchor"); + store + } + + /// Inserts `block` and moves the head to it; returns its root. + fn extend_chain(store: &mut Store, block: SignedBeaconBlock) -> H256 { + let root = block.message_hash_tree_root(); + store.insert_signed_block(root, block).expect("insert"); + store.set_head_payload_status(root, PayloadStatus::Full); + store + .update_checkpoints(ForkCheckpoints::head_only(root)) + .expect("advance head"); + root + } + + fn reveal( + store: &mut Store, + slot: u64, + root: H256, + hash: H256, + ) -> gloas::SignedExecutionPayloadEnvelope { + let envelope = test_support::envelope(root, slot, hash); + store.insert_verified_payload(slot, &envelope); + envelope + } + + fn hash(byte: u8) -> H256 { + H256::repeat_byte(byte) + } + + /// Canonical slots 1 to 3, every payload revealed and built on, plus a + /// sibling of slot 2 with its own envelope. Returns the three canonical + /// envelopes. + fn chain_with_every_payload_built_on() -> (Store, Vec) { + let mut store = beacon_store(); + let root_1 = extend_chain(&mut store, gloas_block(1, H256::ZERO, hash(0))); + let root_2 = extend_chain(&mut store, gloas_block(2, root_1, hash(1))); + let root_3 = extend_chain(&mut store, gloas_block(3, root_2, hash(2))); + // A fork at slot 2 that the head does not descend from. + let sibling = gloas_block(2, root_1, hash(9)); + let sibling_root = sibling.message_hash_tree_root(); + store + .insert_signed_block(sibling_root, sibling) + .expect("insert"); + reveal(&mut store, 2, sibling_root, hash(0x99)); + let envelopes = vec![ + reveal(&mut store, 1, root_1, hash(1)), + reveal(&mut store, 2, root_2, hash(2)), + reveal(&mut store, 3, root_3, hash(3)), + ]; + (store, envelopes) + } + + #[test] + fn by_range_serves_only_canonical_envelopes_in_slot_order() { + let (store, envelopes) = chain_with_every_payload_built_on(); + + let served = envelopes_by_range(&store, 1, 3).expect("in window"); + + assert_eq!(served, envelopes); + assert_eq!( + envelopes_by_range(&store, 2, 1).expect("in window"), + envelopes[1..2] + ); + } + + #[test] + fn by_range_omits_a_payload_the_next_block_does_not_build_on() { + let mut store = beacon_store(); + let root_1 = extend_chain(&mut store, gloas_block(1, H256::ZERO, hash(0))); + // Slot 2 builds on slot 1's empty branch, not on the payload revealed. + let root_2 = extend_chain(&mut store, gloas_block(2, root_1, hash(0))); + reveal(&mut store, 1, root_1, hash(1)); + let head_envelope = reveal(&mut store, 2, root_2, hash(2)); + + let served = envelopes_by_range(&store, 1, 2).expect("in window"); + + assert_eq!(served, vec![head_envelope]); + } + + #[test] + fn by_range_judges_the_last_block_of_a_window_by_its_successor_past_the_window() { + let mut store = beacon_store(); + let root_1 = extend_chain(&mut store, gloas_block(1, H256::ZERO, hash(0))); + // Slot 3 follows an empty slot 2 and builds on slot 1's payload. + extend_chain(&mut store, gloas_block(3, root_1, hash(1))); + let envelope = reveal(&mut store, 1, root_1, hash(1)); + + assert_eq!( + envelopes_by_range(&store, 1, 1).expect("in window"), + vec![envelope] + ); + } + + /// A phase0 block, standing in for any block before gloas. + fn pre_gloas_block(slot: u64, parent_root: H256) -> SignedBeaconBlock { + use ethlambda_types::beacon::containers::phase0; + + SignedBeaconBlock::Phase0(phase0::SignedBeaconBlock { + message: phase0::BeaconBlock { + slot, + proposer_index: 0, + parent_root, + state_root: H256::ZERO, + body: phase0::BeaconBlockBody { + randao_reveal: Default::default(), + eth1_data: Default::default(), + graffiti: H256::ZERO, + proposer_slashings: Default::default(), + attester_slashings: Default::default(), + attestations: Default::default(), + deposits: Default::default(), + voluntary_exits: Default::default(), + }, + }, + signature: Default::default(), + }) + } + + #[test] + fn by_range_caps_the_window_at_max_request_payloads() { + let mut store = beacon_store(); + let total = MAX_REQUEST_PAYLOADS + 2; + let payload_hash = |slot: u64| H256::repeat_byte(slot as u8); + let mut parent = H256::ZERO; + for slot in 1..=total { + // Each block builds on the previous slot's payload, so every + // envelope is on the chain and only the cap can cut the answer. + let block = gloas_block(slot, parent, payload_hash(slot - 1)); + parent = extend_chain(&mut store, block); + reveal(&mut store, slot, parent, payload_hash(slot)); + } + + let served = envelopes_by_range(&store, 1, total).expect("in window"); + + assert_eq!(served.len() as u64, MAX_REQUEST_PAYLOADS); + let last_slot = served + .last() + .expect("non-empty") + .message + .payload + .slot_number; + assert_eq!(last_slot, MAX_REQUEST_PAYLOADS); + // A count past the cap is cut to it, not refused. + let wide = envelopes_by_range(&store, 1, u64::MAX).expect("in window"); + assert_eq!(wide, served); + } + + #[test] + fn by_range_answers_empty_for_a_pre_gloas_window_that_has_blocks() { + let mut store = beacon_store(); + let root_1 = extend_chain(&mut store, pre_gloas_block(1, H256::ZERO)); + let root_2 = extend_chain(&mut store, pre_gloas_block(2, root_1)); + let root_3 = extend_chain(&mut store, gloas_block(3, root_2, hash(0))); + let envelope = reveal(&mut store, 3, root_3, hash(3)); + + assert!( + envelopes_by_range(&store, 1, 2) + .expect("in window") + .is_empty() + ); + assert_eq!( + envelopes_by_range(&store, 1, 3).expect("in window"), + vec![envelope] + ); + } + + #[test] + fn by_range_answers_empty_for_an_overflowing_window() { + let (store, _) = chain_with_every_payload_built_on(); + + assert!( + envelopes_by_range(&store, u64::MAX, 5) + .expect("in window") + .is_empty() + ); + } + + #[test] + fn by_range_refuses_a_start_below_the_anchor() { + let mut store = Store::init_beacon( + Arc::new(InMemoryBackend::new()), + 0, + Config::mainnet(), + H256::ZERO, + Checkpoint::default(), + 10, + ); + store + .insert_signed_block(H256::ZERO, gloas_block(10, H256::ZERO, H256::ZERO)) + .expect("insert anchor"); + + assert!(envelopes_by_range(&store, 9, 1).is_err()); + assert!(envelopes_by_range(&store, 10, 1).is_ok()); + } + + #[test] + fn by_root_serves_known_envelopes_in_request_order_and_skips_unknown_roots() { + let (store, envelopes) = chain_with_every_payload_built_on(); + let roots: Vec = envelopes + .iter() + .map(|e| e.message.beacon_block_root) + .collect(); + let asked = vec![roots[2], H256::repeat_byte(0xee), roots[0]]; + + let served = envelopes_by_root(&store, &asked); + + assert_eq!(served, vec![envelopes[2].clone(), envelopes[0].clone()]); + } + + #[test] + fn by_root_looks_at_no_more_than_the_limit() { + let (store, envelopes) = chain_with_every_payload_built_on(); + let known = envelopes[0].message.beacon_block_root; + let mut asked = vec![H256::repeat_byte(0xee); MAX_REQUEST_PAYLOADS as usize]; + asked.push(known); + + assert!(envelopes_by_root(&store, &asked).is_empty()); + } +} diff --git a/crates/net/p2p/src/req_resp/handlers.rs b/crates/net/p2p/src/req_resp/handlers.rs index 4478e933b..4d7b30623 100644 --- a/crates/net/p2p/src/req_resp/handlers.rs +++ b/crates/net/p2p/src/req_resp/handlers.rs @@ -23,22 +23,25 @@ use tracing::{debug, error, info, trace, warn}; use ethlambda_state_transition::beacon::das; use ethlambda_types::beacon::config::Config; use ethlambda_types::beacon::constants; -use ethlambda_types::beacon::containers::SignedBeaconBlock; -use ethlambda_types::beacon::containers::fulu::{ - ColumnIndices, DataColumnSidecar, DataColumnsByRootIdentifier, -}; +use ethlambda_types::beacon::containers::fulu::{ColumnIndices, DataColumnsByRootIdentifier}; +use ethlambda_types::beacon::containers::{DataColumnSidecar, SignedBeaconBlock}; use ethlambda_types::beacon::fork::ForkName; use ethlambda_types::checkpoint::Checkpoint; use ethlambda_types::primitives::HashTreeRoot as _; use ethlambda_types::{block::SignedBlock, primitives::H256}; +pub(crate) use super::envelope_client::release_envelope_gate; +use super::envelope_client::{ + handle_envelope_fetch_failure, handle_envelopes_by_range_response, + handle_envelopes_by_root_response, request_beacon_envelopes_by_range, +}; use super::{ - Request, Response, ResponsePayload, + Request, Response, ResponsePayload, envelopes, messages::{ResponseCode, error_message}, }; use crate::beacon::BeaconWire; use crate::beacon::column_checks; -use crate::beacon::decode::{decode_data_column_sidecar, fork_at_slot}; +use crate::beacon::decode::fork_at_slot; use crate::beacon::handler::{self as beacon_handler, StatusVersion}; use crate::beacon::messages::{ BeaconMetaData, BeaconStatus, DataColumnsByRangeRequest, Goodbye, Ping, @@ -46,6 +49,7 @@ use crate::beacon::messages::{ use crate::beacon::protocols::{ MAX_REQUEST_BLOCKS as MAX_BEACON_REQUEST_BLOCKS, MAX_REQUEST_BLOCKS_DENEB, }; +use crate::beacon::transition; use crate::discovery::enr::node_id_from_peer_id; use crate::lean::messages::{BlocksByRootRequest, RequestedBlockRoots, Status}; use crate::lean::protocols::MAX_REQUEST_BLOCKS; @@ -56,6 +60,7 @@ use crate::{ RangeSyncState, ReqRespProtocol, ReqRespRequestId, UNKNOWN_CUSTODY_RANGE_PEERS, metrics, p2p_protocol, }; +use ethlambda_types::time::unix_now_ms; use libp2p::request_response::ResponseChannel; /// `protocol` names which [`ReqResp`](super::ReqResp) field `event` came @@ -167,6 +172,26 @@ pub async fn handle_req_resp_message( ) .await; } + Request::ExecutionPayloadEnvelopesByRange(request) => { + trace!( + kind = "execution_payload_envelopes_by_range_request", + peer_count, "P2P message received" + ); + envelopes::handle_execution_payload_envelopes_by_range_request( + server, peer, request, channel, + ) + .await; + } + Request::ExecutionPayloadEnvelopesByRoot(roots) => { + trace!( + kind = "execution_payload_envelopes_by_root_request", + peer_count, "P2P message received" + ); + envelopes::handle_execution_payload_envelopes_by_root_request( + server, peer, roots, channel, + ) + .await; + } } } request_response::Message::Response { @@ -222,9 +247,9 @@ pub async fn handle_req_resp_message( // event: this response is the terminal outcome for // the id either way. match server.outbound_requests.remove(&request_id) { - Some(PendingRequestKind::Columns(block_root)) => { + Some(PendingRequestKind::Columns(block_root, requested)) => { handle_data_column_sidecars_response( - server, peer, block_root, sidecars, ctx, + server, peer, block_root, &requested, sidecars, ctx, ) .await; } @@ -248,7 +273,10 @@ pub async fn handle_req_resp_message( // exactly the way #608 fixed, since no further // event will ever name this id again. Some( - PendingRequestKind::Root(_) | PendingRequestKind::Range { .. }, + kind @ (PendingRequestKind::Root(_) + | PendingRequestKind::Range { .. } + | PendingRequestKind::EnvelopeRoot(_) + | PendingRequestKind::EnvelopeRange { .. }), ) => { error!( %peer, @@ -256,6 +284,7 @@ pub async fn handle_req_resp_message( count = sidecars.len(), "Data column sidecars response answered a non-column request id" ); + retire_unanswered(server, kind, peer, ctx).await; } None => { debug!( @@ -267,6 +296,56 @@ pub async fn handle_req_resp_message( } } } + ResponsePayload::ExecutionPayloadEnvelopes(envelopes) => { + trace!( + kind = "execution_payload_envelopes_response", + peer_count, "P2P message received" + ); + // Removed here, like the other terminal answers: this + // response is the whole exchange for the id. + match server.outbound_requests.remove(&request_id) { + Some(PendingRequestKind::EnvelopeRoot(block_root)) => { + handle_envelopes_by_root_response( + server, peer, block_root, envelopes, ctx, + ) + .await; + } + Some(PendingRequestKind::EnvelopeRange { + start_slot, + end_slot, + }) => { + handle_envelopes_by_range_response( + server, peer, start_slot, end_slot, envelopes, ctx, + ) + .await; + } + // Unreachable by construction, as for the + // column payload above: only the two envelope + // kinds are sent on the envelope protocols. + Some( + kind @ (PendingRequestKind::Root(_) + | PendingRequestKind::Range { .. } + | PendingRequestKind::Columns(..) + | PendingRequestKind::ColumnRange { .. }), + ) => { + error!( + %peer, + ?request_id, + count = envelopes.len(), + "Envelopes response answered a non-envelope request id" + ); + retire_unanswered(server, kind, peer, ctx).await; + } + None => { + debug!( + %peer, + ?request_id, + count = envelopes.len(), + "Received envelopes response for unknown request_id" + ); + } + } + } ResponsePayload::Blocks(blocks) => { trace!(kind = "blocks_response", peer_count, "P2P message received"); // Dispatched on what was asked for first and on the @@ -305,13 +384,16 @@ pub async fn handle_req_resp_message( // data column protocol, which answers with // `DataColumnSidecars`, never with `Blocks`. Some( - PendingRequestKind::Columns(_) - | PendingRequestKind::ColumnRange { .. }, + kind @ (PendingRequestKind::Columns(..) + | PendingRequestKind::ColumnRange { .. } + | PendingRequestKind::EnvelopeRoot(_) + | PendingRequestKind::EnvelopeRange { .. }), ) => { error!( %peer, - "Blocks response answered a data column request id" + "Blocks response answered a non-block request id" ); + retire_unanswered(server, kind, peer, ctx).await; } None => { debug!(%peer, ?request_id, "Received blocks response for unknown request_id"); @@ -334,7 +416,7 @@ pub async fn handle_req_resp_message( // forever and deduplicates every later fetch. handle_fetch_failure(server, root, peer, ctx).await; } - Some(PendingRequestKind::Columns(block_root)) => { + Some(PendingRequestKind::Columns(block_root, _)) => { // Same reasoning as the `Root` arm above: an // error response is the whole exchange, so // this is the only place that can retire it. @@ -346,6 +428,16 @@ pub async fn handle_req_resp_message( // just means these columns come from gossip or // from the by-root path instead. Some(PendingRequestKind::ColumnRange { .. }) => {} + Some(PendingRequestKind::EnvelopeRoot(block_root)) => { + // Same reasoning as the `Root` arm above. + handle_envelope_fetch_failure(server, block_root, peer, ctx).await; + } + // Nothing retries it, and the chain actor re-asks + // by root; but range pacing waits on the answer, + // so it must be released here. + Some(PendingRequestKind::EnvelopeRange { .. }) => { + release_envelope_gate(server, ctx).await; + } None => {} } } @@ -383,7 +475,7 @@ pub async fn handle_req_resp_message( "BlocksByRange request failed; retry is disabled" ); } - Some(PendingRequestKind::Columns(block_root)) => { + Some(PendingRequestKind::Columns(block_root, _)) => { handle_column_fetch_failure(server, block_root, peer, ctx).await; } // Nothing waits on a range prefetch, so a failure is only @@ -401,6 +493,21 @@ pub async fn handle_req_resp_message( "DataColumnsByRange request failed; columns fall back to the by-root path" ); } + Some(PendingRequestKind::EnvelopeRoot(block_root)) => { + handle_envelope_fetch_failure(server, block_root, peer, ctx).await; + } + Some(PendingRequestKind::EnvelopeRange { + start_slot, + end_slot, + }) => { + debug!( + %peer, + start_slot, + end_slot, + "ExecutionPayloadEnvelopesByRange request failed; envelopes fall back to the by-root path" + ); + release_envelope_gate(server, ctx).await; + } // The handshake is the only *tracked* request kind absent // here: every other outcome for a `Root`, `Range` or `Columns` // id is handled above, so reaching `None` means either the @@ -462,14 +569,18 @@ fn lean_blocks(blocks: Vec) -> Vec { /// /// Every request handler on either chain ends here, which is most of what the /// two have in common above encoding. -fn respond(server: &mut P2PServer, channel: ResponseChannel, payload: ResponsePayload) { +pub(super) fn respond( + server: &mut P2PServer, + channel: ResponseChannel, + payload: ResponsePayload, +) { server .swarm_handle .send_response(channel, Response::success(payload)); } /// Answer a request with an error code and a reason. -fn refuse( +pub(super) fn refuse( server: &mut P2PServer, channel: ResponseChannel, code: ResponseCode, @@ -791,44 +902,17 @@ pub async fn fetch_block_from_peer(server: &mut P2PServer, root: H256) -> bool { return false; } - // Exclude peers that already returned empty responses for this root - let failed = server - .pending_root_requests - .get(&root) - .map(|p| &p.failed_peers); - let pool: Vec<_> = if failed.is_none_or(|f| f.is_empty()) { - server.connected_peers.keys().copied().collect() - } else { - let failed = failed.unwrap(); + let Some((peer, excluded)) = choose_fetch_peer( + &server.connected_peers, server - .connected_peers - .keys() - .copied() - .filter(|p| !failed.contains(p)) - .collect() - }; - - // Fall back to full set if all peers have failed (new peers may have connected, - // or previously-failing peers may have caught up). Clear failed_peers so subsequent - // retries start a fresh round of elimination. - let pool = if pool.is_empty() { - debug!(%root, "All peers failed for this block, retrying with full peer set"); - if let Some(pending) = server.pending_root_requests.get_mut(&root) { - pending.failed_peers.clear(); - } - server.connected_peers.keys().copied().collect() - } else { - pool - }; - - let peer = match pool.choose(&mut rand::thread_rng()) { - Some(&p) => p, - None => { - debug!(%root, "Failed to select random peer"); - return false; - } + .pending_root_requests + .get_mut(&root) + .map(|pending| &mut pending.failed_peers), + &root, + ) else { + debug!(%root, "Failed to select random peer"); + return false; }; - let excluded = server.connected_peers.len() - pool.len(); let sent = if server.wire.is_beacon() { trace!(%peer, %root, excluded, "Sending BeaconBlocksByRoot request for missing block"); @@ -884,6 +968,39 @@ pub async fn fetch_block_from_peer(server: &mut P2PServer, root: H256) -> bool { true } +/// Pick a random connected peer that has not already failed this lookup. +/// +/// Returns the peer and how many were excluded. When every peer has failed it +/// falls back to the full set (new peers may have connected, or failing ones +/// may have caught up) and clears `failed`, so the next retries start a fresh +/// round of elimination. Shared by the block and envelope by-root fetches, +/// whose peer choice has nothing to do with what is being fetched. +pub(super) fn choose_fetch_peer( + connected: &HashMap, + failed: Option<&mut HashSet>, + root: &H256, +) -> Option<(PeerId, usize)> { + let pool: Vec = match failed.as_deref() { + Some(failed) if !failed.is_empty() => connected + .keys() + .copied() + .filter(|peer| !failed.contains(peer)) + .collect(), + _ => connected.keys().copied().collect(), + }; + let pool = if pool.is_empty() { + debug!(%root, "All peers failed for this lookup, retrying with full peer set"); + if let Some(failed) = failed { + failed.clear(); + } + connected.keys().copied().collect() + } else { + pool + }; + let peer = *pool.choose(&mut rand::thread_rng())?; + Some((peer, connected.len() - pool.len())) +} + /// Record what columns `peer` custodies, given the custody group count it /// advertised. /// @@ -1056,6 +1173,7 @@ pub async fn fetch_data_columns_from_peer( let mut sent_count = 0usize; for (peer, columns) in by_peer { let count = columns.len(); + let requested = columns.clone(); let column_indices = match ColumnIndices::try_from(columns) { Ok(indices) => indices, Err(err) => { @@ -1084,9 +1202,10 @@ pub async fn fetch_data_columns_from_peer( debug!(%block_root, %peer, "Failed to send DataColumnsByRoot request (swarm adapter closed)"); continue; }; - server - .outbound_requests - .insert(request_id, PendingRequestKind::Columns(block_root)); + server.outbound_requests.insert( + request_id, + PendingRequestKind::Columns(block_root, requested), + ); sent_count += 1; } @@ -1199,7 +1318,10 @@ async fn request_next_range_batch(server: &mut P2PServer) -> bool { /// nothing failed. The batch is re-checked when a peer's metadata arrives (see /// [`resume_range_batch_held_for_custody`]), on every call that would have sent /// it anyway, and at the deadline. -async fn request_next_beacon_range_batch(server: &mut P2PServer, ctx: &Context) -> bool { +pub(super) async fn request_next_beacon_range_batch( + server: &mut P2PServer, + ctx: &Context, +) -> bool { let Some((peer, batch)) = server .range_sync_state .as_ref() @@ -1325,13 +1447,11 @@ fn range_batch_needs_columns(server: &P2PServer, batch: &std::ops::Range) - /// range sync skips its custody-peer check before PeerDAS for the same reason: /// a batch with no columns to fetch has no custodian to wait for. /// -/// Gloas answers yes as fulu does, though its columns have a different shape. -/// A batch ending in gloas can begin in fulu, whose blocks need their columns. -/// A batch lying entirely in gloas also reaches this, since -/// `beacon_fetched_through` advances past the gloas blocks the chain actor -/// refuses. It is then held for custody for nothing, and its column prefetch -/// fails to decode against fulu's shape; that failure logs at debug and -/// penalizes no peer. No gloas column is ever consumed from here. +/// Gloas answers yes as fulu does, though its columns have a different shape +/// and decode as gloas sidecars by the chunk's context bytes. A batch ending in +/// gloas can begin in fulu, whose blocks need their columns; a batch lying +/// entirely in gloas still needs them too, since a later change makes the chain +/// actor gate a block's payload envelope on its columns. fn range_needs_columns( config: &Config, custody_columns: &[u64], @@ -1405,23 +1525,33 @@ async fn handle_fetch_failure( return; }; - pending.failed_peers.insert(peer); - - if pending.attempts >= MAX_FETCH_RETRIES { - error!(%root, %peer, attempts=%pending.attempts, + let attempts = pending.attempts; + let Some(backoff) = retire_root_attempt(pending, peer) else { + error!(%root, %peer, attempts, "Block fetch failed after max retries, giving up"); server.pending_root_requests.remove(&root); return; - } + }; - let backoff_ms = INITIAL_BACKOFF_MS * BACKOFF_MULTIPLIER.pow(pending.attempts - 1); - let backoff = Duration::from_millis(backoff_ms); + debug!(%root, %peer, attempts, ?backoff, "Block fetch failed, scheduling retry"); - debug!(%root, %peer, attempts=%pending.attempts, ?backoff, "Block fetch failed, scheduling retry"); + send_after(backoff, ctx.clone(), p2p_protocol::RetryBlockFetch { root }); +} - pending.attempts += 1; +/// Charge `peer` with a failed attempt and say how long to wait before the +/// next one: `None` once the ladder is exhausted. The step shared by the block +/// and envelope by-root lookups, whose bookkeeping is the same +/// [`PendingRequest`]. +pub(super) fn retire_root_attempt(pending: &mut PendingRequest, peer: PeerId) -> Option { + pending.failed_peers.insert(peer); - send_after(backoff, ctx.clone(), p2p_protocol::RetryBlockFetch { root }); + if pending.attempts >= MAX_FETCH_RETRIES { + return None; + } + + let backoff_ms = INITIAL_BACKOFF_MS * BACKOFF_MULTIPLIER.pow(pending.attempts - 1); + pending.attempts += 1; + Some(Duration::from_millis(backoff_ms)) } /// Retire one failed attempt at fetching `block_root`'s missing columns. @@ -1582,7 +1712,7 @@ async fn handle_status_request( // `refuse` above (which needs `&mut server`) is behind us. let wire = server.wire.beacon().expect("checked above"); - if peer_status.fork_digest() != wire.fork_digest { + if !wire.holds_digest(peer_status.fork_digest()) { // Not grounds for closing the stream: the peer told us who it is and we // answer honestly. Counting it is how a digest that has moved under us // becomes visible. @@ -1709,7 +1839,7 @@ async fn handle_status_response( let Some(wire) = server.wire.beacon() else { return; }; - if status.fork_digest() != wire.fork_digest { + if !wire.holds_digest(status.fork_digest()) { warn!( %peer, peer_digest = %hex::encode(status.fork_digest()), @@ -1719,6 +1849,18 @@ async fn handle_status_response( metrics::inc_beacon_status_digest_mismatch(); return; } + // Answered at any held digest, but only synced from on the current one (or + // the next, within clock skew): after a boundary, a peer still on the old + // digest has not upgraded, and its chain is not the one being followed. + if !transition::is_syncable_digest(wire, status.fork_digest(), unix_now_ms()) { + debug!( + %peer, + peer_digest = %hex::encode(status.fork_digest()), + our_digest = %hex::encode(wire.fork_digest), + "Not syncing from a peer on a superseded fork digest" + ); + return; + } let peer_head_slot = status.head_slot(); trace!( %peer, @@ -1790,7 +1932,7 @@ fn handle_metadata_response(server: &mut P2PServer, peer: PeerId, metadata: Beac /// to block requests" (data column sidecar requests name the same code for /// the same reason), where an empty stream claims we looked and had nothing, /// and `INVALID_REQUEST` would blame the asker for a request that was fine. -fn beacon_block_store_or_refuse( +pub(super) fn beacon_block_store_or_refuse( server: &mut P2PServer, peer: PeerId, channel: ResponseChannel, @@ -1953,21 +2095,18 @@ async fn handle_data_column_sidecars_by_root_request( continue; }; for &column in identifier.columns.iter() { - let Ok(Some(encoded)) = - server - .store - .get_data_column_sidecar(slot, &identifier.block_root, column) - else { - continue; - }; - match decode_data_column_sidecar(&encoded) { - Ok(sidecar) => sidecars.push(sidecar), + match server + .store + .get_data_column(slot, &identifier.block_root, column) + { + Ok(Some(sidecar)) => sidecars.push(sidecar), + Ok(None) => {} Err(err) => error!( %peer, slot, column, %err, - "Stored data column sidecar failed to decode" + "Stored data column sidecar failed to read" ), } } @@ -2051,7 +2190,7 @@ async fn handle_data_column_sidecars_by_range_request( let columns = request.columns.to_vec(); let sidecars: Vec<_> = server .store - .data_column_sidecars_in_range(request.start_slot, end_slot, &columns) + .data_column_rows_in_range(request.start_slot, end_slot, &columns) .inspect_err(|err| { warn!( start_slot = request.start_slot, @@ -2062,8 +2201,10 @@ async fn handle_data_column_sidecars_by_range_request( }) .unwrap_or_default() .into_iter() - .filter_map(|encoded| { - decode_data_column_sidecar(&encoded) + .filter_map(|(slot, encoded)| { + server + .store + .decode_data_column_sidecar(slot, &encoded) .inspect_err( |err| error!(%peer, %err, "Stored data column sidecar failed to decode"), ) @@ -2340,7 +2481,7 @@ async fn handle_data_column_sidecars_range_response( let in_range: Vec = sidecars .into_iter() .filter(|sidecar| { - let slot = sidecar.signed_block_header.message.slot; + let slot = sidecar.slot(); let keep = slot >= start_slot && slot <= end_slot; if !keep { debug!(%peer, slot, start_slot, end_slot, "Dropping an out-of-range data column sidecar"); @@ -2360,16 +2501,49 @@ async fn handle_data_column_sidecars_range_response( column_checks::check_and_forward(server, in_range); } +/// Keep the sidecars of `sidecars` that answer the request: the requested +/// block's, at a requested column. Returns them with how many were dropped. +/// +/// Mirrors [`handle_blocks_by_root_response`], which keeps only the block the +/// request named. Nothing downstream re-asks whether a sidecar was wanted, and +/// an unsolicited one would otherwise be parked or stored on a peer's say-so. +fn retain_requested_columns( + sidecars: Vec, + block_root: H256, + requested: &[u64], +) -> (Vec, usize) { + let received = sidecars.len(); + let kept: Vec<_> = sidecars + .into_iter() + .filter(|sidecar| { + sidecar.block_root() == block_root && requested.contains(&sidecar.index()) + }) + .collect(); + let dropped = received - kept.len(); + (kept, dropped) +} + async fn handle_data_column_sidecars_response( server: &mut P2PServer, peer: PeerId, block_root: H256, + requested: &[u64], sidecars: Vec, ctx: &Context, ) { let received = sidecars.len(); trace!(%peer, %block_root, received, "Received DataColumnsByRoot response"); + let (sidecars, unrequested) = retain_requested_columns(sidecars, block_root, requested); + if unrequested > 0 { + debug!( + %peer, + %block_root, + unrequested, + "Dropping data column sidecars the DataColumnsByRoot request did not ask for" + ); + } + if sidecars.is_empty() { debug!(%peer, %block_root, "DataColumnsByRoot response carried no sidecars"); handle_column_fetch_failure(server, block_root, peer, ctx).await; @@ -2459,6 +2633,18 @@ async fn handle_beacon_blocks_by_range_response( debug!(%peer, received, accepted, "Beacon blocks received"); + // The batch's blocks are all in the actor's mailbox now; ask for the + // envelopes of the same span only after that. On a gloas span the next + // batch is then held back until the answer has been delivered (see + // `RangeSyncState::envelopes_pending`). + let waiting_on_envelopes = request_beacon_envelopes_by_range( + server, + peer, + start_slot, + end_slot.saturating_sub(start_slot).saturating_add(1), + ) + .await; + // Highest slot *handed to* the actor, not the highest imported: see // [`beacon_sync_target`] for why range sync reads this beside the store's // head rather than the head alone. @@ -2472,14 +2658,43 @@ async fn handle_beacon_blocks_by_range_response( return; } } + if waiting_on_envelopes { + return; + } request_next_beacon_range_batch(server, ctx).await; } +/// Retire a request whose id was answered on a protocol that never carries its +/// kind. The codec cannot produce such an answer, but if one ever arrived the +/// exchange is over all the same: leaving the entry would keep its root in the +/// matching pending map, deduplicating every later ask for it, or leave range +/// sync waiting on a request that already ended. +async fn retire_unanswered( + server: &mut P2PServer, + kind: PendingRequestKind, + peer: PeerId, + ctx: &Context, +) { + match kind { + PendingRequestKind::Root(root) => handle_fetch_failure(server, root, peer, ctx).await, + PendingRequestKind::Range { .. } => fail_range_request(server, &peer), + PendingRequestKind::Columns(block_root, _) => { + handle_column_fetch_failure(server, block_root, peer, ctx).await; + } + PendingRequestKind::ColumnRange { .. } => {} + PendingRequestKind::EnvelopeRoot(block_root) => { + handle_envelope_fetch_failure(server, block_root, peer, ctx).await; + } + PendingRequestKind::EnvelopeRange { .. } => release_envelope_gate(server, ctx).await, + } +} + #[cfg(test)] -mod tests { +pub(crate) mod tests { use super::*; use crate::ConnectionDirection; use ethlambda_storage::{ForkCheckpoints, backend::InMemoryBackend}; + use ethlambda_types::beacon::containers::gloas; use ethlambda_types::constants::DEFAULT_MILLISECONDS_PER_SLOT; use ethlambda_types::enr::EnrForkId; use ethlambda_types::{ @@ -2501,7 +2716,7 @@ mod tests { /// had a harness for; see 64a6bce9), the fetch path only touches /// `&mut P2PServer`, so building one real server is the whole cost of /// testing it. - async fn unconnected_server() -> P2PServer { + pub(crate) async fn unconnected_server() -> P2PServer { let built = crate::build_swarm(crate::SwarmConfig { node_key: vec![3u8; 32], bootnodes: Vec::new(), @@ -2517,8 +2732,9 @@ mod tests { }) .expect("swarm builds"); - let (_swarm_stream, swarm_handle) = + let (swarm_stream, swarm_handle) = crate::swarm_adapter::start_swarm_adapter(built.swarm, HashMap::new()); + crate::test_support::drain_swarm_events(swarm_stream); let discovery = crate::discovery::spawn_discovery(crate::discovery::DiscoverySpawnConfig { node_key: secp256k1::SecretKey::new(&mut rand::rngs::OsRng) @@ -2559,6 +2775,7 @@ mod tests { connected_peers: HashMap::new(), peer_custody: HashMap::new(), pending_root_requests: HashMap::new(), + pending_envelope_requests: HashMap::new(), pending_column_requests: HashMap::new(), outbound_requests: HashMap::new(), range_sync_state: None, @@ -2575,6 +2792,16 @@ mod tests { seen_columns: ethlambda_state_transition::beacon::gossip::SeenColumns::new( crate::SEEN_COLUMNS_CAPACITY, ), + seen_block_columns: ethlambda_state_transition::beacon::gossip::SeenBlockColumns::new( + crate::SEEN_COLUMNS_CAPACITY, + ), + seen_envelopes: ethlambda_state_transition::beacon::gossip::SeenEnvelopes::new( + crate::SEEN_ENVELOPES_CAPACITY, + ), + seen_payload_attestations: + ethlambda_state_transition::beacon::gossip::SeenPayloadAttestations::new( + crate::SEEN_PAYLOAD_ATTESTATIONS_CAPACITY, + ), seen_aggregates: ethlambda_state_transition::beacon::gossip::aggregate::SeenAggregates::new( crate::SEEN_AGGREGATES_CAPACITY, @@ -2968,4 +3195,31 @@ mod tests { peer_gone.fail_peer(&lone_peer); assert!(range_session_exhausted(&peer_gone)); } + + fn gloas_column(block_root: H256, index: u64) -> DataColumnSidecar { + DataColumnSidecar::Gloas(gloas::DataColumnSidecar { + index, + slot: 5, + beacon_block_root: block_root, + ..Default::default() + }) + } + + #[test] + fn a_by_root_answer_keeps_only_the_requested_block_and_columns() { + let asked = H256::repeat_byte(1); + let other = H256::repeat_byte(2); + let answer = vec![ + gloas_column(asked, 3), + // A column of the right block that was not asked for. + gloas_column(asked, 4), + // A column of a block that was not asked for. + gloas_column(other, 3), + ]; + + let (kept, dropped) = retain_requested_columns(answer, asked, &[3]); + + assert_eq!(kept, vec![gloas_column(asked, 3)]); + assert_eq!(dropped, 2); + } } diff --git a/crates/net/p2p/src/req_resp/messages.rs b/crates/net/p2p/src/req_resp/messages.rs index 54f49bc18..a94f6dac6 100644 --- a/crates/net/p2p/src/req_resp/messages.rs +++ b/crates/net/p2p/src/req_resp/messages.rs @@ -1,10 +1,14 @@ use ethlambda_types::ShortRoot; +use ethlambda_types::beacon::containers::DataColumnSidecar; use ethlambda_types::beacon::containers::SignedBeaconBlock; -use ethlambda_types::beacon::containers::fulu::{DataColumnSidecar, DataColumnsByRootIdentifier}; +use ethlambda_types::beacon::containers::fulu::DataColumnsByRootIdentifier; +use ethlambda_types::beacon::containers::gloas::SignedExecutionPayloadEnvelope; +use ethlambda_types::beacon::primitives::Root; use libssz_types::SszList; use crate::beacon::messages::{ - BeaconMetaData, BeaconStatus, DataColumnsByRangeRequest, Goodbye, Ping, + BeaconMetaData, BeaconStatus, DataColumnsByRangeRequest, + ExecutionPayloadEnvelopesByRangeRequest, Goodbye, Ping, }; use crate::lean::messages::{BlocksByRootRequest, Status}; @@ -89,6 +93,13 @@ pub enum Request { /// codec already enforces on encode. DataColumnsByRoot(Vec), DataColumnsByRange(DataColumnsByRangeRequest), + /// Gloas envelopes in a slot window. Beacon-only. + ExecutionPayloadEnvelopesByRange(ExecutionPayloadEnvelopesByRangeRequest), + /// The beacon block roots of the envelopes asked for. A plain `Vec` for the + /// same reason as [`Self::DataColumnsByRoot`]: the wire type + /// ([`crate::beacon::messages::ExecutionPayloadEnvelopeRoots`]) carries the + /// bound on decode and the codec re-applies it on encode. + ExecutionPayloadEnvelopesByRoot(Vec), } #[derive(Debug, Clone)] @@ -186,6 +197,13 @@ impl std::fmt::Display for Response { // `Debug` alone runs to tens of kilobytes of cell bytes. write!(f, "Success(DataColumnSidecars count={})", sidecars.len()) } + Self::Success { + payload: ResponsePayload::ExecutionPayloadEnvelopes(envelopes), + } => write!( + f, + "Success(ExecutionPayloadEnvelopes count={})", + envelopes.len() + ), Self::Error { code, message } => { let message = String::from_utf8_lossy(message); write!(f, "Error({code:?}: {message})") @@ -272,6 +290,8 @@ pub enum ResponsePayload { /// protocols: which one produced the answer is known from the outbound /// request id, and the chunk framing is the same either way. DataColumnSidecars(Vec), + /// The envelopes answering either gloas envelope protocol. + ExecutionPayloadEnvelopes(Vec), } /// Error message type for non-success responses. diff --git a/crates/net/p2p/src/req_resp/mod.rs b/crates/net/p2p/src/req_resp/mod.rs index 005fbbc0f..9fd25569f 100644 --- a/crates/net/p2p/src/req_resp/mod.rs +++ b/crates/net/p2p/src/req_resp/mod.rs @@ -1,12 +1,15 @@ pub(crate) mod behaviour; pub(crate) mod codec; pub(crate) mod encoding; +mod envelope_client; +mod envelopes; pub mod handlers; pub(crate) mod messages; pub(crate) use behaviour::{ReqResp, ReqRespEvent}; pub use codec::Codec; pub use encoding::{MAX_COMPRESSED_PAYLOAD_SIZE, MAX_PAYLOAD_SIZE}; +pub use envelope_client::fetch_envelope_from_peer; pub use handlers::{ build_status, fetch_block_from_peer, fetch_data_columns_from_peer, handle_req_resp_message, request_beacon_block_by_root, request_beacon_blocks_by_range, diff --git a/crates/net/p2p/src/swarm_adapter.rs b/crates/net/p2p/src/swarm_adapter.rs index f060c3fb0..1303c52d7 100644 --- a/crates/net/p2p/src/swarm_adapter.rs +++ b/crates/net/p2p/src/swarm_adapter.rs @@ -366,6 +366,12 @@ fn execute_command(swarm: &mut libp2p::Swarm, cmd: SwarmCommand) { ReqRespProtocol::DataColumnSidecarsByRoot => behaviour .data_column_sidecars_by_root .send_request(&peer, request), + ReqRespProtocol::ExecutionPayloadEnvelopesByRange => behaviour + .execution_payload_envelopes_by_range + .send_request(&peer, request), + ReqRespProtocol::ExecutionPayloadEnvelopesByRoot => behaviour + .execution_payload_envelopes_by_root + .send_request(&peer, request), }; if let Some(tx) = request_id_tx { let _ = tx.send(ReqRespRequestId { protocol, id }); diff --git a/crates/net/rpc/src/beacon/blocks.rs b/crates/net/rpc/src/beacon/blocks.rs index 790747914..3e9a90594 100644 --- a/crates/net/rpc/src/beacon/blocks.rs +++ b/crates/net/rpc/src/beacon/blocks.rs @@ -54,7 +54,7 @@ async fn get_block( Encoding::Ssz => ssz_response(block.to_ssz()), Encoding::Json => crate::json_response(Envelope { version: fork.as_str(), - execution_optimistic: store.is_beacon_optimistic(root), + execution_optimistic: crate::shared::optimistic::block_is_optimistic(&store, root), finalized: is_finalized(&store, block.slot()), data: block, }), @@ -70,7 +70,7 @@ async fn get_block_root(Path(block_id): Path, State(store): State }; crate::json_response(serde_json::json!({ - "execution_optimistic": store.is_beacon_optimistic(root), + "execution_optimistic": crate::shared::optimistic::block_is_optimistic(&store, root), "finalized": is_finalized(&store, block.slot()), "data": { "root": root }, })) diff --git a/crates/net/rpc/src/beacon/config.rs b/crates/net/rpc/src/beacon/config.rs index 4c044b545..249e2ab31 100644 --- a/crates/net/rpc/src/beacon/config.rs +++ b/crates/net/rpc/src/beacon/config.rs @@ -18,13 +18,14 @@ //! //! The configuration keys include gloas's (`GLOAS_*` and the other gloas //! timing and churn keys), since `Config` holds them and this serializes the -//! whole of it. The preset and constant keys are the ones lighthouse reports, -//! less two kinds: gloas's own preset and constant keys (`DOMAIN_BEACON_BUILDER` -//! and `DOMAIN_PTC_ATTESTER` among them), which `beacon::preset` and -//! `beacon::constants` hold but this list does not report yet, and keys the -//! specification does not define at all (`GAS_LIMIT_ADJUSTMENT_FACTOR`, `RESP_TIMEOUT`, -//! `TTFB_TIMEOUT`). Constants lighthouse leaves out, such as -//! `JUSTIFICATION_BITS_LENGTH`, are left out here too. +//! whole of it. The preset and constant keys are the ones lighthouse reports +//! plus gloas's own, from `presets/*/gloas.yaml` and `beacon-chain.md`'s +//! constants (`PTC_SIZE`, `DOMAIN_BEACON_BUILDER`, ...), less keys the +//! specification does not define at all (`GAS_LIMIT_ADJUSTMENT_FACTOR`, +//! `RESP_TIMEOUT`, `TTFB_TIMEOUT`). Constants lighthouse leaves out, such as +//! `JUSTIFICATION_BITS_LENGTH`, are left out here too. Gloas's `MAX_*_SIZE` +//! preset keys bound gossip message sizes, which the node does not run on, so +//! they are left out as well. use axum::{Router, extract::State, response::Response, routing::get}; use ethlambda_storage::Store; @@ -156,6 +157,12 @@ fn extra_entries() -> impl Iterator { FIELD_ELEMENTS_PER_EXT_BLOB, CELLS_PER_EXT_BLOB, NUMBER_OF_COLUMNS, + // Gloas, in the order of the specification's `presets/*/gloas.yaml`. + PTC_SIZE, + MAX_PAYLOAD_ATTESTATIONS, + MAX_BUILDER_DEPOSIT_REQUESTS_PER_PAYLOAD, + MAX_BUILDER_EXIT_REQUESTS_PER_PAYLOAD, + MAX_BUILDERS_PER_WITHDRAWALS_SWEEP, ); let domains = entries!(hex_string; constants: @@ -171,13 +178,24 @@ fn extra_entries() -> impl Iterator { DOMAIN_SYNC_COMMITTEE_SELECTION_PROOF, DOMAIN_CONTRIBUTION_AND_PROOF, DOMAIN_BLS_TO_EXECUTION_CHANGE, + DOMAIN_BEACON_BUILDER, + DOMAIN_PTC_ATTESTER, + DOMAIN_PROPOSER_PREFERENCES, + DOMAIN_BUILDER_DEPOSIT, ); - // One-byte constants, so each goes out as a one-byte array would. - let withdrawal_prefixes = entries!(|prefix: u8| hex_string([prefix]); constants: + // One-byte constants (withdrawal prefixes and execution request types), so + // each goes out as a one-byte array would. + let one_byte_constants = entries!(|prefix: u8| hex_string([prefix]); constants: BLS_WITHDRAWAL_PREFIX, ETH1_ADDRESS_WITHDRAWAL_PREFIX, COMPOUNDING_WITHDRAWAL_PREFIX, + DEPOSIT_REQUEST_TYPE, + WITHDRAWAL_REQUEST_TYPE, + CONSOLIDATION_REQUEST_TYPE, + BUILDER_WITHDRAWAL_PREFIX, + BUILDER_DEPOSIT_REQUEST_TYPE, + BUILDER_EXIT_REQUEST_TYPE, ); // `VERSIONED_HASH_VERSION_KZG` is a `Bytes1` like the withdrawal prefixes, @@ -189,12 +207,17 @@ fn extra_entries() -> impl Iterator { VERSIONED_HASH_VERSION_KZG, UNSET_DEPOSIT_REQUESTS_START_INDEX, FULL_EXIT_REQUEST_AMOUNT, + BUILDER_INDEX_FLAG, + BUILDER_INDEX_SELF_BUILD, + BUILDER_PAYMENT_THRESHOLD_NUMERATOR, + BUILDER_PAYMENT_THRESHOLD_DENOMINATOR, + PAYLOAD_BUILDER_VERSION, ); preset .into_iter() .chain(domains) - .chain(withdrawal_prefixes) + .chain(one_byte_constants) .chain(other_constants) } @@ -290,6 +313,39 @@ mod tests { ); } + #[tokio::test] + async fn the_spec_carries_gloas_preset_and_constant_keys() { + let json = get_spec_json().await; + let data = &json["data"]; + + assert_eq!(data["PTC_SIZE"], preset::PTC_SIZE.to_string()); + assert_eq!( + data["MAX_PAYLOAD_ATTESTATIONS"], + preset::MAX_PAYLOAD_ATTESTATIONS.to_string() + ); + assert_eq!( + data["MAX_BUILDERS_PER_WITHDRAWALS_SWEEP"], + preset::MAX_BUILDERS_PER_WITHDRAWALS_SWEEP.to_string() + ); + assert_eq!(data["DOMAIN_BEACON_BUILDER"], "0x0b000000"); + assert_eq!(data["DOMAIN_PTC_ATTESTER"], "0x0c000000"); + assert_eq!(data["DOMAIN_PROPOSER_PREFERENCES"], "0x0d000000"); + assert_eq!(data["DOMAIN_BUILDER_DEPOSIT"], "0x0e000000"); + assert_eq!(data["DEPOSIT_REQUEST_TYPE"], "0x00"); + assert_eq!(data["CONSOLIDATION_REQUEST_TYPE"], "0x02"); + assert_eq!(data["BUILDER_WITHDRAWAL_PREFIX"], "0xb0"); + assert_eq!(data["BUILDER_DEPOSIT_REQUEST_TYPE"], "0x03"); + assert_eq!(data["BUILDER_EXIT_REQUEST_TYPE"], "0x04"); + assert_eq!(data["BUILDER_INDEX_SELF_BUILD"], "18446744073709551615"); + assert_eq!(data["BUILDER_INDEX_FLAG"], "1099511627776"); + assert_eq!( + data["BUILDER_PAYMENT_THRESHOLD_NUMERATOR"], + constants::BUILDER_PAYMENT_THRESHOLD_NUMERATOR.to_string() + ); + // A `Config` field, so it comes from the config half of the object. + assert!(data["MIN_BUILDER_WITHDRAWABILITY_DELAY"].is_string()); + } + #[test] fn no_key_is_reported_twice() { let config = Config::mainnet(); diff --git a/crates/net/rpc/src/beacon/envelopes.rs b/crates/net/rpc/src/beacon/envelopes.rs new file mode 100644 index 000000000..5d077c881 --- /dev/null +++ b/crates/net/rpc/src/beacon/envelopes.rs @@ -0,0 +1,193 @@ +//! `/eth/v1/beacon/execution_payload_envelopes/{block_id}`. +//! +//! Gloas moves a block's execution payload out of the block into a separate +//! signed envelope. This serves the one the node verified, which is also what +//! makes the node usable as a checkpoint-sync source for the payload of its +//! finalized block. + +use axum::{ + Router, + extract::{Path, State}, + http::{HeaderMap, header}, + response::{IntoResponse, Response}, + routing::get, +}; +use ethlambda_storage::Store; +use ethlambda_types::beacon::fork::ForkName; +use libssz::SszEncode; + +use crate::{ + beacon::{ApiError, Envelope as ResponseEnvelope, blocks::is_finalized}, + shared::{ + block_id::{BlockId, IdError}, + content::{Encoding, ssz_response, with_consensus_version}, + optimistic::envelope_is_optimistic, + }, +}; + +pub(crate) fn routes() -> Router { + Router::new().route( + "/eth/v1/beacon/execution_payload_envelopes/{block_id}", + get(get_envelope), + ) +} + +/// The envelope for `block_id`. +/// +/// 404 covers an unknown block and a block with no stored envelope alike: the +/// store holds verified envelopes only, so a pre-gloas block, a withheld +/// payload and a payload not yet received are indistinguishable here, and the +/// specification names no other status for them. +async fn get_envelope( + Path(block_id): Path, + State(store): State, + headers: HeaderMap, +) -> Response { + let not_found = ApiError::NotFound("Execution payload envelope not found"); + let root = match BlockId::parse(&block_id).and_then(|id| id.resolve_beacon(&store)) { + Ok(root) => root, + Err(IdError::Malformed) => return ApiError::BadRequest("invalid block id").into_response(), + Err(IdError::NotFound) => return not_found.into_response(), + }; + let envelope = match store.get_execution_payload_envelope(&root) { + Ok(Some(envelope)) => envelope, + Ok(None) => return not_found.into_response(), + Err(_) => return ApiError::Internal("store read failed").into_response(), + }; + // Verified against its block's bid, so the payload's slot is the block's. + let slot = envelope.message.payload.slot_number; + + let accept = headers.get(header::ACCEPT).and_then(|v| v.to_str().ok()); + let response = match Encoding::from_accept(accept) { + Encoding::Ssz => ssz_response(envelope.to_ssz()), + Encoding::Json => crate::json_response(ResponseEnvelope { + version: ForkName::Gloas.as_str(), + execution_optimistic: envelope_is_optimistic(&store, root), + finalized: is_finalized(&store, slot), + data: envelope, + }), + }; + + with_consensus_version(response, ForkName::Gloas) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::test_utils::{beacon_fixture, gloas_envelope, gloas_fixture}; + use axum::{body::Body, http::Request, http::StatusCode}; + use ethlambda_types::{beacon::fork_choice::PayloadStatusEnum, primitives::H256}; + use http_body_util::BodyExt as _; + use tower::ServiceExt as _; + + async fn get(store: Store, uri: &str, accept: Option<&str>) -> axum::response::Response { + let mut request = Request::builder().uri(uri); + if let Some(accept) = accept { + request = request.header("accept", accept); + } + routes() + .with_state(store) + .oneshot(request.body(Body::empty()).unwrap()) + .await + .unwrap() + } + + fn url(root: H256) -> String { + format!("/eth/v1/beacon/execution_payload_envelopes/0x{root:x}") + } + + async fn json_of(response: axum::response::Response) -> serde_json::Value { + let body = response.into_body().collect().await.unwrap().to_bytes(); + serde_json::from_slice(&body).unwrap() + } + + #[tokio::test] + async fn a_stored_envelope_is_served_as_json_with_the_flags() { + let mut fixture = gloas_fixture(); + let (root, slot) = fixture.g1; + fixture + .store + .insert_verified_payload(slot, &gloas_envelope(root, slot)); + + // No verdict recorded: NOT_VALIDATED, so optimistic. + let response = get(fixture.store.clone(), &url(root), None).await; + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + response.headers().get("eth-consensus-version").unwrap(), + "gloas" + ); + let json = json_of(response).await; + assert_eq!(json["version"], "gloas"); + assert_eq!(json["execution_optimistic"], true); + assert_eq!(json["finalized"], false); + assert_eq!(json["data"]["message"]["builder_index"], "3"); + + fixture + .store + .insert_beacon_block_payload_status(root, slot, PayloadStatusEnum::Valid); + let json = json_of(get(fixture.store, &url(root), None).await).await; + assert_eq!(json["execution_optimistic"], false); + } + + #[tokio::test] + async fn a_stored_envelope_is_served_as_ssz_on_request() { + let mut fixture = gloas_fixture(); + let (root, slot) = fixture.g1; + let envelope = gloas_envelope(root, slot); + fixture.store.insert_verified_payload(slot, &envelope); + + let response = get(fixture.store, &url(root), Some("application/octet-stream")).await; + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + response + .headers() + .get(axum::http::header::CONTENT_TYPE) + .unwrap(), + crate::SSZ_CONTENT_TYPE + ); + assert_eq!( + response.headers().get("eth-consensus-version").unwrap(), + "gloas" + ); + let body = response.into_body().collect().await.unwrap().to_bytes(); + assert_eq!(body.as_ref(), envelope.to_ssz().as_slice()); + } + + #[tokio::test] + async fn a_gloas_block_without_a_stored_envelope_is_a_404() { + let fixture = gloas_fixture(); + let response = get(fixture.store, &url(fixture.g1.0), None).await; + assert_eq!(response.status(), StatusCode::NOT_FOUND); + let json = json_of(response).await; + assert_eq!(json["message"], "Execution payload envelope not found"); + } + + #[tokio::test] + async fn an_unknown_block_is_a_404_and_a_malformed_id_a_400() { + let fixture = beacon_fixture(64); + let unknown = url(H256::from([0xee; 32])); + assert_eq!( + get(fixture.store.clone(), &unknown, None).await.status(), + StatusCode::NOT_FOUND + ); + assert_eq!( + get( + fixture.store, + "/eth/v1/beacon/execution_payload_envelopes/nope", + None + ) + .await + .status(), + StatusCode::BAD_REQUEST + ); + } + + /// The specification's error list has no status for a pre-gloas block, and + /// such a block has no envelope, so it reads as not found. + #[tokio::test] + async fn a_pre_gloas_block_has_no_envelope() { + let fixture = gloas_fixture(); + let response = get(fixture.store, &url(fixture.head_root), None).await; + assert_eq!(response.status(), StatusCode::NOT_FOUND); + } +} diff --git a/crates/net/rpc/src/beacon/headers.rs b/crates/net/rpc/src/beacon/headers.rs index 5345e4e74..de3a178f6 100644 --- a/crates/net/rpc/src/beacon/headers.rs +++ b/crates/net/rpc/src/beacon/headers.rs @@ -66,7 +66,7 @@ async fn get_header(Path(block_id): Path, State(store): State) -> // which is unreachable here: this router is only ever mounted on a beacon // store. crate::json_response(serde_json::json!({ - "execution_optimistic": store.is_beacon_optimistic(root), + "execution_optimistic": crate::shared::optimistic::block_is_optimistic(&store, root), "finalized": is_finalized(&store, block.slot()), "data": { "root": root, diff --git a/crates/net/rpc/src/beacon/mod.rs b/crates/net/rpc/src/beacon/mod.rs index da2afe031..fea683c3f 100644 --- a/crates/net/rpc/src/beacon/mod.rs +++ b/crates/net/rpc/src/beacon/mod.rs @@ -11,13 +11,14 @@ use axum::{ response::{IntoResponse, Response}, }; use ethlambda_storage::Store; -use ethlambda_types::beacon::{config::Config, primitives::Epoch}; +use ethlambda_types::beacon::{config::Config, fork::ForkName, primitives::Epoch}; use serde::Serialize; use crate::shared::block_id::IdError; pub(crate) mod blocks; pub(crate) mod config; +pub(crate) mod envelopes; pub(crate) mod events; pub(crate) mod genesis; pub(crate) mod headers; @@ -64,23 +65,33 @@ pub(crate) enum ApiError { ServiceUnavailable(&'static str), } -/// Refuses `epoch` when the node's schedule places it at a fork the node does -/// not follow (gloas, see -/// [`ethlambda_types::beacon::fork::ForkName::is_followed`]). +/// Refuses `epoch` when the node's schedule places it at gloas: validator +/// duties are not supported from that fork. /// -/// The head is a fulu state, so an answer for a gloas epoch would carry fulu's -/// shapes under gloas's fork (a vote a validator client would sign as valid, -/// an aggregate of the wrong shape). Refused by name, as the submit endpoints -/// refuse the gloas header. -pub(crate) fn refuse_unfollowed_epoch( +/// The node follows gloas but only as a follower. It has no builder bid +/// handling, no payload-timeliness committee duty and no gloas proposer +/// flow, and its attestation pool holds electra-shaped votes. An answer for a +/// gloas epoch would carry fulu's shapes under gloas's fork (a vote a +/// validator client would sign as valid, an aggregate of the wrong shape), so +/// it is refused by name, as the submit endpoints refuse the gloas header. +/// Deliberately not [`ethlambda_types::beacon::fork::ForkName::is_followed`]: +/// following a fork and serving its validator duties are different questions. +pub(crate) fn refuse_validator_duties_from_gloas( config: &Config, epoch: Epoch, message: &'static str, ) -> Result<(), ApiError> { - if config.fork_at_epoch(epoch).is_followed() { - Ok(()) - } else { - Err(ApiError::BadRequest(message)) + match config.fork_at_epoch(epoch) { + ForkName::Phase0 + | ForkName::Altair + | ForkName::Bellatrix + | ForkName::Capella + | ForkName::Deneb + | ForkName::Electra + | ForkName::Fulu => Ok(()), + // `Config::fork_at_epoch` never returns Lean; refused like gloas + // rather than answered. + ForkName::Gloas | ForkName::Lean => Err(ApiError::BadRequest(message)), } } @@ -119,6 +130,7 @@ pub(crate) fn routes(version: &'static str, peer_id: String) -> Router { Router::new() .merge(blocks::routes()) .merge(events::routes()) + .merge(envelopes::routes()) .merge(headers::routes()) .merge(states::routes()) .merge(genesis::routes()) @@ -146,6 +158,18 @@ mod tests { assert_eq!(json["message"], "block not found"); } + #[test] + fn validator_duties_are_refused_from_the_gloas_epoch_even_though_gloas_is_followed() { + let mut config = Config::mainnet(); + config.gloas_fork_epoch = 10; + assert!(ForkName::Gloas.is_followed()); + let refuse = |epoch| refuse_validator_duties_from_gloas(&config, epoch, "refused"); + assert!(refuse(0).is_ok()); + assert!(refuse(9).is_ok()); + assert!(matches!(refuse(10), Err(ApiError::BadRequest("refused")))); + assert!(matches!(refuse(11), Err(ApiError::BadRequest("refused")))); + } + #[test] fn a_malformed_id_is_a_400_and_an_absent_one_a_404() { assert_eq!( diff --git a/crates/net/rpc/src/beacon/pool.rs b/crates/net/rpc/src/beacon/pool.rs index 6d2222847..a86e94a96 100644 --- a/crates/net/rpc/src/beacon/pool.rs +++ b/crates/net/rpc/src/beacon/pool.rs @@ -49,7 +49,7 @@ use ethlambda_types::{ use serde::{Deserialize, Serialize}; use tracing::{debug, warn}; -use crate::beacon::{ApiError, refuse_unfollowed_epoch, validator::head}; +use crate::beacon::{ApiError, refuse_validator_duties_from_gloas, validator::head}; pub(crate) fn routes() -> Router { Router::new() @@ -377,7 +377,7 @@ async fn get_aggregate_attestation( // The pool holds electra-shaped votes, and the response is labelled with // the slot's fork, so a gloas slot would be served as gloas with a body of // the wrong shape. - if let Err(err) = refuse_unfollowed_epoch( + if let Err(err) = refuse_validator_duties_from_gloas( &store.config(), compute_epoch_at_slot(query.slot), "gloas aggregates are not supported", diff --git a/crates/net/rpc/src/beacon/proposal.rs b/crates/net/rpc/src/beacon/proposal.rs index fb0162b37..6de77cf45 100644 --- a/crates/net/rpc/src/beacon/proposal.rs +++ b/crates/net/rpc/src/beacon/proposal.rs @@ -369,11 +369,18 @@ async fn build_payload( ); ExecutionAddress::ZERO }); - let el_hash = |root: H256| store.beacon_el_block_hash(root).unwrap_or(H256::ZERO); + // The same rule `forkchoiceUpdated` uses for a checkpoint block (a gloas + // one is its bid's parent hash), so the two calls never disagree. let forkchoice = ForkchoiceStateV1 { head_block_hash: inputs.parent_hash, - safe_block_hash: el_hash(store.beacon_justified_checkpoint().root), - finalized_block_hash: el_hash(store.beacon_finalized_checkpoint().root), + safe_block_hash: ethlambda_blockchain::checkpoint_hash( + store, + store.beacon_justified_checkpoint().root, + ), + finalized_block_hash: ethlambda_blockchain::checkpoint_hash( + store, + store.beacon_finalized_checkpoint().root, + ), }; let attributes = PayloadAttributesV3 { timestamp: inputs.timestamp, diff --git a/crates/net/rpc/src/beacon/states.rs b/crates/net/rpc/src/beacon/states.rs index b23cd3040..b4dda41f4 100644 --- a/crates/net/rpc/src/beacon/states.rs +++ b/crates/net/rpc/src/beacon/states.rs @@ -96,7 +96,7 @@ async fn get_state( // borrow. Encoding::Json => crate::json_response(Envelope { version: fork.as_str(), - execution_optimistic: store.is_beacon_optimistic(root), + execution_optimistic: crate::shared::optimistic::block_is_optimistic(&store, root), finalized: is_finalized(&store, state.slot()), data: state.as_ref(), }), @@ -113,7 +113,7 @@ async fn get_fork(Path(state_id): Path, State(store): State) -> R Err(err) => return err.into_response(), }; crate::json_response(serde_json::json!({ - "execution_optimistic": store.is_beacon_optimistic(root), + "execution_optimistic": crate::shared::optimistic::block_is_optimistic(&store, root), "finalized": is_finalized(&store, state.slot()), "data": state.fork(), })) @@ -133,7 +133,7 @@ async fn get_finality_checkpoints( // the state is the only place a *previous* justified checkpoint is kept // at all: the store keeps one justified row and one finalized row. crate::json_response(serde_json::json!({ - "execution_optimistic": store.is_beacon_optimistic(root), + "execution_optimistic": crate::shared::optimistic::block_is_optimistic(&store, root), "finalized": is_finalized(&store, state.slot()), "data": { "previous_justified": state.previous_justified_checkpoint(), @@ -343,7 +343,7 @@ fn validators_response(store: &Store, state_id: &str, request: ValidatorsRequest .collect(); crate::json_response(serde_json::json!({ - "execution_optimistic": store.is_beacon_optimistic(root), + "execution_optimistic": crate::shared::optimistic::block_is_optimistic(store, root), "finalized": is_finalized(store, state.slot()), "data": entries, })) @@ -389,7 +389,7 @@ fn validator_response( .map_err(|_| ApiError::Internal("the registry and balances disagree in length"))?; let status = ValidatorStatus::of(validator, balance, compute_epoch_at_slot(state.slot())); Ok(serde_json::json!({ - "execution_optimistic": store.is_beacon_optimistic(root), + "execution_optimistic": crate::shared::optimistic::block_is_optimistic(store, root), "finalized": is_finalized(store, state.slot()), "data": ValidatorEntry { index, diff --git a/crates/net/rpc/src/beacon/validator.rs b/crates/net/rpc/src/beacon/validator.rs index 66d17ceba..7df755fb4 100644 --- a/crates/net/rpc/src/beacon/validator.rs +++ b/crates/net/rpc/src/beacon/validator.rs @@ -38,7 +38,7 @@ use ethlambda_state_transition::beacon::{ helpers::altair::compute_sync_committee_period, }; -use crate::beacon::{ApiError, refuse_unfollowed_epoch}; +use crate::beacon::{ApiError, refuse_validator_duties_from_gloas}; pub(crate) fn routes() -> Router { Router::new() @@ -163,7 +163,7 @@ fn sync_duties( } Ok(serde_json::json!({ - "execution_optimistic": store.is_beacon_optimistic(head_root), + "execution_optimistic": crate::shared::optimistic::block_is_optimistic(store, head_root), "data": duties, })) } @@ -449,7 +449,7 @@ fn proposer_duties( dependent: DependentRoot, ) -> Result { let epoch = parse_epoch(epoch)?; - refuse_unfollowed_epoch( + refuse_validator_duties_from_gloas( &store.config(), epoch, "gloas proposer duties are not supported", @@ -551,7 +551,7 @@ fn attester_duties( .map(|index| index.parse::()) .collect::, _>>() .map_err(|_| ApiError::BadRequest("invalid validator index"))?; - refuse_unfollowed_epoch( + refuse_validator_duties_from_gloas( &store.config(), epoch, "gloas attester duties are not supported", @@ -642,7 +642,7 @@ async fn get_attestation_data( } fn attestation_data(store: &Store, slot: Slot) -> Result { - refuse_unfollowed_epoch( + refuse_validator_duties_from_gloas( &store.config(), compute_epoch_at_slot(slot), "gloas attestation data is not supported", diff --git a/crates/net/rpc/src/lib.rs b/crates/net/rpc/src/lib.rs index a3b8dca99..c63d9327e 100644 --- a/crates/net/rpc/src/lib.rs +++ b/crates/net/rpc/src/lib.rs @@ -246,7 +246,9 @@ pub(crate) mod test_utils { use ethlambda_types::{ beacon::{ config::Config, - containers::{BeaconState, SignedBeaconBlock, phase0, shared::BeaconBlockHeader}, + containers::{ + BeaconState, SignedBeaconBlock, gloas, phase0, shared::BeaconBlockHeader, + }, preset, }, block::{Block, BlockBody, BlockHeader}, @@ -469,6 +471,117 @@ pub(crate) mod test_utils { } } + /// [`beacon_fixture`]'s chain with three gloas blocks on top of its head. + /// + /// ```text + /// anchor(phase0) - head(phase0) - g1 - g2 - g3 + /// ``` + /// + /// `g1` builds on a pre-gloas parent, so on its FULL branch. `g2`'s bid + /// names `g1`'s payload as its parent block hash (FULL). `g3`'s names + /// something else (EMPTY), so it builds on `g1`'s payload through `g2`'s + /// skipped one. No verdict or envelope is recorded for any of them. + pub(crate) struct GloasFixture { + pub(crate) store: Store, + pub(crate) head_root: H256, + pub(crate) g1: (H256, u64), + pub(crate) g2: (H256, u64), + pub(crate) g3: (H256, u64), + } + + /// A gloas block whose bid names `parent_block_hash` and `block_hash`. + pub(crate) fn gloas_beacon_block( + slot: u64, + parent_root: H256, + parent_block_hash: H256, + block_hash: H256, + ) -> SignedBeaconBlock { + let mut body = gloas::BeaconBlockBody::default(); + let bid = &mut body.signed_execution_payload_bid.message; + bid.parent_block_hash = parent_block_hash; + bid.block_hash = block_hash; + SignedBeaconBlock::Gloas(gloas::SignedBeaconBlock { + message: gloas::BeaconBlock { + slot, + proposer_index: 0, + parent_root, + state_root: H256::ZERO, + body, + }, + signature: Default::default(), + }) + } + + pub(crate) fn gloas_fixture() -> GloasFixture { + let BeaconFixture { + mut store, + head_root, + head_slot, + .. + } = beacon_fixture(64); + + let hash = |n: u8| H256::from([n; 32]); + let mut chain = Vec::new(); + let mut parent = (head_root, head_slot); + for (parent_hash, block_hash) in + [(hash(0), hash(1)), (hash(1), hash(2)), (hash(9), hash(3))] + { + let slot = parent.1 + 1; + let block = gloas_beacon_block(slot, parent.0, parent_hash, block_hash); + let root = block.message_hash_tree_root(); + store + .insert_signed_block(root, block) + .expect("insert gloas block"); + chain.push((root, slot)); + parent = (root, slot); + } + + GloasFixture { + store, + head_root, + g1: chain[0], + g2: chain[1], + g3: chain[2], + } + } + + /// A signed envelope for `root` with every other field at its zero value. + pub(crate) fn gloas_envelope(root: H256, slot: u64) -> gloas::SignedExecutionPayloadEnvelope { + let payload = gloas::ExecutionPayload { + parent_hash: Default::default(), + fee_recipient: Default::default(), + state_root: Default::default(), + receipts_root: Default::default(), + logs_bloom: vec![0u8; preset::BYTES_PER_LOGS_BLOOM] + .try_into() + .expect("built at exactly BYTES_PER_LOGS_BLOOM"), + prev_randao: Default::default(), + block_number: 7, + gas_limit: 0, + gas_used: 0, + timestamp: 0, + extra_data: Default::default(), + base_fee_per_gas: Default::default(), + block_hash: Default::default(), + transactions: Default::default(), + withdrawals: Default::default(), + blob_gas_used: 0, + excess_blob_gas: 0, + block_access_list: Default::default(), + slot_number: slot, + }; + gloas::SignedExecutionPayloadEnvelope { + message: gloas::ExecutionPayloadEnvelope { + payload, + execution_requests: Default::default(), + builder_index: 3, + beacon_block_root: root, + parent_beacon_block_root: H256::ZERO, + }, + signature: Default::default(), + } + } + /// A beacon store whose anchor, and so head, is `state`, under a block at /// the state's slot. Returns the store and that block's root. /// diff --git a/crates/net/rpc/src/shared/mod.rs b/crates/net/rpc/src/shared/mod.rs index b74db81db..a42313af4 100644 --- a/crates/net/rpc/src/shared/mod.rs +++ b/crates/net/rpc/src/shared/mod.rs @@ -2,3 +2,4 @@ pub(crate) mod block_id; pub(crate) mod content; +pub(crate) mod optimistic; diff --git a/crates/net/rpc/src/shared/optimistic.rs b/crates/net/rpc/src/shared/optimistic.rs new file mode 100644 index 000000000..e7b127ad1 --- /dev/null +++ b/crates/net/rpc/src/shared/optimistic.rs @@ -0,0 +1,223 @@ +//! The `execution_optimistic` flag a Beacon API response carries. +//! +//! Before gloas a block's payload ran inside the block, so the flag is whether +//! that block sits in the store's optimistic set. From gloas on the payload +//! travels in a separate envelope that may arrive late or never, so two +//! different questions share the flag: +//! +//! - an envelope response is optimistic iff that payload's verdict is not +//! `VALID` ([`envelope_is_optimistic`]); +//! - a block, state or header response is optimistic iff the latest FULL +//! payload the block builds on is not `VALID` ([`block_is_optimistic`]). +//! The block's own payload does not count: the block itself was imported +//! without it, and a withheld payload must not make the chain look +//! unverified. +//! +//! Everything here only reads the store, and an answer that cannot be worked +//! out is optimistic: this flag gates signing, so an unknown gets the cautious +//! reading. The payload links are therefore looked up or derived here instead +//! of through fork choice, whose `ensure_payload_link` records what it derives +//! and panics on a failed read. + +use ethlambda_storage::Store; +use ethlambda_types::{ + beacon::{ + containers::SignedBeaconBlock, + fork_choice::{BlockPayloadLink, PayloadStatus, PayloadStatusEnum}, + }, + primitives::H256, +}; + +/// `root`'s payload link: the one fork choice recorded, or one derived from the +/// stored blocks without recording it. `None` when it cannot be worked out. +/// +/// The derivation is `derive_payload_link`'s: a gloas block's parent status is +/// FULL when its bid names the parent's block hash (or the parent is +/// pre-gloas), and unknown when the parent is not stored, which is the anchor. +fn payload_link(store: &Store, root: H256) -> Option { + if let Some(link) = store.payload_link(&root) { + return Some(link); + } + match store.get_signed_block(&root).ok()?? { + SignedBeaconBlock::Phase0(_) + | SignedBeaconBlock::Altair(_) + | SignedBeaconBlock::Bellatrix(_) + | SignedBeaconBlock::Capella(_) + | SignedBeaconBlock::Deneb(_) + | SignedBeaconBlock::Electra(_) + | SignedBeaconBlock::Fulu(_) => Some(BlockPayloadLink::PreGloas), + SignedBeaconBlock::Gloas(block) => { + let parent_status = match store.get_signed_block(&block.message.parent_root).ok()? { + None => None, + Some(SignedBeaconBlock::Gloas(parent)) => { + let wanted = block + .message + .body + .signed_execution_payload_bid + .message + .parent_block_hash; + let held = parent + .message + .body + .signed_execution_payload_bid + .message + .block_hash; + Some(if wanted == held { + PayloadStatus::Full + } else { + PayloadStatus::Empty + }) + } + Some(_) => Some(PayloadStatus::Full), + }; + Some(BlockPayloadLink::Gloas { parent_status }) + } + // A lean block is never in a beacon store. + SignedBeaconBlock::Lean(_) => None, + } +} + +/// Whether the payload of the block at `root` is not (known to be) `VALID`. +/// +/// A pre-gloas block's payload is its own, so it reads the optimistic set; a +/// gloas root reads its recorded verdict, where none recorded is +/// `NOT_VALIDATED`. Unknown links are not valid. +fn payload_is_not_valid(store: &Store, root: H256) -> bool { + match payload_link(store, root) { + Some(BlockPayloadLink::PreGloas) => store.is_beacon_optimistic(root), + Some(BlockPayloadLink::Gloas { .. }) => { + store.beacon_block_payload_status(root) != PayloadStatusEnum::Valid + } + None => true, + } +} + +/// Whether the payload of the block at `root` is not yet vouched for by the +/// execution layer. +/// +/// An unrecorded gloas verdict reads as `NOT_VALIDATED`, so an envelope the +/// store holds but has no verdict for is optimistic. +pub(crate) fn envelope_is_optimistic(store: &Store, root: H256) -> bool { + payload_is_not_valid(store, root) +} + +/// Whether the block at `root` rests on a payload that is not `VALID`. +/// +/// Walks back from `root` through the blocks whose payload branch the child +/// skipped (`Empty`) until it reaches the FULL payload the chain actually +/// builds on, then reads that payload's verdict. A pre-gloas block is its own +/// payload, so it reads the optimistic set as before. A chain that runs out +/// (an anchor whose parent is below the retained window) has nothing left to +/// doubt, so it is not optimistic. A block or link that cannot be read is +/// optimistic. +pub(crate) fn block_is_optimistic(store: &Store, root: H256) -> bool { + let mut current = root; + loop { + let Some(link) = payload_link(store, current) else { + return true; + }; + if !link.is_gloas() { + return store.is_beacon_optimistic(current); + } + let parent = match store.get_signed_block(¤t) { + Ok(Some(block)) => block.parent_root(), + Ok(None) | Err(_) => return true, + }; + match link.parent_status() { + Some(PayloadStatus::Full) => return payload_is_not_valid(store, parent), + Some(PayloadStatus::Empty) => current = parent, + // A pending parent status never comes out of a block's own bid. + Some(PayloadStatus::Pending) | None => return false, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::test_utils::{beacon_fixture, gloas_fixture}; + + #[test] + fn a_pre_gloas_block_reads_the_optimistic_set() { + let mut fixture = beacon_fixture(64); + let root = fixture.head_root; + assert!(!block_is_optimistic(&fixture.store, root)); + + fixture + .store + .insert_beacon_optimistic_root(root, fixture.head_slot); + assert!(block_is_optimistic(&fixture.store, root)); + } + + #[test] + fn a_gloas_block_is_optimistic_iff_the_payload_it_builds_on_is_not_valid() { + let mut fixture = gloas_fixture(); + let (g1, g1_slot) = fixture.g1; + let (g2, _) = fixture.g2; + let (g3, _) = fixture.g3; + + // g1 builds on its pre-gloas parent, which is not optimistic. + assert!(!block_is_optimistic(&fixture.store, g1)); + // g2 builds on g1's payload, which has no verdict yet. + assert!(block_is_optimistic(&fixture.store, g2)); + // g3 skipped g2's payload, so it too rests on g1's. + assert!(block_is_optimistic(&fixture.store, g3)); + + fixture + .store + .insert_beacon_block_payload_status(g1, g1_slot, PayloadStatusEnum::Valid); + assert!(!block_is_optimistic(&fixture.store, g2)); + assert!(!block_is_optimistic(&fixture.store, g3)); + + // g1's own payload being doubtful does not make g1 optimistic. + fixture + .store + .insert_beacon_block_payload_status(g1, g1_slot, PayloadStatusEnum::Syncing); + assert!(!block_is_optimistic(&fixture.store, g1)); + assert!(block_is_optimistic(&fixture.store, g2)); + } + + #[test] + fn a_gloas_block_on_an_optimistic_pre_gloas_parent_is_optimistic() { + let mut fixture = gloas_fixture(); + let head = fixture.head_root; + fixture.store.insert_beacon_optimistic_root(head, 65); + assert!(block_is_optimistic(&fixture.store, fixture.g1.0)); + } + + #[test] + fn a_block_the_store_cannot_produce_is_optimistic() { + let fixture = gloas_fixture(); + let unknown = H256::from([0xee; 32]); + assert!(block_is_optimistic(&fixture.store, unknown)); + assert!(envelope_is_optimistic(&fixture.store, unknown)); + } + + #[test] + fn a_gloas_anchor_with_an_unknown_parent_is_not_optimistic() { + let mut fixture = gloas_fixture(); + let block = crate::test_utils::gloas_beacon_block( + 500, + H256::from([0xab; 32]), + H256::ZERO, + H256::ZERO, + ); + let root = block.message_hash_tree_root(); + fixture + .store + .insert_signed_block(root, block) + .expect("insert"); + + assert!(!block_is_optimistic(&fixture.store, root)); + } + + #[test] + fn judging_a_block_records_nothing_in_the_store() { + let fixture = gloas_fixture(); + let (g2, _) = fixture.g2; + // Drop whatever import recorded, so the helper has to derive. + let before = fixture.store.payload_link(&g2); + block_is_optimistic(&fixture.store, g2); + assert_eq!(fixture.store.payload_link(&g2), before); + } +} diff --git a/crates/storage/src/api/tables.rs b/crates/storage/src/api/tables.rs index 42f74fc09..86ed0d1a4 100644 --- a/crates/storage/src/api/tables.rs +++ b/crates/storage/src/api/tables.rs @@ -61,6 +61,25 @@ pub enum Table { /// re-deriving anything. Emptied by the replay that verifies a row and by /// the finality eviction that gives up on one. PendingDataColumns, + /// Verified gloas execution payload envelopes: + /// (slot || beacon_block_root) -> `SignedExecutionPayloadEnvelope` + /// + /// A row exists only for an envelope that passed verification, so the + /// table is the durable form of the store's `verified_payloads` set: a + /// restart rebuilds that set from it. Keyed slot-first, with the slot of + /// the block the envelope fulfills (the envelope carries none itself), so + /// a by-range handler can scan a slot window, like `DataColumns`. + /// + /// Never pruned, as beacon block rows are not either. + ExecutionPayloadEnvelopes, + /// Block timeliness: beacon_block_root -> one byte per deadline (0 or 1) + /// + /// The durable form of the store's `block_timeliness` map. Fork choice + /// reads it on the ordinary head-computation path, and a block whose entry + /// is lost fails the whole head walk, so it cannot be left to memory. + /// + /// Never pruned; a row is a few bytes. + BlockTimeliness, /// Beacon per-block unrealized justified checkpoint: root -> /// `(slot, Checkpoint)`. /// @@ -100,7 +119,7 @@ pub enum Table { } /// All table variants. -pub const ALL_TABLES: [Table; 11] = [ +pub const ALL_TABLES: [Table; 13] = [ Table::BlockHeaders, Table::BlockBodies, Table::BlockProof, @@ -111,6 +130,8 @@ pub const ALL_TABLES: [Table; 11] = [ Table::LiveChain, Table::DataColumns, Table::PendingDataColumns, + Table::ExecutionPayloadEnvelopes, + Table::BlockTimeliness, Table::BeaconUnrealizedJustifications, ]; @@ -128,6 +149,8 @@ impl Table { Table::LiveChain => "live_chain", Table::DataColumns => "data_columns", Table::PendingDataColumns => "pending_data_columns", + Table::ExecutionPayloadEnvelopes => "execution_payload_envelopes", + Table::BlockTimeliness => "block_timeliness", Table::BeaconUnrealizedJustifications => "beacon_unrealized_justifications", } } diff --git a/crates/storage/src/error.rs b/crates/storage/src/error.rs index 432c1582f..7e35c2709 100644 --- a/crates/storage/src/error.rs +++ b/crates/storage/src/error.rs @@ -81,4 +81,13 @@ pub enum Error { .checkpoint.slot, .checkpoint.root )] AnchorStateLost { checkpoint: Checkpoint }, + /// A stored data column sidecar did not decode as the fork its slot + /// names. The table keeps no tag beside the bytes, so the fork is read off + /// the row's slot and the config's schedule; a disagreement means the + /// row was written under another schedule or corrupted at rest. + #[error("data column sidecar at slot {slot} does not decode as a {fork:?} sidecar")] + UndecodableDataColumn { + slot: u64, + fork: ethlambda_types::beacon::fork::ForkName, + }, } diff --git a/crates/storage/src/store.rs b/crates/storage/src/store.rs index 36e8c259e..39b17a44f 100644 --- a/crates/storage/src/store.rs +++ b/crates/storage/src/store.rs @@ -19,10 +19,16 @@ use ethlambda_types::{ beacon::{ config::Config, constants::NUM_BLOCK_TIMELINESS_DEADLINES, - containers::{BeaconState, Checkpoint as BeaconCheckpoint, SignedBeaconBlock}, + containers::{ + BeaconState, Checkpoint as BeaconCheckpoint, DataColumnSidecar, SignedBeaconBlock, + gloas, + }, fork::ForkName, - fork_choice::{BlockPayloadLink, LatestMessage, PayloadStatusV1, PowBlock}, - preset::{Preset, SLOTS_PER_EPOCH}, + fork_choice::{ + BlockPayloadLink, LatestMessage, PayloadStatus, PayloadStatusEnum, PayloadStatusV1, + PowBlock, + }, + preset::{PTC_SIZE, Preset, SLOTS_PER_EPOCH}, primitives::ExecutionBlockHash, }, block::{ @@ -649,17 +655,18 @@ impl GossipSignatureBuffer { } /// Beacon fork-choice state that is per-slot or per-epoch scratch rather than -/// chain history: most of it does not survive a restart, and is not worth the -/// write amplification of persisting. +/// chain history: apart from the entries named below, nothing here +/// survives a restart, and nothing else here is worth the write +/// amplification of persisting. /// -/// `proposer_boost_root` resets every slot, `block_timeliness` is read only by -/// the same-slot reorg helpers, `equivocating_indices` is rebuilt by replaying -/// attester slashings on sync, `latest_messages` is rebuilt by the first epoch -/// of attestations, and `pow_blocks` stands in for a call to an execution -/// client that a restarted node would simply make again. `optimistic_roots` -/// and `payload_statuses` are likewise answers an execution client can be -/// asked for again, and `el_block_hashes` and `payload_links` are caches over -/// data already decodable from the block itself. +/// `proposer_boost_root` resets every slot, pre-gloas `block_timeliness` is +/// read only by the same-slot reorg helpers, `equivocating_indices` is rebuilt +/// by replaying attester slashings on sync, `latest_messages` is rebuilt by +/// the first epoch of attestations, and `pow_blocks` stands in for a call to +/// an execution client that a restarted node would simply make again. +/// `optimistic_roots` and `payload_statuses` are likewise answers an execution +/// client can be asked for again, and `el_block_hashes` and `payload_links` +/// are caches over data already decodable from the block itself. /// /// `unrealized_justifications` is an exception: it is a write-through /// cache over `Table::BeaconUnrealizedJustifications`, not scratch that a @@ -671,37 +678,36 @@ impl GossipSignatureBuffer { /// [`Store::unrealized_justification`] and /// [`Store::set_unrealized_justification`]. /// -/// Gloas's own four (`verified_payloads`, `payload_timeliness_vote`, -/// `payload_data_availability_vote`, and `block_timeliness`'s widened shape) -/// do not all fit the "a restarted node just gets it again" pattern above, -/// even though they live in this same scratch. `payload_timeliness_vote` and -/// `payload_data_availability_vote` do: they are the payload timeliness -/// committee's per-slot ballot, gone the moment the slot it was cast for is -/// no longer being decided, the same kind of throwaway `pow_blocks` is. -/// `verified_payloads` and `block_timeliness` do not, and both must be -/// persisted before this runs against a live, restarting node: losing -/// `verified_payloads` on restart does not error, it silently drops every -/// full payload branch from fork choice (`is_payload_verified` answers -/// `false` for a block this store genuinely verified before restarting), -/// and losing a `block_timeliness` entry the gloas fork choice's -/// `should_apply_proposer_boost` needs makes that call fail outright with -/// `Error::SpecAssert`. Unlike pre-gloas, where `block_timeliness` is read -/// only by the same-slot reorg helpers, gloas's own weight computation -/// reads it on the ordinary head-computation path, so that failure aborts -/// the whole `gloas_get_weight`/`gloas_get_head` walk, not just an optional -/// reorg decision. Nothing in this crate persists either yet. +/// Gloas's `verified_payloads` and `block_timeliness` are exceptions too: they +/// live in this scratch for reading, but a gloas block's entries are written +/// through to `Table::ExecutionPayloadEnvelopes` and `Table::BlockTimeliness`, +/// and [`Store::from_db_state`] rebuilds both for every block at or above the +/// finalized one. Losing `verified_payloads` would not error, it would +/// silently drop every full payload branch from fork choice. Losing a +/// `block_timeliness` entry does not error either: a missing entry reads as +/// not timely by either deadline, which quietly changes what +/// `should_apply_proposer_boost` withholds. Pre-gloas timeliness is never +/// persisted: nothing reads it after a restart. +/// +/// `payload_timeliness_vote` and `payload_data_availability_vote` stay in +/// memory on purpose. They are the payload timeliness committee's per-slot +/// ballot, read only while the slot they were cast for is being decided. A +/// resumed store reseeds them empty (every member `None`) for each restored +/// gloas block, which is what a block just imported holds, and the next +/// block's payload attestations refill them: the loss lasts about a slot. /// /// Most of this is uncapped: the per-validator maps are bounded by the -/// validator set, and the per-block ones (`block_timeliness`, -/// `unrealized_justifications`, and gloas's own three) grow with the blocks -/// this process has imported. Three are the exception, `el_block_hashes`, -/// `payload_links` and `optimistic_roots`, each pruned to the unfinalized -/// window by its own `prune_*` method. All three fill on a path that runs for -/// the whole life of the process and has no other way of emptying them: -/// `forkchoiceUpdated` reads the first once per head move, the head walk -/// reads the second for every block it visits, and an execution client doing a -/// long state sync answers `NOT_VALIDATED` to every block, which writes the -/// third once per import. +/// validator set, and the per-block ones (`block_timeliness` and gloas's own +/// three) grow with the blocks this process has imported. Three are the +/// exception, `el_block_hashes`, `payload_links` and `optimistic_roots`, each +/// pruned to the unfinalized window by its own `prune_*` method. All three +/// fill on a path that runs for the whole life of the process and has no other +/// way of emptying them: `forkchoiceUpdated` reads the first once per head +/// move, the head walk reads the second for every block it visits, and an +/// execution client doing a long state sync answers `NOT_VALIDATED` to every +/// block, which writes the third once per import. `unrealized_justifications` +/// is bounded too: [`Store::prune_unrealized_justifications`] evicts it along +/// with the table rows it caches. #[derive(Default)] pub(crate) struct BeaconScratch { pub(crate) proposer_boost_root: H256, @@ -720,13 +726,9 @@ pub(crate) struct BeaconScratch { pub(crate) unrealized_justifications: HashMap, /// Gloas: beacon block roots whose execution payload envelope has been /// locally delivered and verified (`on_execution_payload_envelope`'s own - /// `store.payloads[root] = envelope`), standing in for the specification's - /// `store.payloads` for exactly the question this crate asks of it today, - /// `is_payload_verified`'s membership test. The envelope itself is not - /// kept: nothing here reads one back out yet, and it is not clear this - /// scratch is even where a kept envelope should live, since (see this - /// struct's own doc) `verified_payloads` needs to survive a restart and - /// nothing else here does. + /// `store.payloads[root] = envelope`), the in-memory index of + /// `Table::ExecutionPayloadEnvelopes` for `is_payload_verified`'s + /// membership test. The envelopes themselves are read from the table. pub(crate) verified_payloads: HashSet, /// Each imported block's payload link (which rules it is handled under, /// and which of its parent's payload branches it builds on), so the head @@ -740,6 +742,22 @@ pub(crate) struct BeaconScratch { /// finality. A pruned link is derived again by decoding if anything asks /// for it. pub(crate) payload_links: HashMap, + /// The head node fork choice last picked, as its block root and payload + /// branch: `Full` when the node is the block with the payload revealed, + /// `Empty` when it is the block alone. A pre-gloas head is `Full` (see + /// `fork_choice::get_head_node`). + /// + /// The root is stored with the status so a reader can tell whether the + /// status belongs to the head it sees: the head root lives in the backend + /// and this does not, so the two cannot be written atomically, and a + /// status alone could be paired with the wrong root. Readers compare roots + /// and treat a mismatch as unknown. + /// + /// `None` until the first head computation of this process: nothing + /// persists it, and a resumed directory recomputes it on the first tick + /// rather than storing a second source of truth for what the head walk + /// derives. Readers treat `None` as "not known to be `Full`". + pub(crate) head_node_payload_status: Option<(H256, PayloadStatus)>, /// Gloas: the payload timeliness committee's per-member vote on whether /// the block's payload showed up on time, one slot per `PTC_SIZE` index, /// `None` until that member votes. Keyed by beacon block root, like @@ -775,6 +793,13 @@ pub(crate) struct BeaconScratch { /// forkchoiceUpdated reads it once per head move for the whole life of the /// process. pub(crate) el_block_hashes: HashMap, + /// Gloas: the execution client's verdict on each beacon block's payload, + /// against the block's slot for pruning. Keyed by beacon root because a + /// gloas block's payload hash lives in its bid and envelope rather than in + /// `el_block_hashes`. Read by the attestation gossip rules as the + /// specification's `block_payload_statuses`; an absent root reads as + /// `NOT_VALIDATED`. + pub(crate) block_payload_statuses: HashMap, } /// Encode a LiveChain key (slot, root) to bytes. @@ -863,6 +888,31 @@ fn decode_beacon_block_value(bytes: &[u8]) -> SignedBeaconBlock { SignedBeaconBlock::from_ssz(fork, ssz).expect("valid signed block") } +/// Whether `block` builds on the payload whose hash is `payload_block_hash`: +/// gloas's bid `parent_block_hash` rule for a payload being part of the chain. +fn builds_on_payload(block: &SignedBeaconBlock, payload_block_hash: &ExecutionBlockHash) -> bool { + match block { + SignedBeaconBlock::Gloas(block) => { + block + .message + .body + .signed_execution_payload_bid + .message + .parent_block_hash + == *payload_block_hash + } + // A pre-gloas block cannot follow a gloas one. + SignedBeaconBlock::Phase0(_) + | SignedBeaconBlock::Altair(_) + | SignedBeaconBlock::Bellatrix(_) + | SignedBeaconBlock::Capella(_) + | SignedBeaconBlock::Deneb(_) + | SignedBeaconBlock::Electra(_) + | SignedBeaconBlock::Fulu(_) => false, + SignedBeaconBlock::Lean(_) => Store::lean_only("builds_on_payload"), + } +} + /// `root`'s slot on a beacon chain, read without a `Store`, for the writer /// thread's anchor decision. /// @@ -873,9 +923,10 @@ fn decode_beacon_block_value(bytes: &[u8]) -> SignedBeaconBlock { /// Beacon directories only: it decodes the row as a tagged /// [`SignedBeaconBlock`], which is the wrong shape for a lean directory's bare /// [`BlockHeader`] (unlike [`block_fields`](Store::block_fields), which -/// dispatches on `self.chain`). Nothing does today: [`StateWriter::write`](crate::state_writer)'s -/// non-`Lean` match arm is this function's only caller, so that invariant is -/// enforced by having exactly one caller rather than by a runtime check. +/// dispatches on `self.chain`). Its callers are [`StateWriter::write`](crate::state_writer)'s +/// non-`Lean` match arm and the beacon-only envelope and resume methods on +/// [`Store`], so that invariant is enforced by every caller being a beacon +/// path rather than by a runtime check. pub(crate) fn beacon_block_slot(backend: &dyn StorageBackend, root: &H256) -> Option { let view = backend.begin_read().expect("read view"); view.read_with(Table::BlockHeaders, &root.to_ssz(), |bytes| { @@ -1148,12 +1199,12 @@ impl Store { }; info!(?chain, anchor_slot, "Loaded store from persisted DB state"); - Ok(Some(Self::from_parts( - backend, - Arc::new(config), - chain, - anchor_slot, - ))) + let store = Self::from_parts(backend, Arc::new(config), chain, anchor_slot); + match chain { + Chain::Lean => {} + Chain::Beacon => store.restore_beacon_scratch(), + } + Ok(Some(store)) } /// Checks that both the justified and finalized checkpoints have a @@ -2510,6 +2561,34 @@ impl Store { // ============ Beacon Head ============ + /// The payload status of the head node fork choice last picked, for the + /// store's *current* head root only: `None` before this process has + /// computed a head, and also while the recorded node is not the current + /// head (a head move whose status is not recorded yet). Never a status + /// belonging to another root. + pub fn head_payload_status(&self) -> Option { + let head = self.head().ok()?; + self.payload_status_of_head_node(head) + } + + /// The recorded status, if it was recorded for `head`. + fn payload_status_of_head_node(&self, head: H256) -> Option { + self.beacon + .lock() + .unwrap() + .head_node_payload_status + .filter(|(root, _)| *root == head) + .map(|(_, status)| status) + } + + /// Records the head node fork choice just picked: its block `root` and + /// payload `status`. Order against [`Store::update_checkpoints`] does not + /// matter, since readers only use the status for a head whose root equals + /// the recorded one. + pub fn set_head_payload_status(&self, root: H256, status: PayloadStatus) { + self.beacon.lock().unwrap().head_node_payload_status = Some((root, status)); + } + /// The beacon fork-choice head as `(slot, root)`, or `None` if the head /// row names a block this store has no header for. /// @@ -3348,9 +3427,11 @@ impl Store { // ============ Beacon Fork-Choice Scratch ============ // - // None of these carry a `Result`: the beacon fork choice's error type - // lives in `ethlambda-types` and cannot name a storage error, so these - // accessors take and return plain values like the rest of this scratch. + // The accessors on in-memory scratch carry no `Result`: the beacon fork + // choice's error type lives in `ethlambda-types` and cannot name a storage + // error, so they take and return plain values. The gloas write-through + // ones (`set_gloas_block_timeliness`, `insert_verified_payload`) write a + // row and so `expect` on the backend like the other infallible writers. /// The block root proposer boost currently applies to. Resets every slot. pub fn proposer_boost_root(&self) -> H256 { @@ -3381,7 +3462,10 @@ impl Store { .copied() } - /// Records `root`'s two timeliness deadlines. See [`Store::block_timeliness`]. + /// Records `root`'s two timeliness deadlines in memory only. See + /// [`Store::block_timeliness`]. A gloas block uses + /// [`Store::set_gloas_block_timeliness`] instead, since only gloas reads + /// the entry back after a restart. pub fn set_block_timeliness( &mut self, root: H256, @@ -3394,6 +3478,22 @@ impl Store { .insert(root, timely); } + /// Records a gloas block's timeliness in memory and in + /// `Table::BlockTimeliness`, so a resumed store still has it. + pub fn set_gloas_block_timeliness( + &mut self, + root: H256, + timely: [bool; NUM_BLOCK_TIMELINESS_DEADLINES], + ) { + let entries = vec![(root.to_ssz(), timely.map(u8::from).to_vec())]; + let mut batch = self.backend.begin_write().expect("write batch"); + batch + .put_batch(Table::BlockTimeliness, entries) + .expect("put block timeliness"); + batch.commit().expect("commit"); + self.set_block_timeliness(root, timely); + } + /// Whether `index` has been observed equivocating (via a processed /// attester slashing). pub fn is_equivocating(&self, index: u64) -> bool { @@ -3419,12 +3519,284 @@ impl Store { self.beacon.lock().unwrap().verified_payloads.contains(root) } - /// Gloas: records that `root`'s execution payload envelope has been - /// verified. - pub fn insert_verified_payload(&mut self, root: H256) { + /// Gloas: stores a verified execution payload envelope and records that + /// its block's payload is verified. + /// + /// `slot` is the slot of the block the envelope fulfills, which the + /// caller has already read while verifying the envelope against it; the + /// envelope carries none and the row is keyed by it. + pub fn insert_verified_payload( + &mut self, + slot: u64, + envelope: &gloas::SignedExecutionPayloadEnvelope, + ) { + let root = envelope.message.beacon_block_root; + debug_assert_eq!( + beacon_block_slot(self.backend.as_ref(), &root), + Some(slot), + "the slot must be the envelope's block's own" + ); + let entries = vec![(encode_slot_root_key(slot, &root), envelope.to_ssz())]; + let mut batch = self.backend.begin_write().expect("write batch"); + batch + .put_batch(Table::ExecutionPayloadEnvelopes, entries) + .expect("put execution payload envelope"); + batch.commit().expect("commit"); self.beacon.lock().unwrap().verified_payloads.insert(root); } + /// Gloas: forgets that `root`'s payload is verified and deletes its stored + /// envelope, for a payload the execution layer later found invalid. + /// + /// The FULL node of the block stops existing for fork choice (it is + /// derived from [`Self::has_verified_payload`]) and the envelope is no + /// longer served to peers. Returns whether `root` was verified. + pub fn remove_verified_payload(&mut self, root: &H256) -> bool { + let was_verified = { + let mut scratch = self.beacon.lock().unwrap(); + // The cached hash goes too: nothing may resolve a payload this + // store no longer holds as a FULL node. + scratch.el_block_hashes.remove(root); + scratch.verified_payloads.remove(root) + }; + if let Some(slot) = beacon_block_slot(self.backend.as_ref(), root) { + let keys = vec![encode_slot_root_key(slot, root)]; + let mut batch = self.backend.begin_write().expect("write batch"); + batch + .delete_batch(Table::ExecutionPayloadEnvelopes, keys) + .expect("delete execution payload envelope"); + batch.commit().expect("commit"); + } + was_verified + } + + /// Gloas: the roots whose payload is verified, in no particular order. + pub fn verified_payload_roots(&self) -> Vec { + self.beacon + .lock() + .unwrap() + .verified_payloads + .iter() + .copied() + .collect() + } + + /// Gloas: the verified envelope stored for `root`, if any. + pub fn get_execution_payload_envelope( + &self, + root: &H256, + ) -> Result, Error> { + let Some(slot) = beacon_block_slot(self.backend.as_ref(), root) else { + return Ok(None); + }; + let view = self.backend.begin_read().expect("read view"); + let bytes = view + .get( + Table::ExecutionPayloadEnvelopes, + &encode_slot_root_key(slot, root), + ) + .expect("get"); + Ok(bytes.map(|bytes| { + gloas::SignedExecutionPayloadEnvelope::from_ssz_bytes(&bytes) + .expect("a stored envelope decodes") + })) + } + + /// Gloas: the stored envelopes of payloads on the canonical chain, for the + /// slots `start_slot..=end_slot`, in ascending slot order. + /// + /// Serves `ExecutionPayloadEnvelopesByRange`. A payload is part of the + /// chain when the next canonical block builds on it: that block's bid + /// `parent_block_hash` equals the payload's `block_hash`. A block whose + /// successor builds on its empty branch has its envelope left out, even + /// when this node holds one. The head has no successor, so its envelope is + /// served only when the head node fork choice picked is `Full` + /// ([`Self::head_payload_status`]); before this process has computed a head + /// the status is unknown and the head's envelope is withheld. + /// + /// Canonical blocks come from `BlockRoots` and the whole answer is read + /// from one view, like [`get_signed_blocks_by_slot_range`](Self::get_signed_blocks_by_slot_range). + /// The successor of the last block in the window may lie past `end_slot`, + /// so the index is read on beyond it, up to the head, until one is found; + /// that search is skipped when the last block holds no envelope to judge. + pub fn canonical_execution_payload_envelopes( + &self, + start_slot: u64, + end_slot: u64, + ) -> Result, Error> { + if self.chain != Chain::Beacon { + Self::lean_only("canonical_execution_payload_envelopes"); + } + let recorded_head = self.beacon.lock().unwrap().head_node_payload_status; + let view = self.backend.begin_read().expect("read view"); + let canonical_root = |slot: u64| -> Option { + view.get(Table::BlockRoots, &encode_block_root_key(slot)) + .expect("get block root") + .map(|bytes| H256::from_ssz_bytes(&bytes).expect("valid block root")) + }; + + let envelope_row = |slot: u64, root: &H256| { + view.get( + Table::ExecutionPayloadEnvelopes, + &encode_slot_root_key(slot, root), + ) + .expect("get") + }; + + let mut window: Vec<(u64, H256)> = (start_slot..=end_slot) + .filter_map(|slot| canonical_root(slot).map(|root| (slot, root))) + .collect(); + // Head and its slot come out of the same view as the rest, so a head + // move partway through cannot pair one branch's index with another's. + let head_root = view + .get(Table::Metadata, KEY_HEAD) + .expect("get head") + .map(|bytes| H256::from_ssz_bytes(&bytes).expect("valid head root")); + // The head's envelope is served only when the recorded node is this + // view's head and FULL; a status recorded for another root says + // nothing about this one. + let head_is_full = head_root.is_some() + && recorded_head == head_root.map(|root| (root, PayloadStatus::Full)); + let head_slot = head_root + .and_then(|head| view.get(Table::BlockHeaders, &head.to_ssz()).expect("get")) + .map(|bytes| decode_beacon_block_value(&bytes).slot()); + if let Some(&(last_slot, last_root)) = window.last() + && envelope_row(last_slot, &last_root).is_some() + && let Some(head_slot) = head_slot + { + let successor = (end_slot.saturating_add(1)..=head_slot) + .find_map(|slot| canonical_root(slot).map(|root| (slot, root))); + window.extend(successor); + } + + let mut envelopes = Vec::new(); + for (index, &(slot, root)) in window.iter().enumerate() { + if slot > end_slot { + break; + } + let Some(bytes) = envelope_row(slot, &root) else { + continue; + }; + let envelope = gloas::SignedExecutionPayloadEnvelope::from_ssz_bytes(&bytes) + .expect("a stored envelope decodes"); + let builds_on_it = match window.get(index + 1) { + // No successor: this is the head, whose envelope is part of + // the chain only on the FULL node. + None => head_is_full, + Some((_, next_root)) => { + let next = view + .get(Table::BlockHeaders, &next_root.to_ssz()) + .expect("get") + .map(|bytes| decode_beacon_block_value(&bytes)); + match next { + Some(next) => { + builds_on_payload(&next, &envelope.message.payload.block_hash) + } + None => false, + } + } + }; + if builds_on_it { + envelopes.push(envelope); + } + } + Ok(envelopes) + } + + /// Rebuilds the in-memory `verified_payloads` and `block_timeliness` from + /// their tables, for every block fork choice can still reach. + /// + /// Walks `LiveChain` from the finalized block's slot up: the finalized + /// block is included because its payload status decides which branch its + /// children build on. Older rows stay on disk unread, but the walk still + /// iterates every `LiveChain` key to find the window (the index is not + /// pruned on a beacon directory), so the scan grows with chain length; + /// only the point reads are limited to the window. + /// + /// Also seeds both payload-committee vote vectors, empty, for every gloas + /// block in the window. Nothing persists the votes, and the head walk and + /// `notify_ptc_messages` read a gloas block's vectors, so an absent entry + /// would fail them with a `SpecAssert`. Empty is what `on_block` gives a + /// block it just imported, and is true here: this process has seen no + /// votes for it. + fn restore_beacon_scratch(&self) { + let finalized = self.beacon_finalized_checkpoint(); + // Without the finalized block the epoch start would be wrong, since a + // finalized block can sit below its checkpoint's epoch start; the + // anchor is the lowest slot this directory holds. + let floor = + beacon_block_slot(self.backend.as_ref(), &finalized.root).unwrap_or(self.anchor_slot); + + let view = self.backend.begin_read().expect("read view"); + let mut timeliness = HashMap::new(); + let mut payloads = HashSet::new(); + let mut gloas_roots = Vec::new(); + for (key, _) in view + .prefix_iterator(Table::LiveChain, &[]) + .expect("iterator") + .filter_map(Result::ok) + { + let (slot, root) = decode_slot_root_key(&key); + if slot < floor { + continue; + } + if let Some(bytes) = view + .get(Table::BlockTimeliness, &root.to_ssz()) + .expect("get") + { + let bytes: [u8; NUM_BLOCK_TIMELINESS_DEADLINES] = bytes + .as_slice() + .try_into() + .expect("a stored timeliness has one byte per deadline"); + timeliness.insert(root, bytes.map(|byte| byte != 0)); + } + let fork = view + .get(Table::BlockHeaders, &root.to_ssz()) + .expect("get") + .and_then(|bytes| bytes.first().copied()) + .and_then(ForkName::from_selector); + let is_gloas = match fork { + Some(ForkName::Gloas) => true, + Some( + ForkName::Phase0 + | ForkName::Altair + | ForkName::Bellatrix + | ForkName::Capella + | ForkName::Deneb + | ForkName::Electra + | ForkName::Fulu, + ) => false, + // A beacon directory never holds a lean block row. + Some(ForkName::Lean) => unreachable!("lean block row in a beacon directory"), + // A LiveChain row with no block: nothing to seed. + None => false, + }; + if is_gloas { + gloas_roots.push(root); + } + if view + .get(Table::ExecutionPayloadEnvelopes, &key) + .expect("get") + .is_some() + { + payloads.insert(root); + } + } + drop(view); + + let mut scratch = self.beacon.lock().unwrap(); + scratch.block_timeliness.extend(timeliness); + scratch.verified_payloads.extend(payloads); + for root in gloas_roots { + scratch + .payload_timeliness_vote + .insert(root, vec![None; PTC_SIZE]); + scratch + .payload_data_availability_vote + .insert(root, vec![None; PTC_SIZE]); + } + } + /// `root`'s payload link, if one has been recorded. See /// [`BlockPayloadLink`] for what it holds and why a missing entry is not /// an error. @@ -3586,6 +3958,66 @@ impl Store { .cloned() } + /// The execution client's verdict on a gloas block's payload, as the + /// specification's `block_payload_statuses.get(root, + /// PAYLOAD_STATUS_NOT_VALIDATED)`: a root with no recorded verdict reads as + /// `Syncing`, which is `NOT_VALIDATED`. + /// + /// Gloas roots only: a pre-gloas root's verdict is its block's own, which + /// `fork_choice::block_payload_status` derives. Envelope import is what + /// records a gloas root's verdict; nothing else writes it. The map is + /// scratch, so after a restart the verdicts of gloas blocks imported before + /// it are absent and votes for their full payload are ignored until those + /// blocks age out of the attestation window. A known, bounded gap. + pub fn beacon_block_payload_status(&self, root: H256) -> PayloadStatusEnum { + self.beacon + .lock() + .unwrap() + .block_payload_statuses + .get(&root) + .map_or(PayloadStatusEnum::Syncing, |(_slot, status)| *status) + } + + /// Whether any recorded gloas payload verdict is still `NOT_VALIDATED`. + /// + /// The cheap half of resolving optimistic payloads, like + /// [`Self::has_beacon_optimistic_roots`]: with a healthy execution client + /// it is always `false`, so callers can skip a `block_index` scan. + pub fn has_unvalidated_block_payloads(&self) -> bool { + self.beacon + .lock() + .unwrap() + .block_payload_statuses + .values() + .any(|(_slot, status)| status.is_not_validated()) + } + + /// Records the execution client's verdict on `root`'s payload, against the + /// block's slot. A later verdict replaces an earlier one, since an + /// optimistic payload is resolved one way or the other. + pub fn insert_beacon_block_payload_status( + &mut self, + root: H256, + slot: u64, + status: PayloadStatusEnum, + ) { + self.beacon + .lock() + .unwrap() + .block_payload_statuses + .insert(root, (slot, status)); + } + + /// Drops payload verdicts strictly below `finalized_slot`, which no + /// attestation rule can ask about any more. + pub fn prune_beacon_block_payload_statuses(&mut self, finalized_slot: u64) { + self.beacon + .lock() + .unwrap() + .block_payload_statuses + .retain(|_root, (slot, _status)| *slot >= finalized_slot); + } + /// Records an execution client's answer for a payload. The fixture format /// allows the same payload's status to be updated several times over a /// case, so this overwrites rather than preserving a first answer. @@ -3946,6 +4378,66 @@ impl Store { Ok(()) } + /// Decode a stored sidecar row, as the fork its slot's epoch names. + /// + /// The rows carry no fork tag (and gaining one would need a `DB_VERSION` + /// bump), but a fork is a function of the slot and the config, both known + /// here, so the schedule tells a fulu row from a gloas one. + pub fn decode_data_column_sidecar( + &self, + slot: u64, + encoded: &[u8], + ) -> Result { + let fork = self.config.fork_at_epoch(slot / SLOTS_PER_EPOCH); + DataColumnSidecar::from_ssz(fork, encoded) + .map_err(|_| Error::UndecodableDataColumn { slot, fork }) + } + + /// [`Self::put_data_column_sidecar`], for a sidecar in hand. + pub fn put_data_column(&self, sidecar: &DataColumnSidecar) -> Result<(), Error> { + self.put_data_column_sidecar( + sidecar.slot(), + &sidecar.block_root(), + sidecar.index(), + sidecar.to_ssz(), + ) + } + + /// [`Self::put_pending_data_column_sidecar`], for a sidecar in hand. + pub fn put_pending_data_column(&self, sidecar: &DataColumnSidecar) -> Result<(), Error> { + self.put_pending_data_column_sidecar( + sidecar.slot(), + &sidecar.block_root(), + sidecar.index(), + sidecar.to_ssz(), + ) + } + + /// [`Self::take_pending_data_column_sidecar`], decoded. A row that does not + /// decode is still taken, so it cannot be replayed into the same failure. + pub fn take_pending_data_column( + &self, + slot: u64, + block_root: &H256, + column_index: u64, + ) -> Result, Error> { + self.take_pending_data_column_sidecar(slot, block_root, column_index)? + .map(|encoded| self.decode_data_column_sidecar(slot, &encoded)) + .transpose() + } + + /// [`Self::get_data_column_sidecar`], decoded. + pub fn get_data_column( + &self, + slot: u64, + block_root: &H256, + column_index: u64, + ) -> Result, Error> { + self.get_data_column_sidecar(slot, block_root, column_index)? + .map(|encoded| self.decode_data_column_sidecar(slot, &encoded)) + .transpose() + } + /// Drop every parked row this directory holds. /// /// Called once at startup. The only index into `PendingDataColumns` is the @@ -4111,6 +4603,22 @@ impl Store { end_slot: u64, columns: &[u64], ) -> Result>, Error> { + Ok(self + .data_column_rows_in_range(start_slot, end_slot, columns)? + .into_iter() + .map(|(_, encoded)| encoded) + .collect()) + } + + /// [`Self::data_column_sidecars_in_range`] with each row's slot beside its + /// bytes, which is what [`Self::decode_data_column_sidecar`] needs to pick + /// the fork a range spanning one boundary decodes each row as. + pub fn data_column_rows_in_range( + &self, + start_slot: u64, + end_slot: u64, + columns: &[u64], + ) -> Result)>, Error> { let view = self.backend.begin_read().expect("read view"); let mut found = Vec::new(); for slot in start_slot..end_slot { @@ -4132,7 +4640,7 @@ impl Store { .try_into() .expect("a column key ends in an eight-byte index"); if columns.contains(&u64::from_be_bytes(index_bytes)) { - found.push(value.to_vec()); + found.push((slot, value.to_vec())); } } } @@ -7357,6 +7865,305 @@ mod tests { assert_eq!(store.block_timeliness(&H256::from([2u8; 32])), None); } + /// An envelope for `root` with every field at its zero value. + fn test_envelope(root: H256) -> gloas::SignedExecutionPayloadEnvelope { + let payload = gloas::ExecutionPayload { + parent_hash: Default::default(), + fee_recipient: Default::default(), + state_root: Default::default(), + receipts_root: Default::default(), + logs_bloom: vec![0u8; ethlambda_types::beacon::preset::BYTES_PER_LOGS_BLOOM] + .try_into() + .expect("built at exactly BYTES_PER_LOGS_BLOOM"), + prev_randao: Default::default(), + block_number: 7, + gas_limit: 0, + gas_used: 0, + timestamp: 0, + extra_data: Default::default(), + base_fee_per_gas: Default::default(), + block_hash: Default::default(), + transactions: Default::default(), + withdrawals: Default::default(), + blob_gas_used: 0, + excess_blob_gas: 0, + block_access_list: Default::default(), + slot_number: 0, + }; + gloas::SignedExecutionPayloadEnvelope { + message: gloas::ExecutionPayloadEnvelope { + payload, + execution_requests: Default::default(), + builder_index: 3, + beacon_block_root: root, + parent_beacon_block_root: H256::ZERO, + }, + signature: Default::default(), + } + } + + /// A beacon directory whose finalized checkpoint is the epoch-1 boundary + /// checkpoint naming `finalized_root`, so the finalized slot is the first + /// slot of that epoch. + fn beacon_backend_finalized_at(finalized_root: H256) -> Arc { + let backend = Arc::new(InMemoryBackend::new()); + let finalized = BeaconCheckpoint { + epoch: 1, + root: finalized_root, + }; + Store::init_beacon( + backend.clone(), + 0, + Config::mainnet(), + finalized_root, + Store::beacon_checkpoint_as_stored(finalized), + 0, + ); + backend + } + + fn reopen(backend: Arc) -> Store { + Store::from_db_state(backend) + .expect("reopen") + .expect("populated directory") + } + + /// A gloas signed block with an empty body and a zero signature. + fn gloas_test_block(slot: u64, parent_root: H256) -> SignedBeaconBlock { + SignedBeaconBlock::Gloas(gloas::SignedBeaconBlock { + message: gloas::BeaconBlock { + slot, + proposer_index: 0, + parent_root, + state_root: H256::ZERO, + body: Default::default(), + }, + signature: Default::default(), + }) + } + + /// The head's envelope belongs to the served chain only on the FULL node: + /// an EMPTY head, or a head whose status this process has not computed, + /// withholds it. + #[test] + fn the_heads_envelope_is_served_only_when_the_head_node_is_full() { + let mut store = Store::init_beacon( + Arc::new(InMemoryBackend::new()), + 0, + Config::mainnet(), + H256::ZERO, + Store::beacon_checkpoint_as_stored(BeaconCheckpoint::default()), + 0, + ); + store + .insert_signed_block(H256::ZERO, gloas_test_block(0, H256::ZERO)) + .expect("insert anchor"); + let block = gloas_test_block(1, H256::ZERO); + let root = block.message_hash_tree_root(); + store.insert_signed_block(root, block).expect("insert head"); + store + .update_checkpoints(ForkCheckpoints::head_only(root)) + .expect("advance head"); + let envelope = test_envelope(root); + store.insert_verified_payload(1, &envelope); + + assert_eq!(store.head_payload_status(), None); + assert!( + store + .canonical_execution_payload_envelopes(1, 1) + .unwrap() + .is_empty() + ); + + store.set_head_payload_status(root, PayloadStatus::Empty); + assert_eq!(store.head_payload_status(), Some(PayloadStatus::Empty)); + assert!( + store + .canonical_execution_payload_envelopes(1, 1) + .unwrap() + .is_empty() + ); + + // A FULL status recorded for another root (the previous head, say) + // must not release this head's envelope. + store.set_head_payload_status(H256::repeat_byte(0xee), PayloadStatus::Full); + assert_eq!(store.head_payload_status(), None); + assert!( + store + .canonical_execution_payload_envelopes(1, 1) + .unwrap() + .is_empty() + ); + + store.set_head_payload_status(root, PayloadStatus::Full); + assert_eq!( + store.canonical_execution_payload_envelopes(1, 1).unwrap(), + vec![envelope] + ); + } + + #[test] + fn an_unverified_payload_is_forgotten_across_reopen() { + let root = H256::from([1u8; 32]); + let backend = beacon_backend_finalized_at(root); + let mut store = reopen(backend.clone()); + store + .insert_signed_block(root, gloas_test_block(SLOTS_PER_EPOCH, H256::ZERO)) + .expect("insert block"); + store.insert_verified_payload(SLOTS_PER_EPOCH, &test_envelope(root)); + store.insert_beacon_el_block_hash(root, SLOTS_PER_EPOCH, H256::from([9u8; 32])); + + assert!(store.remove_verified_payload(&root)); + assert_eq!(store.beacon_el_block_hash(root), None); + assert!(!store.remove_verified_payload(&root)); + + assert!(!store.has_verified_payload(&root)); + assert_eq!(store.get_execution_payload_envelope(&root).unwrap(), None); + assert!(!reopen(backend).has_verified_payload(&root)); + } + + #[test] + fn a_verified_payload_survives_reopen() { + let root = H256::from([1u8; 32]); + let backend = beacon_backend_finalized_at(root); + let mut store = reopen(backend.clone()); + store + .insert_signed_block(root, gloas_test_block(SLOTS_PER_EPOCH, H256::ZERO)) + .expect("insert block"); + let envelope = test_envelope(root); + store.insert_verified_payload(SLOTS_PER_EPOCH, &envelope); + assert!(store.has_verified_payload(&root)); + + let reopened = reopen(backend); + + assert!(reopened.has_verified_payload(&root)); + assert_eq!( + reopened + .get_execution_payload_envelope(&root) + .expect("read"), + Some(envelope) + ); + } + + #[test] + fn block_timeliness_survives_reopen() { + let root = H256::from([1u8; 32]); + let backend = beacon_backend_finalized_at(root); + let mut store = reopen(backend.clone()); + store + .insert_signed_block(root, gloas_test_block(SLOTS_PER_EPOCH, H256::ZERO)) + .expect("insert block"); + store.set_gloas_block_timeliness(root, [true, false]); + + let reopened = reopen(backend); + + assert_eq!(reopened.block_timeliness(&root), Some([true, false])); + } + + #[test] + fn pre_gloas_timeliness_is_not_persisted() { + let root = H256::from([1u8; 32]); + let backend = beacon_backend_finalized_at(root); + let mut store = reopen(backend.clone()); + store + .insert_signed_block(root, beacon_test_block(SLOTS_PER_EPOCH, H256::ZERO)) + .expect("insert block"); + store.set_block_timeliness(root, [true, true]); + + let reopened = reopen(backend); + + assert_eq!(reopened.block_timeliness(&root), None); + } + + #[test] + fn resume_reseeds_empty_vote_vectors_for_gloas_blocks_only() { + let gloas_root = H256::from([1u8; 32]); + let phase0_root = H256::from([2u8; 32]); + let backend = beacon_backend_finalized_at(gloas_root); + let mut store = reopen(backend.clone()); + store + .insert_signed_block(gloas_root, gloas_test_block(SLOTS_PER_EPOCH, H256::ZERO)) + .expect("insert block"); + store + .insert_signed_block( + phase0_root, + beacon_test_block(SLOTS_PER_EPOCH + 1, gloas_root), + ) + .expect("insert block"); + + let reopened = reopen(backend); + + let empty = vec![None; PTC_SIZE]; + assert_eq!( + reopened.payload_timeliness_vote(&gloas_root), + Some(empty.clone()) + ); + assert_eq!( + reopened.payload_data_availability_vote(&gloas_root), + Some(empty) + ); + assert_eq!(reopened.payload_timeliness_vote(&phase0_root), None); + } + + #[test] + fn resume_skips_payloads_below_finality() { + // The finalized block sits below its checkpoint's epoch start, and its + // child sits between the two: a floor at the epoch start would skip + // both, so the child loading shows the floor is the block's own slot. + let old = H256::from([1u8; 32]); + let finalized = H256::from([2u8; 32]); + let child = H256::from([3u8; 32]); + let backend = beacon_backend_finalized_at(finalized); + let mut store = reopen(backend.clone()); + let slots = [ + (old, SLOTS_PER_EPOCH - 3, H256::ZERO), + (finalized, SLOTS_PER_EPOCH - 2, old), + (child, SLOTS_PER_EPOCH - 1, finalized), + ]; + for (root, slot, parent) in slots { + store + .insert_signed_block(root, gloas_test_block(slot, parent)) + .expect("insert block"); + store.insert_verified_payload(slot, &test_envelope(root)); + store.set_gloas_block_timeliness(root, [true, true]); + } + + let reopened = reopen(backend); + + assert!(!reopened.has_verified_payload(&old)); + assert_eq!(reopened.block_timeliness(&old), None); + assert_eq!(reopened.payload_timeliness_vote(&old), None); + for root in [finalized, child] { + assert!(reopened.has_verified_payload(&root)); + assert_eq!(reopened.block_timeliness(&root), Some([true, true])); + } + // Skipped in memory only: the row stays on disk. + assert!( + reopened + .get_execution_payload_envelope(&old) + .unwrap() + .is_some() + ); + } + + #[test] + fn resume_without_the_finalized_block_floors_at_the_anchor_slot() { + // The finalized block has no row, so the floor falls back; a block + // below the checkpoint's epoch start must still load. + let finalized = H256::from([2u8; 32]); + let block = H256::from([3u8; 32]); + let backend = beacon_backend_finalized_at(finalized); + let mut store = reopen(backend.clone()); + store + .insert_signed_block(block, gloas_test_block(5, H256::ZERO)) + .expect("insert block"); + store.set_gloas_block_timeliness(block, [true, true]); + + let reopened = reopen(backend); + + assert_eq!(reopened.block_timeliness(&block), Some([true, true])); + } + #[test] fn latest_messages_skip_equivocators() { let mut store = Store::test_store(); @@ -7472,6 +8279,32 @@ mod tests { assert!(!store.is_beacon_optimistic(stale)); } + #[test] + fn block_payload_statuses_default_to_not_validated_and_prune_below_finality() { + let mut store = Store::test_store(); + let below = H256::repeat_byte(1); + let at = H256::repeat_byte(2); + assert!( + store.beacon_block_payload_status(below).is_not_validated(), + "an absent root reads as the specification's default" + ); + + store.insert_beacon_block_payload_status(below, 4, PayloadStatusEnum::Valid); + store.insert_beacon_block_payload_status(at, 5, PayloadStatusEnum::Invalid); + assert_eq!( + store.beacon_block_payload_status(below), + PayloadStatusEnum::Valid + ); + + store.prune_beacon_block_payload_statuses(5); + + assert!(store.beacon_block_payload_status(below).is_not_validated()); + assert_eq!( + store.beacon_block_payload_status(at), + PayloadStatusEnum::Invalid + ); + } + #[test] fn el_block_hashes_prune_strictly_below_the_finalized_slot() { let mut store = Store::test_store(); @@ -7917,6 +8750,101 @@ mod tests { vec![marker; 16] } + /// A beacon store whose config puts gloas at epoch 2, so slots below + /// `2 * SLOTS_PER_EPOCH` are fulu and the rest gloas. + fn gloas_scheduled_store() -> Store { + let config = Config { + fulu_fork_epoch: 0, + gloas_fork_epoch: 2, + ..Config::mainnet() + }; + Store::init_beacon( + Arc::new(InMemoryBackend::new()), + 0, + config, + H256::ZERO, + Checkpoint::default(), + 0, + ) + } + + fn fulu_sidecar_at(slot: u64, index: u64) -> DataColumnSidecar { + let mut sidecar = ethlambda_types::beacon::containers::fulu::DataColumnSidecar { + index, + column: Default::default(), + kzg_commitments: Default::default(), + kzg_proofs: Default::default(), + signed_block_header: Default::default(), + kzg_commitments_inclusion_proof: vec![ + H256::ZERO; + ethlambda_types::beacon::preset::KZG_COMMITMENTS_INCLUSION_PROOF_DEPTH + ] + .try_into() + .expect("exactly the required depth"), + }; + sidecar.signed_block_header.message.slot = slot; + DataColumnSidecar::Fulu(sidecar) + } + + fn gloas_sidecar_at(slot: u64, index: u64, root: H256) -> DataColumnSidecar { + DataColumnSidecar::Gloas( + ethlambda_types::beacon::containers::gloas::DataColumnSidecar { + index, + slot, + beacon_block_root: root, + ..Default::default() + }, + ) + } + + #[test] + fn a_gloas_sidecar_round_trips_through_both_column_tables() { + let store = gloas_scheduled_store(); + let root = H256::repeat_byte(7); + let slot = 2 * SLOTS_PER_EPOCH + 3; + let sidecar = gloas_sidecar_at(slot, 4, root); + + store.put_data_column(&sidecar).unwrap(); + assert!(store.has_data_column(slot, &root, 4)); + assert_eq!( + store.get_data_column(slot, &root, 4).unwrap(), + Some(sidecar.clone()) + ); + + let parked = gloas_sidecar_at(slot, 5, root); + store.put_pending_data_column(&parked).unwrap(); + // Parked rows are not custodied ones. + assert!(!store.has_data_column(slot, &root, 5)); + assert_eq!( + store.take_pending_data_column(slot, &root, 5).unwrap(), + Some(parked) + ); + assert_eq!( + store.take_pending_data_column(slot, &root, 5).unwrap(), + None + ); + } + + #[test] + fn a_row_decodes_as_the_fork_its_slot_names() { + let store = gloas_scheduled_store(); + let fulu = fulu_sidecar_at(SLOTS_PER_EPOCH, 1); + let root = fulu.block_root(); + store.put_data_column(&fulu).unwrap(); + assert_eq!( + store.get_data_column(SLOTS_PER_EPOCH, &root, 1).unwrap(), + Some(fulu) + ); + + // The same bytes filed under a gloas slot are not a gloas sidecar. + let gloas_slot = 2 * SLOTS_PER_EPOCH; + let encoded = fulu_sidecar_at(gloas_slot, 1).to_ssz(); + assert!(matches!( + store.decode_data_column_sidecar(gloas_slot, &encoded), + Err(Error::UndecodableDataColumn { .. }) + )); + } + #[test] fn a_sidecar_round_trips_by_slot_root_and_index() { let store = beacon_test_store(Arc::new(InMemoryBackend::new())); diff --git a/docs/beacon_engine.md b/docs/beacon_engine.md index d7c557baa..f26847faf 100644 --- a/docs/beacon_engine.md +++ b/docs/beacon_engine.md @@ -14,19 +14,22 @@ The wire is ## Which methods, and why so few -Four: +Six: | Method | Introduced | Why | |---|---|---| | `engine_newPayloadV4` | prague | Validate one block's payload | +| `engine_newPayloadV5` | amsterdam | Validate the payload a gloas envelope reveals | | `engine_forkchoiceUpdatedV3` | cancun | Say where the head, safe and finalized blocks are | +| `engine_forkchoiceUpdatedV4` | amsterdam | The same, from a gloas head on; also carries the node's custody columns | | `engine_exchangeCapabilities` | common | Startup handshake | | `engine_getClientVersionV1` | identification | Log what the execution client is | Osaka introduces **no new `newPayload`**. Its own document adds only `engine_getPayloadV5` and `engine_getBlobsV2`/`V3`; `engine_newPayloadV5` belongs -to Amsterdam. So the Osaka-current call for a payload is prague's V4, and the -Osaka-current fork choice notification is cancun's V3. +to Amsterdam, where gloas's envelope import uses it. So the Osaka-current call for a payload is prague's V4, and the +Osaka-current fork choice notification is cancun's V3. Amsterdam's pair (newPayload V5, forkchoiceUpdated V4) is +used from gloas on; the startup handshake warns when the execution client does not advertise either, but only on a network whose schedule includes gloas. Three method families a full client would have are deliberately absent: @@ -59,6 +62,10 @@ process_block (beacon arm) └─ record the block's execution hash and its optimistic status ``` +This diagram is the pre-gloas path. A gloas block carries no payload, only a +builder's bid, so nothing is asked about it at import; the question moves to the +envelope that reveals the payload (see [Gloas](#gloas) below). + The order is the specification's own: `is_data_available` runs before `state_transition`, so a block about to be held for its custody columns is never one this node asks an execution client about. @@ -119,11 +126,11 @@ finality: an execution client doing a long state sync answers `NOT_VALIDATED` to every block, so without that bound the set would take one root per import for the life of the process. -Nothing outside `mark_validated`'s own ancestor walk reads the set yet. The -readers it is waiting for are the ones that need to answer "is my head -optimistic?": the Beacon API's `execution_optimistic` response field, and a sync -status that separates a head this node has vouched for from one it has merely -imported. +Besides `mark_validated`'s own ancestor walk, the Beacon API's +`execution_optimistic` response field reads the set (see +[rpc.md](./rpc.md#execution_optimistic-under-gloas)). A sync status that +separates a head this node has vouched for from one it has merely imported does +not exist yet. ### `is_optimistic_candidate_block` @@ -194,6 +201,40 @@ itself. An execution client condemning finalized history means it and this node disagree about what is final, which is an operator emergency rather than something to resolve by emptying fork choice. +## Gloas + +A gloas block commits to a builder's bid and the payload arrives later in a +`SignedExecutionPayloadEnvelope`, so the execution client judges the envelope, +not the block. The order differs from the specification's, which asks the engine +last inside one boolean function: the follower runs the pure consensus checks +first (signature, bid, state, data availability), and only an envelope that +passed them is sent to `engine_newPayloadV5`. That way an unsigned copy with the +honest block hash and a different body cannot earn an `INVALID` that lands on the +honest payload, and the engine never spends time on an envelope this node would +refuse anyway. + +| Answer | What happens to the envelope | +|---|---| +| `VALID` | Applied; the payload is recorded `VALID` | +| `SYNCING`, `ACCEPTED` | Applied and recorded `SYNCING` (not yet validated); a later `forkchoiceUpdated` `VALID` promotes it | +| `INVALID` with a `latestValidHash` | The payload is invalid: not applied, so the block's FULL node never exists; the block and its EMPTY branch stay | +| `INVALID` with a null `latestValidHash`, or `INVALID_BLOCK_HASH` | The envelope's contents do not hash to the block hash it claims: only that envelope is refused, since the builder's real one may still arrive | +| no answer after the retry ladder | Kept (it is consensus-valid) and asked again once a slot | + +The null-`latestValidHash` row exists because ethrex has no `INVALID_BLOCK_HASH` +status: it answers a mismatch with a null hash and an executed-and-failed payload +with its last valid ancestor. `latestValidHash` and `VALID` follow the execution +chain (each block's `bid.parent_block_hash`), not every verified ancestor, since +a block built past an ancestor's EMPTY node skips that ancestor's payload. The +details, and why each departs from the specification, are in +[spec_deviations.md](./spec_deviations.md#the-execution-client-judges-a-gloas-payload-before-the-envelope-is-applied). + +With the engine down, the per-slot retry asks about the oldest held envelope +only and releases the rest once that ask is answered, so an unreachable client +costs one retry ladder per slot however many envelopes wait. This is the gloas +counterpart of the limitation below, and unlike a block the envelope is not +dropped. + ## `forkchoiceUpdated` Sent once after every head recompute, which is once per import cascade and once @@ -218,6 +259,15 @@ epoch boundary leaves the finalized block below the bound. Dropping its hash would make every later call carry `finalized_block_hash = 0x00..0` and stop the execution client advancing its own finalized block for the life of the process. +For a gloas head the hashes come from bids rather than from a cached payload +hash: the head node's `bid.block_hash` when it is FULL and its +`bid.parent_block_hash` when it is EMPTY, the finalized block's +`bid.parent_block_hash`, and the justified block's `bid.parent_block_hash` as +the safe hash. The last is a fallback, since the specification builds the safe +hash on fast confirmation, which the follower does not run. The call is V4 with +the node's custody columns. A head whose payload status has not been computed +yet (the first moments after a restart) sends nothing. + The response carries a `PayloadStatusV1` of its own, and that is the channel by which a block imported on `SYNCING` later becomes `VALID` or is found to be `INVALID`. An invalidation reaching the follower this way can remove the head diff --git a/docs/beacon_stf.md b/docs/beacon_stf.md index e8356ac84..313da7e70 100644 --- a/docs/beacon_stf.md +++ b/docs/beacon_stf.md @@ -345,11 +345,57 @@ wholesale, so its code sits beside the pre-gloas code rather than inside it: `st section of `fork_choice.rs` (fork choice), and `containers/gloas.rs` in `ethlambda-types`. Every gloas case that is not ignored passes on both presets. -The live follower does not follow gloas yet. `process_or_pend_block` refuses a -gloas block and `refuse_unfollowable_fork` refuses a gloas anchor at startup, -because nothing delivers payload envelopes or payload attestations to the chain -actor. Gloas gossip is answered with `Ignore(UnsupportedFork)` rather than -scored, so an honest peer past the fork epoch is not penalized. +The live follower follows gloas (`ForkName::Gloas.is_followed()`): the chain +actor imports gloas blocks and their payload envelopes, a gloas checkpoint +anchor is accepted (it starts EMPTY, with no payload known), and the clock +passing the fork epoch no longer forces the sync status to syncing. Fork choice +records the head node's payload status next to the head root +(`Store::head_payload_status`, in memory, recomputed by the first head +computation after a restart), which the envelope by-range server reads to +withhold the head's envelope while the head is the EMPTY node. The node is +still only a follower: the validator-client endpoints refuse a gloas epoch. + +### The envelope pipeline in the chain actor + +Order of arrival is not order of validity: an envelope can beat its block, a +block can name a parent payload that has not arrived, and the columns an +envelope's commitments need can lag both. `beacon_envelope.rs` holds each case +in a queue with its own bound, and `beacon_columns.rs` bounds the parked +sidecars of the same fork. + +| Queue | Holds | Released when | +|---|---|---| +| `awaiting_import` | an envelope whose block is stored but has no post-state (pending on a parent, held for columns or for its parent's payload), checked against the stored block's bid | the block imports; evicted with the block when it is discarded and at finality; bounded by the stored blocks, not by arrival slot | +| `awaiting_block` | an envelope whose block is not stored at all | the block imports; evicted after `ENVELOPE_AWAITING_BLOCK_TTL_SLOTS`, capped per slot and per root | +| `awaiting_columns` | a consensus-checked envelope whose bid has commitments and whose sampled columns are not all stored | the last column is stored | +| `awaiting_engine` | a consensus-valid envelope the execution client gave no answer for | the per-slot redrive gets an answer (see [beacon_engine.md](./beacon_engine.md#gloas)) | +| `blocks_awaiting_parent_payload` | a block whose parent is FULL with a payload not yet verified, capped per parent | that parent's envelope verifies | + +Data availability moves with the payload: a gloas block's commitments are in its +bid, so the columns gate the envelope that reveals the payload, not the block's +own import. A held block's parent envelope is requested through +`FetchRequest.needs_envelope` the moment the block is held, whatever its source: +`BlockSource::Sync` also marks by-root answers and pending children a cascade +released, so no range batch can be assumed to bring the envelope, and p2p dedups +in-flight roots. The tick re-asks once a slot for every parent still missing +one, which is only what recovers a fetch that gave up; catch-up does not wait +for it. + +The two envelope-before-block queues differ in who has vouched for the root. +A range batch's envelopes arrive when all but its first block are stored and +unimported, so a stored block's own bid is the authority: an envelope matching +it (`envelope_matches_bid`) goes to `awaiting_import` outside the per-slot cap, +and one contradicting it is dropped. Only a root with no stored block, which +nobody has authenticated, is held under the per-slot cap and the TTL. +Everything is swept at finality, except the finalized block's own envelope. + +Fork-choice events are timed at their arrival: the store clock is advanced to +the moment an envelope, block or payload vote reached the node before its +handler runs, because the tick fires once per slot and the timeliness deadlines +would otherwise be judged at the slot's start. Payload votes already verified +in p2p are applied without a second signature check +(`apply_verified_payload_attestation`). The PTC vote vectors are not persisted +and are reseeded empty on resume; see [data_storage.md](./data_storage.md). ### The deferred payload @@ -580,9 +626,8 @@ ignored runs. The ignored cases are deliberate exclusions: state transition and fork choice, and is not in this module's scope. - `networking/gossip_*` cases outside what the node validates: every fork's cases for topics it has no validator for (`IGNORED_HANDLERS`), non-fulu cases - of the four it validates, and the fulu vectors in `SKIPPED`, which assume a - bad-block cache. A gloas message is refused with `Ignore(UnsupportedFork)` - before any rule runs. + of the aggregate and attestation topics (block and data column validate fulu + and gloas), and the vectors in `SKIPPED`, which assume a bad-block cache. - The `heze` fixture tree, one ignored entry. See "Accounting for every fork directory" below. diff --git a/docs/beacon_wire.md b/docs/beacon_wire.md index d278f389e..83addec59 100644 --- a/docs/beacon_wire.md +++ b/docs/beacon_wire.md @@ -44,8 +44,9 @@ bootnodes are largely seed-only, so a crawl is how a peer is reached. ## The fork digest -Computed once at startup from the resolved network's genesis state, never -hardcoded as a digest: +Computed from the resolved network's genesis state and fork schedule, never +hardcoded as a digest. Startup derives the first one and a running node follows +the schedule from there (see below): ``` epoch = (now - genesis_time) / (SECONDS_PER_SLOT * SLOTS_PER_EPOCH) @@ -61,27 +62,50 @@ digest = base[..4] if epoch < FULU_FOR EIP-7892's, which is why mainnet's digest is `8c9f62fe` rather than fulu's bare `82fae541`. -Startup logs the next boundary's epoch and wall-clock time. The digest is -computed once, so crossing one strands the node on topic names nobody publishes -to; restart it to pick up the new digest. +Startup logs the next boundary's epoch, wall-clock time, fork and digest. A +running node crosses it without a restart (`beacon::transition` in +`ethlambda-p2p`, driven by `beacon::fork_schedule`). Both fork activations and +blob-parameter-only forks move the digest, so both are boundaries. Following the +spec's "Transitioning the gossip" rules, the next digest's topics are joined +`SUBSCRIBE_LEAD_EPOCHS` before its boundary and the previous digest's topics +are left `UNSUBSCRIBE_LAG_EPOCHS` after it. At the boundary the node +publishes and advertises the new digest, bumps its metadata sequence number +and moves its discv5 admission filter. While the window is open a gossip +message's fork comes from its own topic's digest, and a `Status` on either +digest is answered, but range sync starts only from a peer on the current +digest (or the next one, within clock skew of the boundary). + +One gap: the ENR this node serves over discv5 keeps the `eth2` entry it started +with, because ethrex's `DiscoveryServer` offers no way to replace its record at +runtime. Peers already connected are unaffected; peers discovering the node +after a boundary read the stale digest and reject the record. ## Gossip Seven global topics, `/eth2/{digest}/{name}/ssz_snappy`, plus two subnet families this node's own node id selects a narrow slice of: the data column subnets it custodies and the attestation subnets it backbones, both described -below. +below. A gloas digest adds two more, `execution_payload` and +`payload_attestation_message`, which `BeaconTopics::for_fork` subscribes under +gloas digests and not under earlier ones. An envelope is validated like a block +(its stateful half on the blocking pool) and goes to the chain actor on `Accept` +and on `Queue`, since the actor holds an envelope whose block is not imported +yet. A payload attestation is validated on the attestation permit pool and goes +to the actor on `Accept` only: a vote for a block not imported here is dropped, +since it is valid only within its own slot. | Topic | Decoded as | | --- | --- | | `beacon_block` | `SignedBeaconBlock`, fork chosen by the block's slot | -| `beacon_aggregate_and_proof` | `SignedAggregateAndProof`, phase0 or electra | +| `beacon_aggregate_and_proof` | `SignedAggregateAndProof`, phase0, electra or gloas | | `attester_slashing` | `AttesterSlashing`, phase0 or electra | | `voluntary_exit` | `SignedVoluntaryExit` | | `proposer_slashing` | `ProposerSlashing` | | `bls_to_execution_change` | `SignedBLSToExecutionChange` | | `sync_committee_contribution_and_proof` | `SignedContributionAndProof` | -| `beacon_attestation_{subnet_id}` | `Attestation`, phase0 or electra | +| `beacon_attestation_{subnet_id}` | `Attestation`, phase0 or electra's `SingleAttestation` (gloas keeps the latter) | +| `execution_payload` | `SignedExecutionPayloadEnvelope`, gloas digests only | +| `payload_attestation_message` | `PayloadAttestationMessage`, gloas digests only | `beacon_attestation_{0..63}` is no longer wholly unsubscribed. This node holds `SUBNETS_PER_NODE` (2 on mainnet) long-lived subscriptions from that family, @@ -118,14 +142,17 @@ plus `SUBNETS_PER_NODE`, still far short of a full subscription to every attestation, sync-committee and data-column subnet, and narrower still than `NUMBER_OF_CUSTODY_GROUPS` columns of custody, which is what a supernode would carry alone. A sidecar decodes as -`fulu::DataColumnSidecar`; the checks it passes before this node keeps or +the fork-agnostic `DataColumnSidecar` enum, its fork taken from the topic's +digest (the bytes carry no tag): fulu's names its block by a signed header and +inclusion proof, gloas's by a slot and block root, with the commitments it is +checked against living in that block's bid. The checks it passes before this node keeps or forwards it are described just below. How a kept sidecar is later served back out over req/resp is under [Data column sidecars](#data-column-sidecars). Every beacon message is held by gossipsub until it has a verdict (`validate_messages()` is on for this wire only). Blocks, data column sidecars, aggregates and subnet attestations are all validated by fulu's -gossip rules (`ethlambda_state_transition::beacon::gossip`, one module per +gossip rules, or gloas's modified ones from the fork on (`ethlambda_state_transition::beacon::gossip`, one module per topic family): the checks that need no state run inline in the p2p actor, the rest on a bounded `spawn_blocking` task whose verdict comes back to the actor (`crate::beacon::verdict`). Two permit pools bound how many of these run at @@ -149,14 +176,32 @@ reaches it at all, on any outcome: verifying and relaying it is the whole of what this node owes the topic (see above), so there is nothing further for the chain actor to do with one. -The rules above are fulu's. A message from a fork this build has no rules for, -gloas today, is IGNOREd with the reason `unsupported_fork` and is never scored -against the peer: the decoders answer `DecodeError::UnsupportedFork`, not `Ssz`, -for a gloas aggregate, attester slashing or subnet attestation, or for a block -at a gloas slot that fails to decode, since an honest -gloas peer sends exactly those once this node's clock reaches the fork. A gloas -subnet attestation has the bytes of an electra one, but its `data.index` is the -payload flag, so decoding it under electra's rules would reject honest votes. +The rules above are fulu's, except that blocks, data columns, aggregates and +subnet attestations also have gloas rules. A gloas aggregate has electra's bytes +but its own progressive `Attestation`, so it decodes to its own +`SignedAggregateAndProof::Gloas` (the signed root differs); a gloas subnet +attestation is electra's `SingleAttestation` unchanged. In both, `data.index` is +the payload flag (0 or 1), and a vote for the full payload (1) is IGNOREd until +the block's envelope has been seen and its payload validated +(`verify_attestation_payload_status`). Gloas is a followed fork, so a block at +a gloas slot that fails to decode is REJECTed like one at any other followed +fork; `unsupported_fork` is kept for a fork the node does not follow +(`ForkName::is_followed`), none today. A gloas aggregate's aggregation bits are +an unbounded progressive bitlist, so one longer than electra's type bound is +REJECTed first, reading its length without expanding it, since expanding is +what the seen-set check does before any other. + +Three deliberate departures from the gloas gossip rules, each for want of state +this node does not keep. Where the specification rejects a message whose block +failed validation, this node (having no bad-block cache) cannot tell that from +a block it has not finished importing, so it queues or ignores instead; the +matching `reject_block_failed_validation` vectors are in `SKIPPED`. An +`execution_payload` whose block state is not in the cache p2p reads (a parent +imported moments ago, say) is queued and forwarded to the chain actor, which +verifies every envelope itself, without being propagated. And a +`payload_attestation_message` is applied to fork choice after p2p verified its +signature, with the actor re-checking only the store-state conditions (known +block, current slot, committee seat). The remaining five global topics are decoded, logged at `debug`, and IGNOREd, since nothing consumes them; an undecodable payload on any topic is REJECTed. @@ -257,6 +302,8 @@ choice. | `beacon_blocks_by_root/2` | both | | `data_column_sidecars_by_root/1` | both | | `data_column_sidecars_by_range/1` | both | +| `execution_payload_envelopes_by_range/1` | both | +| `execution_payload_envelopes_by_root/1` | both | The two data column sidecar protocols are registered because this node custodies the columns its node id selects and can answer for them out of @@ -364,6 +411,41 @@ fails. A mismatch ends the stream, logged at `warn` with both digests: the one w reach it in good faith is a blob schedule of ours that has fallen behind the network's. +### Execution payload envelopes + +Both gloas envelope protocols are registered `Full`: the server answers, and +the client (`req_resp/envelope_client.rs`) asks. A chunk carries one +`SignedExecutionPayloadEnvelope` under the fork digest of its block's epoch. +The envelope has no slot of its own, so the epoch comes from +`payload.slot_number`, which `verify_execution_payload_envelope` pins to the +block's slot. A chunk whose digest names no gloas-or-later scheduled fork, or +not the digest its slot implies, ends the stream. + +| Protocol | Served from | +| --- | --- | +| `execution_payload_envelopes_by_root/1` | a point lookup per root in `Table::ExecutionPayloadEnvelopes`; unknown roots are left out, answers follow the order asked | +| `execution_payload_envelopes_by_range/1` | the canonical blocks of the window (`BlockRoots`), each envelope kept only when the next canonical block builds on its payload; the head's own envelope only when the head node is FULL (`Store::head_payload_status`, which pairs the status with the head root it describes, so a stale status reads as unknown and the envelope is withheld) | + +Both are bounded by `MAX_REQUEST_PAYLOADS`: a by-root list over it fails to +decode, and a by-range `count` over it is truncated, which the specification's +"Clients MAY limit the number of payload envelopes in the response" allows. +A by-range request starting below `Store::anchor_slot` gets +`RESOURCE_UNAVAILABLE`, and a window before gloas is simply empty. + +The client asks in two situations. By root: the chain actor's +`FetchRequest.needs_envelope` (a block whose child waits on its payload) starts +a lookup with the block fetch's dedup, peer choice and retry ladder; a give-up +is recovered by the actor's per-slot re-ask. By range: after a +`beacon_blocks_by_range/2` answer for a span that reaches gloas has been handed +to the actor, the same peer is asked for that span's envelopes, and range sync +holds the next block request until that answer is checked and forwarded (or has +failed). The pause exists because the actor keeps an envelope whose block has +not imported only briefly and in small numbers, so a batch's envelopes must +arrive before the next batch's blocks do. Everything fetched goes through the +same rules as an `execution_payload` gossip message +(`beacon/envelope_checks.rs`) before the actor sees it; a `Queue` verdict is +forwarded too. + ### Data column sidecars Both protocols are registered `Full`, since with the columns on disk this node diff --git a/docs/checkpoint_sync.md b/docs/checkpoint_sync.md index 5c9e34b7b..53569364e 100644 --- a/docs/checkpoint_sync.md +++ b/docs/checkpoint_sync.md @@ -77,11 +77,37 @@ This is the recommended option for production deployments since it reduces trust 3. **Initialize**: the node stores the anchor block's header, its body (present unless the fetched block's own body happens to be empty, same as any other block), and the full state from the checkpoint. On `node`, persisting the block itself also means it can be served over `BlocksByRoot`; without that, peers requesting the anchor by root would get a synthetic block whose hash differs from `latest_finalized.root` and would score-penalize this node. `beacon`'s req/resp protocol has no `BlocksByRoot` handler yet, so that benefit doesn't apply there today; the block is stored anyway, since the pairing check above needs it. +#### Gloas anchor: the payload envelope (beacon only) + +A gloas block's execution payload is not in the block, so a node anchored on +one holds the block and state but not the payload its children may build on. +Once the chain actor is running, a node that anchored on a gloas block with no +stored envelope asks the checkpoint URLs for it, in order: +`GET /eth/v1/beacon/execution_payload_envelopes/{anchor_root}` as SSZ. A +success is handed to the chain actor as `new_execution_payload_envelope`, the +same entry a gossiped envelope takes, so the bid check, the column check and +the verification all apply. The fetch runs in the background and never delays +startup. + +It is a fast path and nothing depends on it. A `404` means the peer does not +know the payload, **not** that the payload is empty, so the node concludes +nothing from it; any failure (network, decoding, an envelope naming another +block) is logged at `info` and ignored. Without the envelope, the by-root +request that fires when a FULL child arrives recovers it. + +An anchor whose bid carries blobs also needs its sampled columns before its +envelope verifies, and those are fetched by root like any other block's. They +sit at the finalized slot, or below it for an anchor state advanced past empty +slots, which is where the chain checks drop every other block's columns. So +the finalized block's own columns are exempt from that rule +(`column::chain_checks_gloas`); without the exemption such an anchor never gets +its payload, and the node never imports past it. + ### Failure and success If any step fails (network error, decoding error, verification failure), the node logs the error and exits. There is no automatic retry; restart the node to try again. The database is not modified until verification succeeds, so a failed checkpoint sync leaves the data directory clean. -After successful initialization, the node starts normally: `node` connects to the P2P network and begins participating from the checkpoint slot; `beacon` joins the resolved network's gossip and logs what it decodes, without advancing its state past the anchor (see `docs/cli.md`, "What `ethlambda beacon` does today"). +After successful initialization, the node starts normally: `node` connects to the P2P network and begins participating from the checkpoint slot; `beacon` joins the resolved network's gossip and follows the chain from the anchor, importing blocks (a gloas anchor starts EMPTY, see [the envelope](#gloas-anchor-the-payload-envelope-beacon-only)). ## Restarts and Existing State @@ -145,7 +171,7 @@ All checks are performed before a downloaded checkpoint anchor is accepted. The | Justified epoch >= finalized epoch | Justified must be at or after finalized | | Block header slot <= state slot | Block header cannot be ahead of the state | | Block's fork matches state's fork | The fetched block must decode as the same fork the state resolved to | -| Anchor is not gloas (`refuse_unfollowable_fork`) | Nothing delivers payload envelopes or payload attestations to the chain actor yet, so a gloas anchor could not be followed. Runs on both anchor sources, the checkpoint provider's state and a loaded network's own genesis state | +| Anchor's fork is followed (`refuse_unfollowable_fork`) | A fork the node cannot follow is refused with `UnsupportedFork` before any store is built. Every fork is followed today, gloas included, so this refuses nothing; a gloas anchor starts with no payload known, so its head is EMPTY and its first FULL child is held until the parent's envelope is fetched. Runs on both anchor sources, the checkpoint provider's state and a loaded network's own genesis state | | Anchor pairing | See [Anchor Pairing](#anchor-pairing) below | Beacon has no same-slot-checkpoints-matching-roots check: nothing here computes two separate checkpoint roots to compare, since a trusted checkpoint-synced anchor is finalized by fiat, and the beacon spec's own construction applies that single checkpoint to all four of the store's justified/finalized slots at once. diff --git a/docs/cli.md b/docs/cli.md index 4661706ad..a013c07a8 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -166,11 +166,11 @@ numbers are accepted quoted or bare, and unrecognised keys (on a current config, the heze schedule this build does not claim) are dropped with one warning line naming each. Its `PRESET_BASE` is checked against the compiled preset; a mismatch is a hard startup error naming the cargo feature -that would fix it. Sepolia's config schedules gloas, whose keys this build -does claim but whose blocks the chain actor refuses, since nothing delivers -payload envelopes or payload attestations to it yet: a separate, explicit -warning at startup names that boundary, since the chain follower stops -tracking the chain there regardless of what parsed cleanly. The keys this +that would fix it. Sepolia's config schedules gloas, which this build follows, so +it parses and runs like any other fork. A scheduled fork the build claims but +does not follow gets a separate, explicit warning at startup, since the chain +follower would stop tracking the chain there regardless of what parsed +cleanly; none exists today. The keys this build runs on compile-time constants for (the custody and subnet counts, the `MAX_REQUEST_*` limits, `MAX_PAYLOAD_SIZE`, the snappy message domains and `MAXIMUM_GOSSIP_CLOCK_DISPARITY`) must equal those @@ -192,30 +192,25 @@ consistency (see [`checkpoint_sync.md`](checkpoint_sync.md)). The full anchor precedence is: a resumable data directory, then `--checkpoint-sync-url`, then, for a loaded network only, that directory's own `genesis.ssz`, then abort. The URL is therefore **required** on a fresh data directory only for a built-in -network: this follower imports nothing past its anchor, so anchoring a built-in -network at genesis would leave it parked at slot 0 while claiming to follow a -live chain. A loaded network's own genesis +network: no built-in network carries a genesis state, and anchoring one at genesis +would leave the follower replaying a chain that has been live for years. A loaded network's own genesis state is a legitimate anchor instead, since a freshly started devnet has no checkpoint provider at slot 0 and this is the only way to join one. A directory already anchored from a previous run resumes without the flag either way, the same way `node`'s does. -`--api-port` is bound here too, off `beacon`'s own anchored store now rather -than an empty one: one HTTP call site (`start_rpc_server`) serves both chains. -The lean-shaped `/lean/v0/...` routes read metadata keys and state variants a -beacon directory never carries, though, so calling one of them, e.g. `GET -/lean/v0/states/finalized`, panics that request rather than answering for a -chain that isn't running. This is deliberate for now: giving the beacon -follower its own HTTP surface is a change of its own. Treat `/metrics` on -`--metrics-port` as the only meaningful HTTP surface of a `beacon` run today. +`--api-port` is bound here too, off `beacon`'s own anchored store: one HTTP call +site serves both chains, but it picks the router by the store's chain, so a +`beacon` run serves the Beacon API under `/eth/v1` and `/eth/v2` and answers a +`/lean/v0/...` path with a 404. See [`rpc.md`](./rpc.md). ## What `ethlambda beacon` does today -It follows the resolved network's gossip and nothing above it. It now anchors a real, -RocksDB-backed store at a checkpoint-synced (or resumed) finalized state, but -does nothing more with it: no state transition, no fork choice, and no block -import past that anchor. The node joins the network, decodes what arrives, and -logs it. See [`beacon_wire.md`](./beacon_wire.md) for what goes on the wire and +It follows the resolved network as a non-validating follower. It anchors a +RocksDB-backed store at a checkpoint-synced (or resumed) finalized state, joins +the network's gossip and req/resp, and the chain actor imports blocks from there +(phase0 through gloas), runs fork choice, and, with `--execution-endpoint`, +asks an execution client about payloads. It never proposes or attests. See [`beacon_wire.md`](./beacon_wire.md) for what goes on the wire and how to check a run against the live network. ### One entry point, two chains @@ -239,16 +234,16 @@ not chain-specific, branches once, and shares the shutdown: | **the one `match`**: produce a `ChainSetup` | genesis config, validator keys, checkpoint sync or resume onto the shared backend, subnets | `beacon::wire_params` (genesis metadata, epoch, fork digest), then checkpoint sync or resume onto the same backend | | build the swarm, spawn P2P, start discv5 | shared; discv5 only with `--discovery.enable` | shared | | start the HTTP server | shared | shared | -| spawn the chain actor and wire it to P2P | yes | no: it imports nothing | +| spawn the chain actor and wire it to P2P | yes | yes: `BlockChain::spawn_beacon` | | ctrl-c, stop and join the actors | shared | shared | `ChainSetup` is what the `match` produces: the wire configuration, the ENR entries that describe it, the store the req/resp handlers answer from, the -node-name roster, and an `Option` holding the validator keys and -`BlockChainConfig`. The operator-supplied half of the discv5 configuration (node +node-name roster, and a `ChainActor` saying what to spawn: lean's validator keys +and `BlockChainConfig`, or beacon's marker. The operator-supplied half of the discv5 configuration (node key, ports, bootnodes, peer target) is the same on either chain, so it is filled -in once below the match. That `Option` being `None` is what ends the mainnet path: -`run_node` returns straight into the shared shutdown after starting the wire. +in once below the match; everything after it, including the chain actor spawn +and the shutdown, is shared. `beacon::wire_params` reads `genesis_time` and `genesis_validators_root` off the resolved network (a built-in network's two constants, or a loaded @@ -259,11 +254,7 @@ digest. It builds nothing: one `build_swarm` serves both chains, dispatching on the `WireConfig` variant for the topics, the req/resp protocol set, the gossipsub `seen_ttl`, the identify version and the connection limits. -The chain actor is the one thing mainnet has none of, so `RunningNode` -carries it as an `Option` and the shared shutdown skips it. - -Not implemented here: block import and fork choice past the anchor. A decoded -block is logged and dropped. +Both chains run a chain actor, so `RunningNode` carries a plain `BlockChain`. ## `validator` flags diff --git a/docs/data_storage.md b/docs/data_storage.md index 2f4f81245..421ae02b4 100644 --- a/docs/data_storage.md +++ b/docs/data_storage.md @@ -2,7 +2,7 @@ This doc explains how ethlambda saves data. Especially, the split between the fork choice `Store` and the `StorageBackend` trait, -what each of the eleven tables holds, and which data is in-memory only. +what each of the thirteen tables holds, and which data is in-memory only. ## Overview @@ -108,8 +108,10 @@ is built from it, and clones are handed to the BlockChain and P2P actors. │ │ LiveChain │ │ (raw XMSS sigs │ │ │ │ DataColumns │ │ awaiting │ │ │ │ PendingDataColumns │ │ aggregation) │ │ - │ │ BeaconUnrealizedJustifications │ │ state_cache (LRU) │ │ - │ └────────────────────────────────┘ └──────────────────────┘ │ + │ │ ExecutionPayloadEnvelopes │ │ state_cache (LRU) │ │ + │ │ BlockTimeliness │ └──────────────────────┘ │ + │ │ BeaconUnrealizedJustifications │ │ + │ └────────────────────────────────┘ │ │ Survives restarts, except │ │ PendingDataColumns, which Lost on restart. │ │ is cleared at startup. │ @@ -118,7 +120,7 @@ is built from it, and clones are handed to the BlockChain and P2P actors. ## The Tables -The eleven variants of the `Table` enum (`crates/storage/src/api/tables.rs`): +The thirteen variants of the `Table` enum (`crates/storage/src/api/tables.rs`): | Table | Key | Value | Pruned? | | ------------------ | --------------------------- | ----------------------------------------- | --------------------------------- | @@ -132,6 +134,8 @@ The eleven variants of the `Table` enum (`crates/storage/src/api/tables.rs`): | `LiveChain` | slot ‖ root | `parent_root` | yes: below finalized | | `DataColumns` | slot ‖ root ‖ column_index | `DataColumnSidecar` (SSZ-encoded) | no (see [DataColumns](#datacolumns) below) | | `PendingDataColumns` | slot ‖ root ‖ column_index | `DataColumnSidecar` (SSZ-encoded), unverified | yes: on replay, below finalized, and wholly at startup | +| `ExecutionPayloadEnvelopes` | slot ‖ root | gloas `SignedExecutionPayloadEnvelope` (SSZ), verified only | no: beacon block rows are kept too | +| `BlockTimeliness` | root | one byte per timeliness deadline (0 or 1), gloas blocks only | no | | `BeaconUnrealizedJustifications` | root | `(slot, Checkpoint)` (beacon only) | yes: below the finalized block's own slot (see [below](#beaconunrealizedjustifications)) | ### Key encoding @@ -479,6 +483,32 @@ counts. `lean_table_bytes{table="data_columns"}` (see [metrics.md](./metrics.md) is that growth made visible; watch it, and size the disk against however long the node runs unattended before a pruner exists. +### ExecutionPayloadEnvelopes and BlockTimeliness + +Gloas only. A row in `ExecutionPayloadEnvelopes` exists for an envelope that +passed verification, keyed by the slot of the block it fulfills (the envelope +has no slot of its own) so a by-range handler can scan a slot window. The +table is the durable form of the store's `verified_payloads` set. Losing that +set on restart would silently drop every full payload branch from fork choice. +`BlockTimeliness` is the durable form of the gloas part of `block_timeliness`. +Pre-gloas blocks write no row, since nothing reads their timeliness after a +restart. + +`Store::from_db_state` reloads both for every `LiveChain` block at or above the +finalized block's slot, and reseeds the in-memory payload-committee vote +vectors empty for each gloas block in that window. The votes themselves are not +persisted: the next block's payload attestations refill them. Neither table is +pruned; rows outside the window are simply not read. + +The execution client's verdict on each reloaded payload is in-memory scratch +too, so the chain actor seeds one at startup: `SYNCING` when an execution +client is configured, so its first `forkchoiceUpdated` `VALID` for the head +promotes them, and `VALID` without one, as the follower trusts every envelope it +applies then. The head node's payload status (`Store::head_payload_status`) is +likewise not stored; it is unknown until the first head computation after a +restart, and the envelope by-range server withholds the head's envelope until it +is known. + ### PendingDataColumns Same key and same encoding as `DataColumns`, holding sidecars that have **not @@ -750,10 +780,10 @@ where a future pruner is meant to land ## In-Memory Only (Lost on Restart) -Five `Store` fields never touch the backend directly, `beacon` with one -exception (see its row below). All are bounded buffers (or, for `beacon`, -bounded in practice by the validator set and the unfinalized window) shared -across `Store` clones: +Five `Store` fields never touch the backend directly, `beacon` with the +exceptions its row below names. All are bounded buffers (or, for +`beacon`, bounded in practice by the validator set and the unfinalized +window) shared across `Store` clones: | Buffer | Capacity | Contents | | ------------------- | --------------- | ---------------------------------------------------------------------------------------- | @@ -761,7 +791,7 @@ across `Store` clones: | `known_payloads` | 512 messages | Fork-choice-active aggregated proofs | | `gossip_signatures` | 2048 signatures | Raw per-validator XMSS signatures awaiting aggregation (each ~3 KB, so ~6 MB worst case) | | `state_cache` | 32 states | LRU memoization of block *and* checkpoint post-states (either chain), each held behind an `Arc` so a hit is not a copy; one bound covers both kinds, keyed apart by a small enum, and a miss is just a reconstruction rather than an error | -| `beacon` | unbounded | Beacon fork-choice scratch: proposer boost root, block timeliness, equivocating validator indices, latest messages, PoW blocks, and unrealized justifications. Most of it is not persisted: proposer boost resets every slot, timeliness is read only by the same-slot reorg helpers, equivocators come back from replaying attester slashings on sync, latest messages from one epoch of attestations, and PoW blocks stand in for an execution-client call a restarted node would simply make again. Unrealized justifications are the exception: this map is a write-through cache in front of `BeaconUnrealizedJustifications` (see [that table](#beaconunrealizedjustifications) above), so it is not lost, only refilled lazily on the next miss | +| `beacon` | unbounded | Beacon fork-choice scratch: proposer boost root, block timeliness, equivocating validator indices, latest messages, PoW blocks, and unrealized justifications. Most of it is not persisted: proposer boost resets every slot, pre-gloas timeliness is read only by the same-slot reorg helpers, equivocators come back from replaying attester slashings on sync, latest messages from one epoch of attestations, and PoW blocks stand in for an execution-client call a restarted node would simply make again. The exceptions are a gloas block's timeliness and verified payload (see [ExecutionPayloadEnvelopes and BlockTimeliness](#executionpayloadenvelopes-and-blocktimeliness)) and unrealized justifications: this map is a write-through cache in front of `BeaconUnrealizedJustifications` (see [that table](#beaconunrealizedjustifications) above), so it is not lost, only refilled lazily on the next miss | The payload buffers evict FIFO when full, and redundant proofs (whose participants are a subset of an existing proof for the same attestation data) @@ -775,7 +805,7 @@ pools. After a restart these buffers start empty: pending attestations and un-aggregated gossip signatures are lost and must be re-collected from the -network. Everything persisted in the eleven tables survives, except `PendingDataColumns`, which is cleared outright: its only index is in memory. +network. Everything persisted in the thirteen tables survives, except `PendingDataColumns`, which is cleared outright: its only index is in memory. ## Startup and Restore diff --git a/docs/discovery.md b/docs/discovery.md index f8285524f..3040ea64a 100644 --- a/docs/discovery.md +++ b/docs/discovery.md @@ -93,8 +93,11 @@ A discovered peer is admitted only if: - it advertises a `quic` port, a `tcp` port, or both. The digest compared against is the caller's, not a constant: lean passes its -hardcoded dummy and `beacon` passes the digest it derived at startup, so one -admission policy serves both chains. +hardcoded dummy and `beacon` passes the digest it derived at startup and moves +it at each fork-digest boundary, so one admission policy serves both chains. The +record this node serves is not refreshed at a boundary (ethrex's +`DiscoveryServer` cannot replace it at runtime), so its `eth2` entry keeps the +startup digest until a restart; see [beacon_wire.md](./beacon_wire.md#the-fork-digest). A differing `next_fork_version` or `next_fork_epoch` is *not* grounds for rejection: the spec permits connecting to a peer that is incompatible with an diff --git a/docs/metrics.md b/docs/metrics.md index 83d0627f2..64644d7c1 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -363,32 +363,38 @@ own section. These are ethlambda-specific, not part of the leanMetrics spec. `kind` is the topic kind, with every `data_column_sidecar_{subnet}` sharing the label `data_column_sidecar` and every `beacon_attestation_{subnet_id}` sharing -`beacon_attestation`. `queue` means IGNORE to gossipsub while the chain actor -still receives the object and parks it; neither the aggregate nor the -attestation topic ever answers `queue`, since the vote block's post-state is -either cached or it is not (`IgnoreReason::UnknownBlock`/`StateUnavailable`), -with nothing to hold the message for. **`verdict_expired_total` should stay at +`beacon_attestation`. The gloas topics `execution_payload` and +`payload_attestation_message` are labelled by their own name. `queue` means +IGNORE to gossipsub while the chain actor still receives the object and parks +it; of the gloas topics only `execution_payload` answers it, for an envelope +whose block is not known yet (`block_unknown`), has no post-state yet +(`block_not_ready`), or has one that is not in the cache the checks read +(`state_not_cached`, as when the block was imported moments ago; the actor +verifies the signature itself, so the envelope is forwarded rather than dropped). The aggregate, attestation and +`payload_attestation_message` topics never answer `queue`, since the vote +block's post-state is either cached or it is not +(`IgnoreReason::UnknownBlock`/`StateUnavailable`), with nothing to hold the +message for. **`verdict_expired_total` should stay at zero**: a rising count means validation is too slow for gossipsub's message cache. The two topics add reasons the others do not, all `reason` label values on `lean_beacon_gossip_validation_total`: `outside_epoch_window`, `covered_bits` (aggregate only), `unknown_block`, `state_unavailable`, -`finalized_not_ancestor`, `ancestry_unknown` on the ignore side; +`finalized_not_ancestor`, `ancestry_unknown`, `payload_envelope_unseen` and +`payload_optimistic` (gloas only) on the ignore side; `epoch_mismatch`, `no_participants` (aggregate only), `non_zero_data_index`, `committee_bits` (aggregate only), `committee_index`, `bits_length` (aggregate only), `not_aggregator` (aggregate only), `not_in_committee`, `unknown_validator`, `selection_proof` (aggregate only), `aggregator_signature` (aggregate only), `aggregate_signature` (aggregate -only), `target_not_ancestor`, `wrong_subnet` (attestation only) on the reject -side. `already_seen`, `overloaded` and `unsupported_fork` are shared with the +only), `target_not_ancestor`, `wrong_subnet` (attestation only), and gloas's +`data_index_out_of_range`, `same_slot_payload_flag` and `payload_invalid` on the +reject side. `already_seen`, `overloaded` and `unsupported_fork` are shared with the other topics: `unsupported_fork` is an ignore reason for a message of a fork this -build has no gossip rules for (gloas today), on the block, data column, -aggregate, attestation and attester-slashing topics, so an honest peer past the -fork epoch is not scored as a bad decoder. The same label value appears as -`result` on `lean_beacon_gossip_messages_total` for every one of those but the -block topic: a gloas block decodes, so it counts as `decoded` there and -`unsupported_fork` shows only as a verdict reason. +build has no gossip rules for, so an honest peer past the fork epoch is not +scored as a bad decoder. Gloas blocks, data columns, aggregates and attestations +are validated, so they carry their own verdict reasons instead. Two permit pools bound the blocking-thread half of validation: `gossip_validation_permits` for blocks and columns, @@ -467,6 +473,9 @@ spec. | `lean_data_column_fetch_failures_total` | Counter | `DataColumnsByRoot` lookups this node gave up on, by reason | On lookup abandonment | reason=no_peers,max_retries | | | `lean_blocks_held_for_columns` | Gauge | Blocks held out of fork choice pending their custody columns | On every hold, release, and finality eviction of the held-block set | | | | `lean_sidecars_awaiting_parent` | Gauge | Sidecars parked until their block's parent has a post-state | On every park, replay, and finality eviction of the parked set | | | +| `lean_envelopes_awaiting_block` | Gauge | Gloas execution payload envelopes held until their block is imported | On every hold, release, and finality eviction of the held envelopes | | | +| `lean_envelopes_awaiting_columns` | Gauge | Gloas execution payload envelopes held until every sampled column is stored | On every hold, release, and finality eviction of the held envelopes | | | +| `lean_blocks_awaiting_parent_payload` | Gauge | Gloas blocks held until their FULL parent's payload envelope is verified | On every hold, release, and finality eviction of the held blocks | | | `lean_data_columns_rejected_total` counts the chain checks, which run in the p2p layer on every sidecar gossip did not accept: every fetched sidecar, a diff --git a/docs/rpc.md b/docs/rpc.md index 34d779ab6..4e1c109f8 100644 --- a/docs/rpc.md +++ b/docs/rpc.md @@ -228,6 +228,7 @@ surface rather than sitting beside it; a `/lean/v0` path on a beacon node is a |--------|------|----------|-------------| | `GET` | `/eth/v2/beacon/blocks/{block_id}` | JSON or SSZ | `SignedBeaconBlock` at `block_id` | | `GET` | `/eth/v1/beacon/blocks/{block_id}/root` | JSON | That block's root | +| `GET` | `/eth/v1/beacon/execution_payload_envelopes/{block_id}` | JSON or SSZ | Gloas `SignedExecutionPayloadEnvelope` of that block (see below) | | `GET` | `/eth/v1/beacon/headers/{block_id}` | JSON | `SignedBeaconBlockHeader`, plus `canonical` | | `GET` | `/eth/v2/debug/beacon/states/{state_id}` | JSON or SSZ | `BeaconState` at `state_id` | | `GET` | `/eth/v1/beacon/states/{state_id}/finality_checkpoints` | JSON | That state's three checkpoints | @@ -272,7 +273,9 @@ the chain actor writes, so no request waits on the actor. `503` while syncing. Attester duties walk every committee of the epoch, a full shuffle per request on mainnet. An epoch the node's schedule places at gloas is a `400` on both duty endpoints: - the head is a fulu state, and this node does not follow gloas. + the head is a fulu state, and the node serves no validator duties from gloas + (it follows the fork but has no builder, payload-timeliness or gloas proposer + support). - **Sync duties** read the head state's `current_sync_committee` for an epoch in the head's own sync committee period and `next_sync_committee` for the one after; any other period is a `400` (an earlier one would need a historical @@ -383,6 +386,46 @@ Serving `/eth/v2/debug/beacon/states/finalized` as SSZ is what makes this client checkpoint-syncable from itself: it is the exact path [`checkpoint_sync.rs`](./checkpoint_sync.md) fetches from other clients. +### `GET /eth/v1/beacon/execution_payload_envelopes/{block_id}` + +Gloas moves a block's execution payload into a separate signed envelope. This +serves the one the node has verified, as `{version: "gloas", execution_optimistic, +finalized, data}` JSON or as SSZ on `Accept: application/octet-stream`, with +`Eth-Consensus-Version: gloas` either way. It is also what a checkpoint-syncing +node asks for the payload of its anchor (see +[`checkpoint_sync.md`](./checkpoint_sync.md)). + +| Status | When | +|--------|------| +| `400` | `block_id` is malformed | +| `404` | the block is unknown, or the node holds no envelope for it | + +The second `404` is deliberately one status for several situations: the store +keeps verified envelopes only, so a withheld payload, a payload not yet +received and a pre-gloas block (which has no envelope; the specification's +error list names no other status for it) are indistinguishable here. A caller +must read `404` as "unknown to this node", not as "the payload is empty". + +### `execution_optimistic` under gloas + +Before gloas a block's payload ran inside the block, so the flag is whether the +block is in the store's optimistic set. From gloas on the payload is a separate +object that may arrive late or never, and two rules share the flag: + +- an **envelope** response is optimistic iff that payload's verdict is not + `VALID` (an unrecorded verdict reads as `NOT_VALIDATED`); +- a **block, header, state or finality-checkpoints** response for a gloas block + is optimistic iff the latest FULL payload the block builds on is not `VALID`. + The walk starts at the block's parent: if the block's bid names the parent's + payload as its parent block hash it reads the parent's verdict, and otherwise + it steps back to the parent and repeats. A pre-gloas block on the way reads + the optimistic set. The block's own payload does not count, so a withheld + payload does not make the chain look unverified. + +Pre-gloas responses are unchanged. One helper implements both rules +(`shared/optimistic.rs`) and every response that carries the flag goes through +it. + ### `GET /eth/v1/config/spec` One flat object holding the network's configuration, the compiled preset, and @@ -390,15 +433,21 @@ the specification's constants, as the Beacon API asks. Validator clients depend on it: lighthouse's refuses a beacon node whose `PRESET_BASE` does not match its own, and treats an absent key as a mismatch. -The key set is lighthouse's, less gloas's preset and constant keys -(`beacon::preset` and `beacon::constants` hold them, such as -`DOMAIN_BEACON_BUILDER` and `DOMAIN_PTC_ATTESTER`, but this list does not -report them yet; gloas's schedule, timing and churn keys are reported, since -they are `Config` fields) and three keys the specification does not define -(`GAS_LIMIT_ADJUSTMENT_FACTOR`, `RESP_TIMEOUT`, `TTFB_TIMEOUT`). Domain types -and withdrawal prefixes are `0x`-prefixed hex; `VERSIONED_HASH_VERSION_KZG` is -a decimal, as lighthouse reports it. `GENESIS_TIME` is absent: -`/eth/v1/beacon/genesis` reports it. +The key set is lighthouse's plus gloas's preset and constant keys, read from +`beacon::preset` and `beacon::constants` rather than typed here: `PTC_SIZE`, +`MAX_PAYLOAD_ATTESTATIONS`, `MAX_BUILDER_DEPOSIT_REQUESTS_PER_PAYLOAD`, +`MAX_BUILDER_EXIT_REQUESTS_PER_PAYLOAD`, `MAX_BUILDERS_PER_WITHDRAWALS_SWEEP`, +the four gloas domains (`DOMAIN_BEACON_BUILDER`, `DOMAIN_PTC_ATTESTER`, +`DOMAIN_PROPOSER_PREFERENCES`, `DOMAIN_BUILDER_DEPOSIT`), the builder +withdrawal prefix and request types, and the `BUILDER_INDEX_*`, +`BUILDER_PAYMENT_THRESHOLD_*` and `PAYLOAD_BUILDER_VERSION` constants. Gloas's +schedule, timing and churn keys are `Config` fields, so they were already +reported. Three keys are left out because the specification does not define +them (`GAS_LIMIT_ADJUSTMENT_FACTOR`, `RESP_TIMEOUT`, `TTFB_TIMEOUT`), and so are +gloas's `MAX_*_SIZE` preset keys, which bound gossip message sizes the node does +not run on. Domain types and withdrawal prefixes are `0x`-prefixed hex; +`VERSIONED_HASH_VERSION_KZG` is a decimal, as lighthouse reports it. +`GENESIS_TIME` is absent: `/eth/v1/beacon/genesis` reports it. The configuration keys come from the `Config` the data directory was initialized with. Some of them (the custody and subnet counts, the diff --git a/docs/spec_deviations.md b/docs/spec_deviations.md index f16345639..168646676 100644 --- a/docs/spec_deviations.md +++ b/docs/spec_deviations.md @@ -234,8 +234,9 @@ that the tests compare like with like. it, and its vote walks the block table and contributes nothing. - **Missing timeliness** (`should_apply_proposer_boost`). A candidate with no `block_timeliness` entry reads as not timely by either deadline, where the - specification indexes the entry directly. The entries are in memory only, so a - restart empties them. The reading can only miss withholding a boost from an + specification indexes the entry directly. A gloas block's entry is persisted + (`Table::BlockTimeliness`) and reloaded on resume; a pre-gloas block's is in + memory only, so a restart empties it. The reading can only miss withholding a boost from an early equivocation, never withhold one wrongly. - **Equivalence:** each is the specification's answer wherever the specification has one; they differ only where it would have raised. @@ -324,3 +325,72 @@ node, and each call walks every latest message and every ancestor step. `gloas_get_weight` remain, as the references the walk is compared with, at every `head` check of every fork-choice fixture and on randomized trees; `compute_head` is the pre-gloas reference. + +## The execution client judges a gloas payload before the envelope is applied + +`verify_execution_payload_envelope` asks the execution engine last, as part of +one boolean function. The follower runs the pure consensus checks first +(`check_execution_payload_envelope`: signature, bid, state, data availability), +then asks the engine with `engine_newPayloadV5` only about an envelope that +passed them, and records it with `accept_execution_payload_envelope`. The +engine's part is answered "valid" inside the checks because its answer has +three outcomes and the specification's `ExecutionEngine` has two. + +- **`VALID`:** applied; the payload is recorded `VALID`. +- **`SYNCING` / `ACCEPTED`:** applied, since the consensus checks passed and the + payload is `NOT_VALIDATED` (`optimistic-sync.md`); recorded `SYNCING`, which + the attestation gossip rules already read as not yet validated. A later + `forkchoiceUpdated` `VALID` for the head, or for a descendant payload, moves + it and every optimistic payload on that chain to `VALID`. Unlike a pre-gloas + block there is no `is_optimistic_candidate_block` gate: a payload is applied + to a block the consensus layer already imported, so there is no + merge-transition poisoning to guard against. +- **`INVALID`:** not applied, so the FULL node of the block never exists; the + payload is recorded `INVALID`, children held for it are dropped and later + blocks that name it as their FULL parent are dropped instead of held. The + block itself and its EMPTY branch stay. `latestValidHash` can reach further + back, in which case the first payload after it is condemned the same way + (`beacon_payloads::resolve_invalid_payload`, which skips blocks that carry no + payload, unlike the pre-gloas `resolve_invalid_block`). A `forkchoiceUpdated` + `INVALID` for a payload already applied removes it again (its FULL node, its + stored envelope, and every child built on it). +- **A hash mismatch (`INVALID_BLOCK_HASH`, or `INVALID` with a null + `latestValidHash`):** the envelope's contents do not hash to the block hash it + claims, so only that envelope (by its own root) is refused; the root's + payload is not condemned, since the builder's real envelope may still arrive. + ethrex has no `INVALID_BLOCK_HASH` status: it answers a block hash mismatch + with `INVALID` and a null `latestValidHash`, and an executed-and-failed + payload with `INVALID` and its last valid ancestor. So a null one is read as + a mismatch and a non-null one as the payload being invalid. A payload that + failed with no valid ancestor to name is read as a mismatch too, which only + costs a refused envelope. +- **No answer after the retry ladder:** the payload is not applied, as for a + block, but unlike a block the envelope is consensus-valid and already held, + so it is kept and the per-slot redrive asks again, about the oldest held + envelope only and the rest once that ask is answered, so a down client costs + one retry ladder per slot however many wait. It is not fetched again: + that would repeat the ladder for every copy that arrives. + +`latestValidHash` and `VALID` follow the execution-layer chain, not every +verified ancestor: a gloas block's EL parent is the ancestor whose payload hash +is its `bid.parent_block_hash`, skipping blocks it built past on their EMPTY +node, and a pre-gloas block's is its parent. Both walks stop at finality. + +## `forkchoiceUpdated` for a gloas head + +- The head hash follows the head node: `bid.block_hash` for a FULL head, + `bid.parent_block_hash` for an EMPTY one. A head whose payload status has not + been computed yet is skipped. +- `finalized_block_hash` is the finalized block's `bid.parent_block_hash`, as + the specification has it. +- `safe_block_hash` is the justified block's `bid.parent_block_hash`. The + specification's `get_safe_execution_block_hash` is built on fast + confirmation, which the follower does not run, so this is the fallback. It + can only be older than the confirmed block, which keeps it safe. +- A pre-gloas checkpoint block, including one beneath a gloas head, keeps its + own payload hash. +- The call is V4 with the node's custody columns when the head block is gloas, + and V3 before; `engine_newPayloadV5` is for envelopes and V4 for pre-gloas + blocks. +- A hash of zero is "nothing to say" on either fork, so a payload whose hash is + zero (the fixtures' placeholder) sends no `forkchoiceUpdated`.