Skip to content

fix: optimize l1 sync with parallel traces and batching 1224 - #1953

Merged
joanestebanr merged 15 commits into
feat/l1-bridge-claim-multidownloaderfrom
perf/l1-multidownloader-h5
Nov 20, 2026
Merged

joanestebanr merged 15 commits into
feat/l1-bridge-claim-multidownloaderfrom
perf/l1-multidownloader-h5

Conversation

@joanestebanr

@joanestebanr joanestebanr commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

🔄 Update — 20 November 2026

  • New head 32ee7feb, stacked on the feat: detect reorgs through the multidownloader in l1 bridge and claim syncers #1994 head 52ab74c8 (the current base).
    The current stack references in this body are updated to this pair.
  • The previous head ccbcea69 is now historical, not current. Every benchmark figure
    below was measured on a former head (03f4c143 / ac08f230 / 5453f356) and is not
    re-measured on 32ee7feb; the old figures and SHAs are preserved explicitly as former-head
    benchmarks
    , not replaced.
  • Testing (scoped, this revision): -race over the full 7 trees
    (cmd, bridgesync, claimsync, multidownloader, sync, config, types) passed; lint
    passed (0 issues). Binaries built with -buildvcs=true report vcsModified=false.
  • Live replay / E2E: not claimed for this revision.

🔄 Changes Summary

Performance follow-up to the shared L1 multidownloader integration (PR1). Two optimizations,
both opt-in and scoped to the multidownloader-backed L1 syncers; the legacy downloader and
every L2 syncer keep their exact existing one-block-at-a-time behaviour.

  • Bounded claim-trace prefetch (claimsync): the ordinary and pre-Etrog ClaimEvent root
    debug_traceTransaction (call-tracer) calls of a downloaded log range are prefetched with a
    bounded pool of 8 concurrent requests (ClaimTracePrefetchConcurrency = 8) instead of one
    synchronous trace per log. Prefetch is installed after the coexistence hook that drops an
    ordinary ClaimEvent superseded by a DetailedClaimEvent, so a suppressed log is never traced;
    the DetailedClaimEvent handler never traces.
  • Atomic publication + double canonical revalidation: root traces are published only when every
    needed request succeeded. Canonical inclusion of the window is revalidated twice — once before any
    trace RPC and once after all workers join, immediately before publishing. A trace carries no block
    hash, so the second read is what ties the snapshot to the revalidated blocks. This adds one extra
    serial canonical header read per distinct block (2 × distinct blocks for the range). A changed or
    absent header returns ErrTransactionInclusionMismatch, so the whole range is refetched and
    nothing is published; every failure path purges the snapshot so a stale generation is never served.
  • Atomic batched persistence (sync / bridgesync / claimsync): the EVM driver may hand an
    ordered batch of already-available blocks to the processor in a single all-or-nothing SQL
    transaction
    , bounded to sync.MaxBatchBlocks = 100. The driver only drains what the downloader
    already produced (never waits to fill a batch). Both batch and single-block paths share the same
    inner per-block helper (block row, bridge/claim handlers, full proof and metadata fields, claim
    proof-decoder fallback, empty blocks). Bridge subscribers and post-commit logs are emitted only
    after the single commit, in original download order.
  • Robustness: a node answering a trace with a nil error but no call frame (e.g. JSON null) is
    treated as a retryable error, never an inclusion mismatch, and is not cached as an empty root;
    the check is scoped to the frame type so a legitimate reverted root and a router/fallback wrapper
    that nests the bridge call both stay valid.
  • Docs/tests: docs/l1_shared_downloader.md section for the two optimizations and the bridge
    tree-cache rollback limitation; new claimsync/trace_prefetch.go (+tests),
    claimsync/processor_batch_test.go, bridgesync/processor_batch_test.go, sync/driver_batch_test.go.

Stacked PR: depends on #1994 — #1994
(feat: detect reorgs through the multidownloader in l1 bridge and claim syncers),
which itself is stacked on #1952. Base of this branch is the current #1994 head
52ab74c8 (at PR open the base was 86c71d8a3b452eb0ddd9d71e17e6af8da36dd497).

🔄 Review follow-up — 16 Nov 2026

The PR head moved from 03f4c143 to ccbcea69 to address the review threads. The new
commit is signed (SSH ED25519) and GitHub reports it verified: valid:

  • ccbcea69 — fix(sync): address trace prefetch and batching review
    (ccbcea6)

Key changes:

  • Enricher install is fail-closed: InstallLogsEnricher returns (prev, bool); Bridge
    and Claim startup abort with ErrLogsEnricherUnsupported when a multidownloader-selected
    syncer cannot install the trace enricher, instead of starting handlers that can never
    resolve a prefetched trace.
  • Canonical revalidation dedups distinct blocks: revalidateCanonical reads one header
    per distinct block per call through a bounded (≤ 8) worker pool, once before the trace
    RPCs and once after all workers join; multidownloader headers are deliberately not
    cached, so the cost stays 2N per range and canonical correctness is preserved.
  • Single transaction lifecycle: ProcessBlock delegates to processBlocks in both
    Bridge and Claim, so the halted check / NewTx / rollback / commit / notification path
    lives in one place; the ad-hoc alias 100 cap was replaced by sync.MaxBatchBlocks.

Testing — local whole-repo CI at the new head ccbcea69:

MallocNanoZone=0 go test -count=1 -short -race -p 1 -covermode=atomic -timeout 15m ./...

→ exit 0, 88 packages ok, 0 failures (sequential, isolated worktree).

Benchmark scope (unchanged): the Cardona and Mainnet figures below were measured on
the former head 03f4c143 (and H0 ac08f230) before this review follow-up. They
are not re-measured on ccbcea69 and are not a guarantee for the current revision;
the fixed ranges, dates, percentages and proof counts are unchanged.

⚠️ Breaking Changes

  • 🛠️ Config: None. ClaimTracePrefetchConcurrency and sync.MaxBatchBlocks are fixed internal
    constants, not operator-facing knobs.
  • 🔌 API/CLI: None.
  • 🗑️ Deprecated Features: None.

📋 Config Updates

  • 🧾 Diff/Config snippet: None. The optimizations activate implicitly for multidownloader-backed
    L1 syncers; legacy/L2 paths are unchanged.

🔌 API Updates

🔌 Bridge service API

  • 🔌 Bridge service API Update: No endpoint changes. Only persistence batching / post-commit
    notification ordering change (same per-block semantics, single commit). Not a breaking interface
    change.

🔌 Proxy API

  • 🔌 AProxy API Update: N/A.

🔌 Others API

  • 🔌 Others API Update: N/A.

✅ Testing

  • 🤖 Automatic — scoped over the 7 affected package trees
    (cmd, bridgesync, claimsync, multidownloader, sync, config, types), toolchain go1.27.3. This is
    not a full-repo run and not E2E:

    • go test -count=1 -short -race over those 7 trees → all ok (log: logs/h5-scoped-test.log).
    • go vet and gofmt clean.
    • golangci-lint run --timeout 5m over the same 7 trees → 0 issues
      (log: logs/h5-safety-lint.log).
    • make build → exit 0 (log: logs/h5-build.log; binaries to git-ignored h5/target/).
    • Secrets scan clean; sidecar logs contain no secrets.
    • New coverage: bounded prefetch pool, canonical revalidation (before/after), nil-frame retryable
      rejection, snapshot purge on failure, atomic batch commit/rollback, notification ordering.
  • 🖱️ Manual / E2E — safety scenario: the actual H5-safety body test exercised a
    reorg-blocked RPC and a malformed-then-healthy retry (log: logs/h5-safety-test.log).

  • 📊 Cardona historical synchronization benchmark — 13 November 2026 (from the actual
    experiment reports, not a re-run):

    • Frozen fixed range, not a live tip: both variants replayed the same frozen historical
      Cardona L1 height range from the fixed InitialBlock = 4,789,186 up to the fixed per-syncer
      cutoffs, so the comparison uses identical block ranges and identical cutoffs. H0 was run on
      12 November 2026 and H5 on 13 November 2026; the comparison is documented on 13 November 2026.
      The current network head on 13 November 2026 is not claimed, no new RPC/network run was
      performed for this PR, and the current chain head is unknown.
    • Inclusive block counts (each syncer replays its range independently):
      • l1infotreesync: initial 4,789,186 → target 12,119,079 ⇒
        12,119,079 − 4,789,186 + 1 = **7,329,894** blocks.
      • l1bridgesync and l1claimsync: initial 4,789,186 → target 12,119,148 ⇒
        12,119,148 − 4,789,186 + 1 = **7,329,963** blocks.
      • That is ~7.33 million historical L1 heights replayed per syncer — not a 0-origin
        scan and not the sum of the three syncers running in parallel.
    • Variant labels (experiment identifiers, not releases or tags):
      • H0 = shared L1 multidownloader integration baseline + canonical-header idempotency
        fix
        , without parallel claim-trace prefetch or block-batched persistence.
      • H5 = the same baseline + 8-way claim-trace prefetch + atomic 100-block SQL batch.

    Human-friendly times (rounded to the nearest second):

    Syncer H0 baseline H5 optimized Time reduction
    l1infotreesync 38 min 25 s 37 min 06 s 3.44 %
    l1bridgesync 49 min 08 s 37 min 21 s 23.99 %
    l1claimsync 1 h 13 min 15 s 41 min 46 s 42.97 %
    total (all three complete) 1 h 13 min 15 s 41 min 46 s 42.97 %
    • The total row is the wall-clock time until all three syncers complete — i.e. the
      max of the three components, not their sum — so it coincides with the l1claimsync
      (longest) row.
    • Relative improvement (derived from the unrounded times, not a stored field):
      (1 − H5/H0) × 100; for the total/max row:
      (1 − 2506.17699265 / 4394.80259056) × 100 ≈ 42.974 % ≈ 42.97 %.
    • Synchronized data within the fixed comparison ranges (rows actually written, from the
      same validation reports; identical for both variants because they replay the exact same frozen
      range and must produce identical business data):
    Synchronized data within the fixed comparison ranges H0 H5
    L1InfoTree leaves (l1info_leaf) 229,510 229,510
    Bridges (bridge) 218,283 218,283
    Claims (claim) 63,067 63,067
    • What these counts are — and are not: these are the synchronized data rows (L1 info-tree
      leaves, bridge events, claims) inside the fixed comparison window, not the scanned block
      heights
      (~7.33 million historical heights per syncer, listed above). They are also not
      overshoot counts
      : rows synced past the cutoffs (the raw reports' larger exp scan max) are
      excluded — the hidden-tail proof confirms srcOnly=0 / expOnly=0 beyond each cutoff.
    • Validated at the fixed cutoffs: L1InfoTree leaf cutoff block 12,119,079; bridge and
      claim cutoff block 12,119,148.
    • Identical implementations: the all-column keyed full-outer merge reports same = total,
      different = 0, refOnly = 0, expOnly = 0, schemaMatch = true for all three tables — so
      there are no missing, extra, or field-level differences between H0 and H5.
    • Evidence: H5 reports/H5/20261113T073149Z-6adddcdb/validation.json and H0
      reports/H0/20261112T235557Z-recover-ac63c644/validation.json.
    • Sources: H0 reports/H0/20261112T235557Z-recover-ac63c644/case-result.json
      (frozen original, pre-hardening); H5 reports/H5/20261113T073149Z-6adddcdb/case-result.json.

📊 Performance — Mainnet — 13–14 November 2026

Same fixed window and same historical dataset for every variant: initialBlock = 16,896,718;
frozen LatestBlock targets L1InfoTreeSync = 26,407,018 and Bridge/Claim = 26,407,089
(inclusive spans: 9,510,301 and 9,510,372 blocks). The target snapshot was captured
2026-11-13T15:58:41Z on chainId 1 (latest 26,407,089, finalized 26,407,018); these
results are from 2026-11-14. This is a frozen snapshot, not the live chain head today.
All three variants replay the same public Katana source (identical for every variant; the
source URL/token is never recorded) — shared source-config digest
786fcb8660dee2c015b70aefcdf627a2a6cad6c505c987d9f79158de77d69043.

  • H0 = PR1 head (commit ac08f230…, binary sha256 256c753b…): shared L1 multidownloader with
    the canonical-header idempotency fix, without prefetch/batching.
  • H5 = the PR head at measurement time (commit 03f4c143…, binary sha256 c9a9bb9f…),
    before the 16 Nov review follow-up (then-current head ccbcea69…, now historical):
    H0 plus the
    8-way claim-trace prefetch and the atomic 100-block SQL batch, including the post-canonical
    hardening
    . Both H0/H5 binaries were built fresh from the then-current signed commits
    (-buildvcs=true, vcsModified=false, embedded vcs.revision = signed HEAD).
syncer H0 baseline H5 optimized Time reduction
L1InfoTreeSync 55 min 08 s 52 min 00 s 5.67 %
Bridge 55 min 10 s 52 min 02 s 5.67 %
Claim 9 h 22 min 52 s 3 h 14 min 08 s 65.51 %
Total (all three complete, parallel max, not sum) 9 h 22 min 52 s 3 h 14 min 08 s 65.51 %
  • Reductions are computed from the unrounded times ((1 − H5/H0) × 100).

  • This H5 is the current post-canonical-hardening revision, unlike the older Cardona H5
    numbers above, which were captured on the earlier pre-hardening build. The Cardona figures above
    are unchanged by this run, and the two benchmarks are not merged.

  • L1InfoTreeSync already ran on the default multidownloader in both variants, so its 5.67 %
    difference is run-to-run timing variation and must not be attributed to the prefetch/batching
    feature; the stock Info run is never labelled "legacy".

  • The total row is the wall clock until all three syncers complete = the max (the
    Claim row), not the sum, i.e. the single-controller wall clock including the final
    validation.

  • One trial per variant, same provider, the existing Docker container kept active, variable RPC
    load and no control run: the differences are not statistically significant.

  • Compared data (identical in all three comparisons) — from the strict compare
    src_count / loc_count / datasets_equal fields, 0 differences: 124,918 L1InfoTree leaves
    (l1info_leaf), bridges 265,822 (bridge), claims 114,300 (claim).
    These are the business rows written inside the fixed window, not scanned block heights and
    not overshoot rows.

  • Comparison strictness: schema match, integrity, claim proof-field formats (32 comma-separated
    32-byte hashes), metadata/address fields, root history, event hashes, keys and all columns —
    0 diff across H0-vs-STOCK, H5-vs-STOCK and H5-vs-H0, all ok=true with every component
    verdict true. This is a data/format comparison: it does not claim cryptographic on-chain
    proof validation or internal RHT payload equality (tree-node tables are not compared;
    validated=false).

  • 3-variant reference (same frozen inputs):

    syncer stock 5453f356 H0 ac08f230 H5 03f4c143
    L1InfoTreeSync 46 min 27 s 55 min 08 s 52 min 00 s
    Bridge 3 h 17 min 30 s 55 min 10 s 52 min 02 s
    Claim 10 h 49 min 31 s 9 h 22 min 52 s 3 h 14 min 08 s
    Wall-clock (parallel max) 10 h 49 min 31 s 9 h 22 min 52 s 3 h 14 min 08 s

    Stock used legacy bridge/claim direct-RPC downloaders and L1 info on the default
    multidownloader; H5 total vs legacy stock: −70.11 %.

  • Evidence (read-only comparisons, no writes): summary.json, reference-integrity.json,
    stock-report.json, h0-report.json, h5-report.json, H0-vs-STOCK.json,
    H5-vs-STOCK.json, H5-vs-H0.json. All three variants: status=completed,
    nodeExit code=0 / SIGINT unexpected=false, harness/reader/fatal = none.

  • Disk / retention (optional note): the mainnet data was produced fresh with no reused cache;
    the only reclaim was an authorized Cardona-only cleanup (case P1, 15 files, ~6.517 GiB)
    plus removal of the H0 intermediate DBs after all three comparisons — the stock + H5 DBs and all
    reports were retained, and the source Docker/config was untouched.

🐞 Issues

🔗 Related PRs

📝 Notes

  • Measurement caveat (important): the figures above are a single trial on the old,
    frozen b9 build captured before the post-canonical hardening and the latest develop
    adaptation. They are therefore not a performance guarantee for the current PR revision.
    No new/magical timings are claimed.
  • Current PR scope is bounded: prefetch uses an 8-call concurrent context; there are
    two canonical header checks per distinct block in total (one before the trace, one after); proofs and the decoder are unchanged; SQL
    batching is bounded to 100 blocks; post-commit Publish calls are retained; legacy/L2 fallbacks
    untouched. The overhead of the post-canonical path is unmeasured.
  • Correctness evidence (H0 and H5 validation): all business tables 0 diff —
    bridge 218283, claim 63067, l1info_leaf 229510; claims valid-proof full root;
    histories/headers compared; internal tree payload (l1_info_rht, rht, rollup_exit_rht)
    excluded. These original H5 runs do not claim node-proof stock.
  • Bridge tree-cache rollback limitation: on a processing failure the bridge processor rolls back
    and runs the PutLeaf rollback callbacks, but db.Tx.Rollback returns early on sql.ErrTxDone
    and skips those callbacks, so the cache is not invalidated on that specific path (documented gap).
  • Commits: 03f4c1437096c7f513e6c83d68c2b6de0389cf6d (branch perf/l1-multidownloader-h5,
    12 files — 7 modified, 5 new), plus the 16 Nov review follow-up ccbcea69.
    Current head 32ee7feb; the previous head ccbcea69d604469a562394b900c64be19380eeb1 is
    now historical.
    All signed (SSH ED25519).

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T08:58:00.412109Z bb7a4ac Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@joanestebanr
joanestebanr marked this pull request as draft November 13, 2026 14:31
@joanestebanr joanestebanr self-assigned this Nov 16, 2026

@joanestebanr joanestebanr left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review with Claude Code (/code-review, high effort). Findings are deduplicated but were not individually verified, so please double-check before acting on them. A manual security pass over the diff found no exploitable issues (SQL uses placeholders/sqlx.In; RPC transactions are validated against the log before caching; no new endpoints or secret logging).

No definite correctness bug found in the batching / trace-prefetch code itself; the inline comments are one robustness risk plus cleanups.

Comment thread claimsync/claimsync.go Resolved
Comment thread claimsync/trace_prefetch.go Resolved
Comment thread bridgesync/processor.go Resolved
@joanestebanr
joanestebanr force-pushed the perf/l1-multidownloader-h5 branch from 03f4c14 to ccbcea6 Compare November 16, 2026 13:34
@joanestebanr
joanestebanr marked this pull request as ready for review November 16, 2026 14:38
@joanestebanr
joanestebanr force-pushed the feat/l1-bridge-claim-multidownloader branch from e0820f7 to c15a55e Compare November 19, 2026 08:53
@joanestebanr
joanestebanr force-pushed the perf/l1-multidownloader-h5 branch from ccbcea6 to bb7a4ac Compare November 19, 2026 08:55
@joanestebanr

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: bb7a4ac2a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@joanestebanr
joanestebanr force-pushed the perf/l1-multidownloader-h5 branch from bb7a4ac to af4c1f1 Compare November 19, 2026 09:06
@joanestebanr
joanestebanr force-pushed the feat/l1-bridge-claim-multidownloader branch from 8ceb2ab to f4e25f7 Compare November 19, 2026 09:24
@joanestebanr
joanestebanr force-pushed the perf/l1-multidownloader-h5 branch 2 times, most recently from f385b6a to 54d7e09 Compare November 19, 2026 09:43
@joanestebanr
joanestebanr force-pushed the feat/l1-bridge-claim-multidownloader branch from 3112484 to cb4400c Compare November 19, 2026 10:27
@joanestebanr
joanestebanr force-pushed the perf/l1-multidownloader-h5 branch from 54d7e09 to 52bd804 Compare November 19, 2026 10:28
@joanestebanr
joanestebanr force-pushed the feat/l1-bridge-claim-multidownloader branch from cb4400c to c90a8b8 Compare November 19, 2026 11:37
@joanestebanr
joanestebanr force-pushed the perf/l1-multidownloader-h5 branch from 52bd804 to 1f499c7 Compare November 19, 2026 11:40
joanestebanr and others added 2 commits November 19, 2026 14:56
…d claim syncers

When the L1 bridge syncer or the L1 claim syncer select the multidownloader
they now run on the native multidownloader sync driver instead of the legacy
driver and the legacy reorg detector. The multidownloader is the single source
of truth for logs, headers, reorgs and finality.

- multidownloader/sync: logs hook, logs enricher and contextual appenders in
  the downloader; lazy mode and SyncNextBlock bootstrap in the driver;
  BlockByBlockProcessor adapter that keeps the post-commit subscriptions
- multidownloader: GetLastReorgData, used for the last reorg event
- IsBlockFinalized reads the finality flag stored by the multidownloader
- reorgdetector: Unsubscribe, so a syncer migrated from the legacy downloader
  leaves no tracked blocks that would block the detector
- mdrsynctest: in-memory multidownloader for end-to-end syncer tests

Co-Authored-By: Claude Fable 5.2 <[email protected]>
… claim syncers

BridgeL1Sync.Downloader and ClaimL1Sync.Downloader default to
"multidownloader". The L2 sections keep "legacy", the only value they support.

A configuration that disables L1Multidownloader must now select
Downloader = "legacy" explicitly in both L1 sections: there is no silent
fallback.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
@joanestebanr
joanestebanr force-pushed the perf/l1-multidownloader-h5 branch from 1f499c7 to 9428907 Compare November 19, 2026 14:49
@joanestebanr
joanestebanr changed the base branch from feat/l1-bridge-claim-multidownloader to feat/l1-bridge-claim-native-md-driver November 19, 2026 14:50
Resuming a multidownloader DB that already has logs for a downloaded
range (Safe/Step re-querying a covered window) hit UNIQUE on
logs(address, block_number, log_index) and stalled the whole downloader.

Mirror the block-header path: INSERT, and on a primary-key collision
keep the stored row when the payload matches. A contradictory payload
is ErrLogIntegrityConflict and never overwrites.

Refs #1224
@joanestebanr
joanestebanr force-pushed the perf/l1-multidownloader-h5 branch from 9428907 to 134a8db Compare November 20, 2026 07:18
A log is canonical only within the block identified by (block number,
block hash). Add one shared guard, CheckLogsMatchBlockHeaders /
CheckLogsMatchBlockHeaderMap, and apply it before any state or storage
mutation, replacing the old TODO in saveLogsAndBlocks:

- StepSafe fails closed: a downloaded batch whose log does not match the
  block header retrieved for its block returns the typed
  ErrLogBlockHeaderMismatch before the in-memory state is advanced, so an
  invalid batch never reaches storage.
- SaveEthLogsWithHeaders rejects, before saveLogsAndBlocks, a log whose
  block number has no header (including a nil header entry), a log whose
  hash contradicts the header, and contradictory duplicate headers for the
  same block number.
- SaveEthLogs validates the logs against the block rows derived from the
  logs themselves, catching a later log in the same block that carries a
  different hash instead of letting the first one silently define the
  block.

The guard runs outside the storage lock and before any write, so a
contradictory batch performs no write at all. Empty batches remain valid.
The same guard backs both the step logic and the storage APIs so the rule
cannot drift.

Refs #1224
Query only addresses pending over the whole range; split at join and leave boundaries and ignore empty segments. Refs #1224
Keep conversion consistent with the shared integrity guard, cover mismatched duplicate replays, and document pending-query boundaries. Refs #1224
@joanestebanr
joanestebanr force-pushed the perf/l1-multidownloader-h5 branch from 134a8db to 32ee7fe Compare November 20, 2026 09:36
@joanestebanr

Copy link
Copy Markdown
Collaborator Author

Rebased on #1994 head 52ab74c; current H5 head 32ee7fe preserves all four H5 patches. Scoped race tests, lint and clean-tree build pass. Mainnet original-H5 snapshot replay confirmed the initial query excludes Bridge from its already-covered 83-block prefix, then includes it from 26414125. Bridge/Claim/MD reached 26455471, Info 26455404; original conflicting log remains one identical row. No UNIQUE/panic/fatal/log-header mismatch errors. This is bounded functional verification (131.396 seconds, intentional SIGINT/exit 0), not a completed five-minute soak or a new historical benchmark. Existing reorg E2E failures remain separately tracked.

@joanestebanr joanestebanr left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review notes on head 32ee7feb. The design looks sound to me: prefetch is installed after the coexistence hook, publication is all-or-nothing, the batch path reuses processBlockInTx, and the chunking in BlockByBlockProcessor keeps retries idempotent. I left a few inline comments: one doc mismatch, one tree-cache note and two nits.

One general point about the description: all the benchmark numbers were measured on former heads (03f4c143, and the older pre-hardening build for Cardona), with one trial per variant on mainnet. The body already says this, but the headline gains (−65.51 % claim sync on mainnet) are what reviewers will remember. Could we re-run at least the mainnet claim syncer on the current head before merging, or move the caveat up next to the tables? Also, bridge sync improves 23.99 % on Cardona but only 5.67 % on mainnet, which is the same delta as L1InfoTreeSync and so may be run-to-run noise. It would help to say whether batching has a measurable effect on bridge sync by itself.

Comment thread docs/l1_shared_downloader.md Outdated
after all workers have joined, immediately before publishing. A trace carries no block hash, so
the second read is what ties the published snapshot to the blocks the range was revalidated
against. This costs **one extra canonical header read per distinct block** (2 × distinct blocks
for the range), read serially, on top of the trace RPCs.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This says the header reads are done serially, and the PR description says the same ("one extra serial canonical header read per distinct block"). Since the 16 Nov follow-up, revalidateCanonical runs them through runBoundedPool with up to ClaimTracePrefetchConcurrency (8) workers (claimsync/trace_prefetch.go:275). Suggest updating both texts to "read through a bounded pool of at most 8 workers".

Comment thread bridgesync/processor.go
// the next batch rebuilds it from the database. Limitation:
// db.Tx.Rollback returns early (sql.ErrTxDone) when the underlying
// transaction already ended and then skips those callbacks, so on that
// path the cache is not invalidated here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About the documented sql.ErrTxDone limitation (also in docs/l1_shared_downloader.md under "Bridge tree-cache rollback note"): it seems to be covered already by AppendOnlyTree.PutLeaf. After a failed batch the stale cache has lastIndex = N+k (k ≥ 1 leaves added in the rolled-back tx), while the retried batch starts at leaf N+1. So leaf.Index != t.lastIndex+1 and initCache(tx) rebuilds the cache from the DB before using it. If k = 0 the cache was never touched. If that's right, the doc could describe it as "self-healing on the next PutLeaf via the index check" rather than an open gap. A test for that case would be good: commit fails or the ctx times out after PutLeaf, then retry the same batch and check the roots.

This path also matters more with batching. withDatabaseTimeout (default DBQueryTimeout = 5m) now covers up to 100 blocks in one tx. When dbCtx expires, database/sql rolls the tx back on its own, and the explicit Rollback() then hits exactly the ErrTxDone branch. That's fine given the self-heal above, but it is now a realistic path rather than a corner case.

Comment thread sync/evmdriver.go

for _, b := range blocks {
if err := d.trackNonFinalizedBlock(ctx, b); err != nil {
return nil

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this swallows the trackNonFinalizedBlock error and returns nil, so the caller does not tear down the generation and the batch is skipped. It copies handleNewBlock (line 476) and is practically unreachable today, because withRetry only returns non-nil for a stopRetryError and returns nil on ctx cancel. Still, for a batch this would silently drop up to MaxBatchBlocks blocks if withRetry ever changes. I'd return err here (and possibly in handleNewBlock too) so a future change fails loudly. This path is not used by the default L1 multidownloader driver, so it is low priority.

// RPCs than the trace phase. On the first error the remaining work is canceled,
// every worker is joined and that error is returned; a canceled download context
// returns the context error after joining.
func (r *TraceResolver) revalidateCanonical(ctx context.Context, targets []traceTarget) error {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit / observability: with the double revalidation, every range costs 2 × distinct claim blocks header RPCs on top of the trace RPCs. When claims are spread over many blocks, that can reach 2× the number of trace calls. The PR says this overhead is unmeasured. Could we add a counter or debug log per range (distinct blocks, header reads, traces, duration) so the cost shows up on a real node and we can decide later whether it is worth reading these headers from the multidownloader storage instead?

Base automatically changed from feat/l1-bridge-claim-native-md-driver to feat/l1-bridge-claim-multidownloader November 20, 2026 11:51
@joanestebanr
joanestebanr force-pushed the feat/l1-bridge-claim-multidownloader branch from eca4aed to f67517a Compare November 20, 2026 11:51
joanestebanr and others added 5 commits November 20, 2026 11:52
Bounded 8-way claim-trace prefetch with double canonical revalidation and
atomic publication, plus atomic 100-block batched L1 persistence in a single
SQL transaction. Opt-in and scoped to multidownloader-backed L1 syncers;
legacy and L2 paths unchanged.

Refs #1224
Address review feedback on the opt-in L1 trace-prefetch and batched
persistence optimizations (refs #1224):

- trace prefetch: revalidate the canonical inclusion of the window both
  before any trace RPC and again after all workers join, immediately
  before publishing, so a published snapshot is tied to the blocks the
  range was revalidated against; purge the snapshot on every failure path
  so a trace from an earlier generation is never served for a failed
  range; reject a nil-error-but-empty RPC root as a retryable error
  (never an inclusion mismatch) while keeping reverted roots and
  router/fallback wrappers valid.
- batching: keep the per-block semantics shared with the single-block
  path, emit subscribers and post-commit logs only after the single
  commit in download order, and roll the whole batch back on failure.
- claimsync: install the trace-prefetch enricher through the fail-closed
  InstallLogsEnricher hook, so a multidownloader-selected syncer aborts
  construction (ErrLogsEnricherUnsupported) instead of starting a syncer
  whose handlers can never resolve a prefetched trace.

Refs #1224
The base branch turned the ClaimEvent and pre-Etrog ClaimEvent handlers
into contextual appenders so their debug_traceTransaction runs with the
active download context. Build both the legacy and the prefetch variants
as contextual appenders, pass the context to the synchronous trace in
extractCallDataFromLog, and update the trace prefetch tests.

Refs #1224

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The H5 optimizations were written for the leftover legacy EVMDriver. After
#1994 the L1 bridge/claim syncers run on the native multidownloader driver,
so install the claim-trace enricher and the atomic 100-block processor
batch on that path and keep the legacy/L2 constructors one-block-at-a-time.

Refs #1224
@joanestebanr
joanestebanr force-pushed the perf/l1-multidownloader-h5 branch from 32ee7fe to d86cc36 Compare November 20, 2026 11:56
@joanestebanr

Copy link
Copy Markdown
Collaborator Author

Rebased on #1994 eca4aed, which includes the Unsubscribe race fix and E2E synchronization updates. Current head d86cc36 also corrects documentation: canonical header reads use a bounded worker pool, not serial reads. Scoped race tests and multidownloader lint pass; focused E2E range assertions pass locally. Waiting for the live E2E checks on the new heads; no full-CI-green claim yet.

@joanestebanr
joanestebanr merged commit 39a7d7b into feat/l1-bridge-claim-multidownloader Nov 20, 2026
40 checks passed
@joanestebanr
joanestebanr deleted the perf/l1-multidownloader-h5 branch November 20, 2026 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant