Repository navigation
fix: optimize l1 sync with parallel traces and batching 1224 - #1953
joanestebanr merged 15 commits into
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
joanestebanr
left a comment
There was a problem hiding this comment.
Automated review with Claude Code (/code-review, high effort). Findings are deduplicated but were not individually verified, so please double-check before acting on them. A manual security pass over the diff found no exploitable issues (SQL uses placeholders/sqlx.In; RPC transactions are validated against the log before caching; no new endpoints or secret logging).
No definite correctness bug found in the batching / trace-prefetch code itself; the inline comments are one robustness risk plus cleanups.
03f4c14 to
ccbcea6
Compare
e0820f7 to
c15a55e
Compare
ccbcea6 to
bb7a4ac
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
bb7a4ac to
af4c1f1
Compare
8ceb2ab to
f4e25f7
Compare
f385b6a to
54d7e09
Compare
3112484 to
cb4400c
Compare
54d7e09 to
52bd804
Compare
cb4400c to
c90a8b8
Compare
52bd804 to
1f499c7
Compare
…d claim syncers When the L1 bridge syncer or the L1 claim syncer select the multidownloader they now run on the native multidownloader sync driver instead of the legacy driver and the legacy reorg detector. The multidownloader is the single source of truth for logs, headers, reorgs and finality. - multidownloader/sync: logs hook, logs enricher and contextual appenders in the downloader; lazy mode and SyncNextBlock bootstrap in the driver; BlockByBlockProcessor adapter that keeps the post-commit subscriptions - multidownloader: GetLastReorgData, used for the last reorg event - IsBlockFinalized reads the finality flag stored by the multidownloader - reorgdetector: Unsubscribe, so a syncer migrated from the legacy downloader leaves no tracked blocks that would block the detector - mdrsynctest: in-memory multidownloader for end-to-end syncer tests Co-Authored-By: Claude Fable 5.2 <[email protected]>
… claim syncers BridgeL1Sync.Downloader and ClaimL1Sync.Downloader default to "multidownloader". The L2 sections keep "legacy", the only value they support. A configuration that disables L1Multidownloader must now select Downloader = "legacy" explicitly in both L1 sections: there is no silent fallback. Co-Authored-By: Claude Fable 5.2 <[email protected]>
1f499c7 to
9428907
Compare
Resuming a multidownloader DB that already has logs for a downloaded range (Safe/Step re-querying a covered window) hit UNIQUE on logs(address, block_number, log_index) and stalled the whole downloader. Mirror the block-header path: INSERT, and on a primary-key collision keep the stored row when the payload matches. A contradictory payload is ErrLogIntegrityConflict and never overwrites. Refs #1224
9428907 to
134a8db
Compare
A log is canonical only within the block identified by (block number, block hash). Add one shared guard, CheckLogsMatchBlockHeaders / CheckLogsMatchBlockHeaderMap, and apply it before any state or storage mutation, replacing the old TODO in saveLogsAndBlocks: - StepSafe fails closed: a downloaded batch whose log does not match the block header retrieved for its block returns the typed ErrLogBlockHeaderMismatch before the in-memory state is advanced, so an invalid batch never reaches storage. - SaveEthLogsWithHeaders rejects, before saveLogsAndBlocks, a log whose block number has no header (including a nil header entry), a log whose hash contradicts the header, and contradictory duplicate headers for the same block number. - SaveEthLogs validates the logs against the block rows derived from the logs themselves, catching a later log in the same block that carries a different hash instead of letting the first one silently define the block. The guard runs outside the storage lock and before any write, so a contradictory batch performs no write at all. Empty batches remain valid. The same guard backs both the step logic and the storage APIs so the rule cannot drift. Refs #1224
Query only addresses pending over the whole range; split at join and leave boundaries and ignore empty segments. Refs #1224
Keep conversion consistent with the shared integrity guard, cover mismatched duplicate replays, and document pending-query boundaries. Refs #1224
134a8db to
32ee7fe
Compare
|
Rebased on #1994 head 52ab74c; current H5 head 32ee7fe preserves all four H5 patches. Scoped race tests, lint and clean-tree build pass. Mainnet original-H5 snapshot replay confirmed the initial query excludes Bridge from its already-covered 83-block prefix, then includes it from 26414125. Bridge/Claim/MD reached 26455471, Info 26455404; original conflicting log remains one identical row. No UNIQUE/panic/fatal/log-header mismatch errors. This is bounded functional verification (131.396 seconds, intentional SIGINT/exit 0), not a completed five-minute soak or a new historical benchmark. Existing reorg E2E failures remain separately tracked. |
joanestebanr
left a comment
There was a problem hiding this comment.
Review notes on head 32ee7feb. The design looks sound to me: prefetch is installed after the coexistence hook, publication is all-or-nothing, the batch path reuses processBlockInTx, and the chunking in BlockByBlockProcessor keeps retries idempotent. I left a few inline comments: one doc mismatch, one tree-cache note and two nits.
One general point about the description: all the benchmark numbers were measured on former heads (03f4c143, and the older pre-hardening build for Cardona), with one trial per variant on mainnet. The body already says this, but the headline gains (−65.51 % claim sync on mainnet) are what reviewers will remember. Could we re-run at least the mainnet claim syncer on the current head before merging, or move the caveat up next to the tables? Also, bridge sync improves 23.99 % on Cardona but only 5.67 % on mainnet, which is the same delta as L1InfoTreeSync and so may be run-to-run noise. It would help to say whether batching has a measurable effect on bridge sync by itself.
| after all workers have joined, immediately before publishing. A trace carries no block hash, so | ||
| the second read is what ties the published snapshot to the blocks the range was revalidated | ||
| against. This costs **one extra canonical header read per distinct block** (2 × distinct blocks | ||
| for the range), read serially, on top of the trace RPCs. |
There was a problem hiding this comment.
This says the header reads are done serially, and the PR description says the same ("one extra serial canonical header read per distinct block"). Since the 16 Nov follow-up, revalidateCanonical runs them through runBoundedPool with up to ClaimTracePrefetchConcurrency (8) workers (claimsync/trace_prefetch.go:275). Suggest updating both texts to "read through a bounded pool of at most 8 workers".
| // the next batch rebuilds it from the database. Limitation: | ||
| // db.Tx.Rollback returns early (sql.ErrTxDone) when the underlying | ||
| // transaction already ended and then skips those callbacks, so on that | ||
| // path the cache is not invalidated here. |
There was a problem hiding this comment.
About the documented sql.ErrTxDone limitation (also in docs/l1_shared_downloader.md under "Bridge tree-cache rollback note"): it seems to be covered already by AppendOnlyTree.PutLeaf. After a failed batch the stale cache has lastIndex = N+k (k ≥ 1 leaves added in the rolled-back tx), while the retried batch starts at leaf N+1. So leaf.Index != t.lastIndex+1 and initCache(tx) rebuilds the cache from the DB before using it. If k = 0 the cache was never touched. If that's right, the doc could describe it as "self-healing on the next PutLeaf via the index check" rather than an open gap. A test for that case would be good: commit fails or the ctx times out after PutLeaf, then retry the same batch and check the roots.
This path also matters more with batching. withDatabaseTimeout (default DBQueryTimeout = 5m) now covers up to 100 blocks in one tx. When dbCtx expires, database/sql rolls the tx back on its own, and the explicit Rollback() then hits exactly the ErrTxDone branch. That's fine given the self-heal above, but it is now a realistic path rather than a corner case.
|
|
||
| for _, b := range blocks { | ||
| if err := d.trackNonFinalizedBlock(ctx, b); err != nil { | ||
| return nil |
There was a problem hiding this comment.
nit: this swallows the trackNonFinalizedBlock error and returns nil, so the caller does not tear down the generation and the batch is skipped. It copies handleNewBlock (line 476) and is practically unreachable today, because withRetry only returns non-nil for a stopRetryError and returns nil on ctx cancel. Still, for a batch this would silently drop up to MaxBatchBlocks blocks if withRetry ever changes. I'd return err here (and possibly in handleNewBlock too) so a future change fails loudly. This path is not used by the default L1 multidownloader driver, so it is low priority.
| // RPCs than the trace phase. On the first error the remaining work is canceled, | ||
| // every worker is joined and that error is returned; a canceled download context | ||
| // returns the context error after joining. | ||
| func (r *TraceResolver) revalidateCanonical(ctx context.Context, targets []traceTarget) error { |
There was a problem hiding this comment.
nit / observability: with the double revalidation, every range costs 2 × distinct claim blocks header RPCs on top of the trace RPCs. When claims are spread over many blocks, that can reach 2× the number of trace calls. The PR says this overhead is unmeasured. Could we add a counter or debug log per range (distinct blocks, header reads, traces, duration) so the cost shows up on a real node and we can decide later whether it is worth reading these headers from the multidownloader storage instead?
eca4aed to
f67517a
Compare
Bounded 8-way claim-trace prefetch with double canonical revalidation and atomic publication, plus atomic 100-block batched L1 persistence in a single SQL transaction. Opt-in and scoped to multidownloader-backed L1 syncers; legacy and L2 paths unchanged. Refs #1224
Address review feedback on the opt-in L1 trace-prefetch and batched persistence optimizations (refs #1224): - trace prefetch: revalidate the canonical inclusion of the window both before any trace RPC and again after all workers join, immediately before publishing, so a published snapshot is tied to the blocks the range was revalidated against; purge the snapshot on every failure path so a trace from an earlier generation is never served for a failed range; reject a nil-error-but-empty RPC root as a retryable error (never an inclusion mismatch) while keeping reverted roots and router/fallback wrappers valid. - batching: keep the per-block semantics shared with the single-block path, emit subscribers and post-commit logs only after the single commit in download order, and roll the whole batch back on failure. - claimsync: install the trace-prefetch enricher through the fail-closed InstallLogsEnricher hook, so a multidownloader-selected syncer aborts construction (ErrLogsEnricherUnsupported) instead of starting a syncer whose handlers can never resolve a prefetched trace. Refs #1224
The base branch turned the ClaimEvent and pre-Etrog ClaimEvent handlers into contextual appenders so their debug_traceTransaction runs with the active download context. Build both the legacy and the prefetch variants as contextual appenders, pass the context to the synchronous trace in extractCallDataFromLog, and update the trace prefetch tests. Refs #1224 Co-Authored-By: Claude Opus 5.5 <[email protected]>
The H5 optimizations were written for the leftover legacy EVMDriver. After #1994 the L1 bridge/claim syncers run on the native multidownloader driver, so install the claim-trace enricher and the atomic 100-block processor batch on that path and keep the legacy/L2 constructors one-block-at-a-time. Refs #1224
32ee7fe to
d86cc36
Compare
|
Rebased on #1994 eca4aed, which includes the Unsubscribe race fix and E2E synchronization updates. Current head d86cc36 also corrects documentation: canonical header reads use a bounded worker pool, not serial reads. Scoped race tests and multidownloader lint pass; focused E2E range assertions pass locally. Waiting for the live E2E checks on the new heads; no full-CI-green claim yet. |
39a7d7b
into
feat/l1-bridge-claim-multidownloader
🔄 Update — 20 November 2026
32ee7feb, stacked on the feat: detect reorgs through the multidownloader in l1 bridge and claim syncers #1994 head52ab74c8(the current base).The current stack references in this body are updated to this pair.
ccbcea69is now historical, not current. Every benchmark figurebelow was measured on a former head (
03f4c143/ac08f230/5453f356) and is notre-measured on
32ee7feb; the old figures and SHAs are preserved explicitly as former-headbenchmarks, not replaced.
-raceover the full 7 trees(
cmd, bridgesync, claimsync, multidownloader, sync, config, types) passed;lintpassed (0 issues). Binaries built with
-buildvcs=truereportvcsModified=false.🔄 Changes Summary
Performance follow-up to the shared L1 multidownloader integration (PR1). Two optimizations,
both opt-in and scoped to the multidownloader-backed L1 syncers; the legacy downloader and
every L2 syncer keep their exact existing one-block-at-a-time behaviour.
claimsync): the ordinary and pre-EtrogClaimEventrootdebug_traceTransaction(call-tracer) calls of a downloaded log range are prefetched with abounded pool of 8 concurrent requests (
ClaimTracePrefetchConcurrency = 8) instead of onesynchronous trace per log. Prefetch is installed after the coexistence hook that drops an
ordinary
ClaimEventsuperseded by aDetailedClaimEvent, so a suppressed log is never traced;the
DetailedClaimEventhandler never traces.needed request succeeded. Canonical inclusion of the window is revalidated twice — once before any
trace RPC and once after all workers join, immediately before publishing. A trace carries no block
hash, so the second read is what ties the snapshot to the revalidated blocks. This adds one extra
serial canonical header read per distinct block (2 × distinct blocks for the range). A changed or
absent header returns
ErrTransactionInclusionMismatch, so the whole range is refetched andnothing is published; every failure path purges the snapshot so a stale generation is never served.
sync/bridgesync/claimsync): the EVM driver may hand anordered batch of already-available blocks to the processor in a single all-or-nothing SQL
transaction, bounded to
sync.MaxBatchBlocks = 100. The driver only drains what the downloaderalready produced (never waits to fill a batch). Both batch and single-block paths share the same
inner per-block helper (block row, bridge/claim handlers, full proof and metadata fields, claim
proof-decoder fallback, empty blocks). Bridge subscribers and post-commit logs are emitted only
after the single commit, in original download order.
null) istreated as a retryable error, never an inclusion mismatch, and is not cached as an empty root;
the check is scoped to the frame type so a legitimate reverted root and a router/fallback wrapper
that nests the bridge call both stay valid.
docs/l1_shared_downloader.mdsection for the two optimizations and the bridgetree-cache rollback limitation; new
claimsync/trace_prefetch.go(+tests),claimsync/processor_batch_test.go,bridgesync/processor_batch_test.go,sync/driver_batch_test.go.🔄 Review follow-up — 16 Nov 2026
The PR head moved from
03f4c143toccbcea69to address the review threads. The newcommit is signed (SSH ED25519) and GitHub reports it
verified: valid:ccbcea69—fix(sync): address trace prefetch and batching review(ccbcea6)
Key changes:
InstallLogsEnricherreturns(prev, bool); Bridgeand Claim startup abort with
ErrLogsEnricherUnsupportedwhen a multidownloader-selectedsyncer cannot install the trace enricher, instead of starting handlers that can never
resolve a prefetched trace.
revalidateCanonicalreads one headerper distinct block per call through a bounded (≤ 8) worker pool, once before the trace
RPCs and once after all workers join; multidownloader headers are deliberately not
cached, so the cost stays
2Nper range and canonical correctness is preserved.ProcessBlockdelegates toprocessBlocksin bothBridge and Claim, so the halted check /
NewTx/ rollback / commit / notification pathlives in one place; the ad-hoc alias
100cap was replaced bysync.MaxBatchBlocks.Testing — local whole-repo CI at the new head
ccbcea69:MallocNanoZone=0 go test -count=1 -short -race -p 1 -covermode=atomic -timeout 15m ./...→ exit 0, 88 packages ok, 0 failures (sequential, isolated worktree).
ClaimTracePrefetchConcurrencyandsync.MaxBatchBlocksare fixed internalconstants, not operator-facing knobs.
📋 Config Updates
L1 syncers; legacy/L2 paths are unchanged.
🔌 API Updates
🔌 Bridge service API
notification ordering change (same per-block semantics, single commit). Not a breaking interface
change.
🔌 Proxy API
🔌 Others API
✅ Testing
🤖 Automatic — scoped over the 7 affected package trees
(
cmd, bridgesync, claimsync, multidownloader, sync, config, types), toolchain go1.27.3. This isnot a full-repo run and not E2E:
go test -count=1 -short -raceover those 7 trees → all ok (log:logs/h5-scoped-test.log).go vetandgofmtclean.golangci-lint run --timeout 5mover the same 7 trees → 0 issues(log:
logs/h5-safety-lint.log).make build→ exit 0 (log:logs/h5-build.log; binaries to git-ignoredh5/target/).rejection, snapshot purge on failure, atomic batch commit/rollback, notification ordering.
🖱️ Manual / E2E — safety scenario: the actual H5-safety body test exercised a
reorg-blocked RPC and a malformed-then-healthy retry (log:
logs/h5-safety-test.log).📊 Cardona historical synchronization benchmark — 13 November 2026 (from the actual
experiment reports, not a re-run):
Cardona L1 height range from the fixed
InitialBlock = 4,789,186up to the fixed per-syncercutoffs, so the comparison uses identical block ranges and identical cutoffs. H0 was run on
12 November 2026 and H5 on 13 November 2026; the comparison is documented on 13 November 2026.
The current network head on 13 November 2026 is not claimed, no new RPC/network run was
performed for this PR, and the current chain head is unknown.
l1infotreesync: initial 4,789,186 → target 12,119,079 ⇒12,119,079 − 4,789,186 + 1 = **7,329,894** blocks.l1bridgesyncandl1claimsync: initial 4,789,186 → target 12,119,148 ⇒12,119,148 − 4,789,186 + 1 = **7,329,963** blocks.scan and not the sum of the three syncers running in parallel.
fix, without parallel claim-trace prefetch or block-batched persistence.
Human-friendly times (rounded to the nearest second):
max of the three components, not their sum — so it coincides with the
l1claimsync(longest) row.
(1 − H5/H0) × 100; for the total/max row:(1 − 2506.17699265 / 4394.80259056) × 100 ≈ 42.974 % ≈ 42.97 %.same validation reports; identical for both variants because they replay the exact same frozen
range and must produce identical business data):
l1info_leaf)bridge)claim)leaves, bridge events, claims) inside the fixed comparison window, not the scanned block
heights (~7.33 million historical heights per syncer, listed above). They are also not
overshoot counts: rows synced past the cutoffs (the raw reports' larger
exp scan max) areexcluded — the hidden-tail proof confirms
srcOnly=0/expOnly=0beyond each cutoff.claim cutoff block 12,119,148.
same = total,different = 0,refOnly = 0,expOnly = 0,schemaMatch = truefor all three tables — sothere are no missing, extra, or field-level differences between H0 and H5.
reports/H5/20261113T073149Z-6adddcdb/validation.jsonand H0reports/H0/20261112T235557Z-recover-ac63c644/validation.json.reports/H0/20261112T235557Z-recover-ac63c644/case-result.json(frozen original, pre-hardening); H5
reports/H5/20261113T073149Z-6adddcdb/case-result.json.📊 Performance — Mainnet — 13–14 November 2026
Same fixed window and same historical dataset for every variant:
initialBlock = 16,896,718;frozen
LatestBlocktargets L1InfoTreeSync = 26,407,018 and Bridge/Claim = 26,407,089(inclusive spans: 9,510,301 and 9,510,372 blocks). The target snapshot was captured
2026-11-13T15:58:41Z on chainId 1 (latest
26,407,089, finalized26,407,018); theseresults are from 2026-11-14. This is a frozen snapshot, not the live chain head today.
All three variants replay the same public Katana source (identical for every variant; the
source URL/token is never recorded) — shared source-config digest
786fcb8660dee2c015b70aefcdf627a2a6cad6c505c987d9f79158de77d69043.ac08f230…, binary sha256256c753b…): shared L1 multidownloader withthe canonical-header idempotency fix, without prefetch/batching.
03f4c143…, binary sha256c9a9bb9f…),before the 16 Nov review follow-up (then-current head
ccbcea69…, now historical):H0 plus the
8-way claim-trace prefetch and the atomic 100-block SQL batch, including the post-canonical
hardening. Both H0/H5 binaries were built fresh from the then-current signed commits
(
-buildvcs=true,vcsModified=false, embeddedvcs.revision= signed HEAD).Reductions are computed from the unrounded times (
(1 − H5/H0) × 100).This H5 is the current post-canonical-hardening revision, unlike the older Cardona H5
numbers above, which were captured on the earlier pre-hardening build. The Cardona figures above
are unchanged by this run, and the two benchmarks are not merged.
L1InfoTreeSync already ran on the default multidownloader in both variants, so its 5.67 %
difference is run-to-run timing variation and must not be attributed to the prefetch/batching
feature; the stock Info run is never labelled "legacy".
The total row is the wall clock until all three syncers complete = the max (the
Claimrow), not the sum, i.e. the single-controller wall clock including the finalvalidation.
One trial per variant, same provider, the existing Docker container kept active, variable RPC
load and no control run: the differences are not statistically significant.
Compared data (identical in all three comparisons) — from the strict compare
src_count/loc_count/datasets_equalfields, 0 differences: 124,918 L1InfoTree leaves(
l1info_leaf), bridges 265,822 (bridge), claims 114,300 (claim).These are the business rows written inside the fixed window, not scanned block heights and
not overshoot rows.
Comparison strictness: schema match, integrity, claim proof-field formats (32 comma-separated
32-byte hashes), metadata/address fields, root history, event hashes, keys and all columns —
0 diff across
H0-vs-STOCK,H5-vs-STOCKandH5-vs-H0, allok=truewith every componentverdict
true. This is a data/format comparison: it does not claim cryptographic on-chainproof validation or internal RHT payload equality (tree-node tables are not compared;
validated=false).3-variant reference (same frozen inputs):
5453f356ac08f23003f4c143Stock used legacy bridge/claim direct-RPC downloaders and L1 info on the default
multidownloader; H5 total vs legacy stock: −70.11 %.
Evidence (read-only comparisons, no writes):
summary.json,reference-integrity.json,stock-report.json,h0-report.json,h5-report.json,H0-vs-STOCK.json,H5-vs-STOCK.json,H5-vs-H0.json. All three variants:status=completed,nodeExit code=0/SIGINT unexpected=false, harness/reader/fatal = none.Disk / retention (optional note): the mainnet data was produced fresh with no reused cache;
the only reclaim was an authorized Cardona-only cleanup (case
P1, 15 files, ~6.517 GiB)plus removal of the H0 intermediate DBs after all three comparisons — the stock + H5 DBs and all
reports were retained, and the source Docker/config was untouched.
🐞 Issues
Closes(closing isleft to PR1 / parent decision). Kept consistent with PR1's
Refs #1224so the two PRs do notclaim to close the issue twice.
🔗 Related PRs
feat/l1-bridge-claim-native-md-driver).📝 Notes
frozen
b9build captured before the post-canonical hardening and the latestdevelopadaptation. They are therefore not a performance guarantee for the current PR revision.
No new/magical timings are claimed.
two canonical header checks per distinct block in total (one before the trace, one after); proofs and the decoder are unchanged; SQL
batching is bounded to 100 blocks; post-commit
Publishcalls are retained; legacy/L2 fallbacksuntouched. The overhead of the post-canonical path is unmeasured.
bridge218283,claim63067,l1info_leaf229510; claims valid-proof full root;histories/headers compared; internal tree payload (
l1_info_rht,rht,rollup_exit_rht)excluded. These original H5 runs do not claim node-proof stock.
and runs the
PutLeafrollback callbacks, butdb.Tx.Rollbackreturns early onsql.ErrTxDoneand skips those callbacks, so the cache is not invalidated on that specific path (documented gap).
03f4c1437096c7f513e6c83d68c2b6de0389cf6d(branchperf/l1-multidownloader-h5,12 files — 7 modified, 5 new), plus the 16 Nov review follow-up
ccbcea69.Current head
32ee7feb; the previous headccbcea69d604469a562394b900c64be19380eeb1isnow historical. All signed (SSH ED25519).