Repository navigation
fix: integrate l1 bridge and claim multidownloader 1224 - #1952
joanestebanr wants to merge 10 commits into
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ac08f2304c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
joanestebanr
left a comment
There was a problem hiding this comment.
Automated review with Claude Code (/code-review, high effort). Findings are deduplicated but were not individually verified, so please double-check before acting on them. A manual security pass over the diff found no exploitable issues (SQL uses placeholders/sqlx.In; RPC transactions are validated against the log before caching; no new endpoints or secret logging).
The four most relevant findings are the first four inline comments (backfill stopping permanently, unbounded tx worker rescans, dh.mutex held across GetEthLogs, and full re-download when switching an existing node to the multidownloader).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0820f79d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // the cursor are never skipped. Rewinding the cursor (not the | ||
| // cached rows) is what preserves correctness: unchanged blocks | ||
| // stay a cache hit and only the reorged region is re-examined. | ||
| dh.rewindTransactionSchedulerFrom(reorgErr.OffendingBlockNumber) |
There was a problem hiding this comment.
Rewind transaction scans to the actual first affected block
When a reorg starts before the block where it is detected, this rewinds the transaction scheduler too little. ReorgProcessor.ProcessReorg searches backward for the common ancestor and deletes transactions starting at firstUnaffectedBlock + 1, which can be lower than OffendingBlockNumber. For example, if detection occurs at block 100 but blocks 90–100 are replaced, an already-advanced scheduler resumes at 100 and never scans the replacement logs in blocks 90–99. Those transactions remain missing from the background cache unless a foreground consumer requests them or the service restarts. Rewind using the actual rollback boundary, or reset the schedules after a successful reorg.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid, fixed in c15a55ee (branch rebased onto develop, new head c15a55ee).
ReorgProcessor.ProcessReorgnow returns the first reorged block (firstUnaffectedBlock + 1, i.e. the actual rollback boundary used byMoveReorgedBlocks) together with the error (multidownloader/types/reorg_processor.go,multidownloader/reorg_processor.go; mock regenerated withmake generate-mocks).- The run loop rewinds the transaction scheduler from
min(firstReorgedBlock, OffendingBlockNumber)instead ofOffendingBlockNumber, so replacement logs between the common ancestor and the detection block are re-scanned. - Tests:
TestEVMMultidownloader_Start/reorg_rewinds_the_transaction_scheduler_to_the_rollback_boundary(schedule at frontier 200, reorg detected at 100 with rollback boundary 90 -> frontier must be 90; it fails with the previous code, 100 != 90), andProcessReorgtests now assert the returned boundary (including the missing-block case where it is lower than the detection block).
e0820f7 to
c15a55e
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c15a55ee38
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| cancel() | ||
| // Wait for the whole generation (downloader + block processing) to | ||
| // exit cleanly before rolling back and restarting. | ||
| d.joinGeneration(downloaderDone, blockProcessingDone) |
There was a problem hiding this comment.
Make all downloader sends cancelable before joining generations
When the L2 GER downloader is blocked sending to a full download buffer during a reorg, this new join never completes. l2gersync/l2_ger_syncer.go also uses EVMDriver, but both downloaderSovereign.Download (evm_downloader_sovereign.go:138) and downloaderLegacy.Download (evm_downloader_legacy.go:199) still use unconditional downloadedCh <- *block sends. Canceling the generation stops the block-processing consumer without unblocking those sends, so the driver waits forever before rolling back or acknowledging the reorg, also stalling the shared L2 reorg detector. Make those implementations' sends cancellation-aware as well before requiring every downloader generation to be joined.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid, fixed in 8ceb2ab8. The generation join (joinGeneration / waiting on downloaderDone, also in SyncNextBlock) was introduced by this PR; on develop the driver only waits for the block-processing goroutine, so the hang was new here.
l2gersync/evm_downloader_sovereign.goandl2gersync/evm_downloader_legacy.go: the block send is nowselect { case <-ctx.Done(): close(downloadedCh); return; case downloadedCh <- *block: }, matchingsync.EVMDownloader.reportBlocks/reportEmptyBlock. These two andsync.EVMDownloaderare the onlysync.Downloaderimplementations.- Tests:
l2gersync/evm_downloader_cancel_test.go(TestDownloaderSovereign_Download_CancelWhileBlockedOnSend,TestDownloaderLegacy_Download_CancelWhileBlockedOnSend) block the downloader on an unread channel, cancel, and requireDownloadto return and close the channel. Both fail (timeout) with the previous unconditional sends.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8ceb2ab8a5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| cancel() | ||
| // Wait for the whole generation (downloader + block processing) to | ||
| // exit cleanly before rolling back and restarting. | ||
| d.joinGeneration(downloaderDone, blockProcessingDone) |
There was a problem hiding this comment.
Propagate generation cancellation into the sovereign GER appender
On a sovereign L2, if an injected GER is not yet indexed by the L1 info tree and an RPC stalls, this join can still prevent reorg processing indefinitely. The block sends are now cancelable, but downloaderSovereign.buildAppender creates context.Background() before calling logGERMissingFromL1InfoTree and isGERRemovedFromL2; those functions perform L1/L2 RPCs that cannot observe the canceled generation. Consequently, the downloader cannot exit, and the driver never reaches rollback or acknowledges the reorg to the shared L2 detector. Install a contextual appender for this downloader and pass the active generation context through its RPC calls, including the GlobalExitRootMap call that currently omits CallOpts.Context.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid, fixed in f4e25f79 (branch also rebased onto current develop 5f03b13). It is reachable only because of the generation join this PR adds, so I fixed every appender that runs on an EVMDriver downloader, not only the sovereign one:
l2gersync(sovereign): the GER appenders are now installed as contextual appenders (InstallContextualLogAppenders), thecontext.Background()in the insert handler is gone, and the download context reacheslogGERMissingFromL1InfoTree(L1 GER map read,IsUpToDate) andisGERRemovedFromL2(removal scan andGlobalExitRootMapwithCallOpts{Context: ctx}).l2gersync(legacy):findLatestInjectedGERpassesCallOpts{Context: ctx}toGlobalExitRootMap.bridgesync:eth_getTransactionByHashanddebug_traceTransactionuseCallContextwith the caller's context, and the contextualBridgeEventappender is installed on every path, not only the multidownloader one.claimsync: theClaimEventand pre-EtrogClaimEventhandlers (the ones that trace the claim tx) are contextual appenders and trace withCallContext.
Tests (each stalls an RPC until its context is canceled; they hang with the previous code):
TestDownloaderSovereign_Download_CancelWhileAppenderRPCStalls:Downloadreturns after cancel.TestBridgeEventAppender_CancelWhileRPCStalls.TestBuildClaimEventHandlerPreEtrog_CancelWhileTraceStalls.
8ceb2ab to
f4e25f7
Compare
3112484 to
cb4400c
Compare
Introduce a single shared L1 multidownloader instance consumed by both the
bridge and claim syncers, selected through the new per-syncer `Downloader`
option ("legacy" default, or "multidownloader"). Startup builds and validates
the shared downloader, starts the L1 consumers/backfill only after it is
initialized, and keeps L2 syncers on the legacy downloader.
Also adapt startup to the REST API rework: a failure to bind the public or
admin listener now returns an error instead of log.Fatalf, so the existing
deferred context cancellation runs on abort rather than calling os.Exit.
Includes multidownloader transaction storage (migration 0003), idempotency and
lifecycle regression tests, config validation, and docs.
Refs #1224
Address review feedback on the shared L1 multidownloader integration (refs #1224): - backfill: make the transaction requirement retry while coverage advances and cancellation-aware, keeping the log frontier at the historical InitialBlockNum floor so reorg inspection and old-record resolution below the committed checkpoint L stay covered; document why an L+1 shortcut is wrong. - multidownloader: track worker generations on incremental reorgs and failed pages, guard atomic coverage/storage reads with an RWMutex, factor a single copy helper, and drop unused production APIs while retaining their tests. - bridgesync: wire the contextual per-download bridge appender and fail closed (ErrContextualAppendersUnsupported / ErrLogsEnricherUnsupported) when the downloader does not expose the hooks, instead of silently falling back to a constructor-time appender or lazy resolution. - test helpers: fix the lifecycle so workers are cancelled and joined and the backend is closed before the test ends. Refs #1224
Make the two RequireAllHealthyOnStart unreachable-service matrix tests (6a/6b) inject a HealthChecker that reports every probed URL unhealthy, instead of relying on a closed httptest server. A resolved trusted-sequencer URL has its port rewritten to DefaultBridgeServicePort before probing, so when a service happens to be listening on that fixed port locally the default HTTP probe succeeds and the tests flake (Start returns nil and the entry is recorded healthy). Injecting the unhealthy response makes the health-matrix outcome deterministic and independent of real HTTP reachability; these are unit tests of the health gating, not of live unreachable HTTP. The bridge URL default-port assertion and the RequireAllHealthyOnStart behaviour are retained. Refs #1224
ProcessReorg searches backwards for the common ancestor and removes data from firstUnaffectedBlock+1, which can be lower than the block where the reorg was detected. The background transaction scheduler was rewound only to OffendingBlockNumber, so replacement logs between the common ancestor and the detection block were never scanned. ReorgProcessor.ProcessReorg now returns the first reorged block and the run loop rewinds the scheduler from min(firstReorgedBlock, offending). Refs #1224 Co-Authored-By: Claude Opus 5.5 <[email protected]>
The EVMDriver now joins the downloader goroutine of a generation before rolling back on a reorg or restarting. The legacy and sovereign L2 GER downloaders sent blocks with an unconditional channel send, so a downloader blocked on a full buffer after the consumer stopped would never return, stalling the driver (and the shared L2 reorg detector). Both sends now select on ctx.Done() and close the channel on cancellation, like sync.EVMDownloader already does. Refs #1224 Co-Authored-By: Claude Opus 5.5 <[email protected]>
The EVMDriver joins the downloader of a generation before rolling back on a reorg, so an appender blocked on an RPC that ignores the generation context stalls the reorg (and the shared reorg detector) indefinitely. - l2gersync: the sovereign GER appenders are installed as contextual appenders and the L1/L2 GER map reads, the removal scan and the L1 info tree status check use the download context (the context.Background() in the insert handler is gone). The legacy downloader passes ctx to the L2 GlobalExitRootMap read. - bridgesync: eth_getTransactionByHash and debug_traceTransaction use CallContext with the caller context, and the contextual BridgeEvent appender is installed on every path, not only the multidownloader one. - claimsync: the ClaimEvent and pre-Etrog ClaimEvent handlers, which trace the claim transaction, are contextual appenders and trace with CallContext. Refs #1224 Co-Authored-By: Claude Opus 5.5 <[email protected]>
ExtractTxnAddresses now fetches the transaction with CallContext bound to the caller context, so the activity RPC scanner tests must stub CallContext instead of Call. Refs #1224 Co-Authored-By: Claude Opus 5.5 <[email protected]>
cb4400c to
c90a8b8
Compare
…ULL from_address With syncFromInBridges=true the backfill selection requires from_address, but an asset event whose trace has no bridgeAsset call legitimately resolves a NULL from_address. The successful update fills txn_sender but leaves from_address NULL, so the record still matches the selection and backfillTable re-selects and re-resolves it forever (a no-progress busy loop). Track the records processed in a run whose from_address legitimately stays NULL in an in-memory completed set keyed by (block_num, block_pos) and exclude them from the selection. Only successfully persisted rows are marked; a row whose extraction failed is never marked, so it stays retryable. No schema change. Refs #1224
… commit A page already queued for retry must not advance the incremental transaction cursor. Below the cursor it is already accounted for; above the cursor it only appears after a retry-queue overflow rewound the cursor to the oldest queued page. Committing that overflow page advanced the cursor to pageEnd+1, skipping every gap between the rewind and the page (the queue that recorded them was just cleared), silently losing them. Only a brand new page may advance the cursor now. Add a regression over more than one maxNewPagesPerScan window with sparse failures that proves the rewound region is re-covered with no holes. Refs #1224
Use bounded keyset pagination over (block_num, block_pos) instead of a growing SQL exclusion list. Refs #1224
eca4aed to
f67517a
Compare
|
Fresh code review at the previous head found two real issues, fixed by f67517a and its parent 351776a: (1) successful backfill rows whose from_address legitimately remains NULL no longer cause a no-progress loop; selection now uses bounded keyset pagination over (block_num, block_pos), and failed rows remain retryable; (2) retry commits no longer advance the transaction scheduler frontier and skip unscheduled pages after overflow. Regression tests were shown to fail before the fixes and pass afterward; bridgesync/multidownloader -race and lint pass. |
🔄 Changes Summary
Initial integration scope for issue #1224: the L1 synchronizers can share a single
L1 multidownloader instead of each running its own dedicated EVM downloader.
their backend independently via the new per-syncer
Downloaderoption(
"legacy"|"multidownloader"). The zero value resolves tolegacy, so theexisting L1InfoTreeSync multidownloader / default downloader behaviour remains
unchanged. Startup-mode selection logging is not claimed to be byte-for-byte
identical.
downloader = "multidownloader"at startup — the shared instance is L1-scoped.shared multidownloader first, then the multidownloader is initialized and started,
and only then the L1 consumers and the txn-sender backfill start. Misuse
(multidownloader requested but not available, or an unsupported selection) fails
fast with a clear error.
write against the canonical
(block_number, block_hash)and only serves inclusionswhose hash still matches the canonical header, so a reorg cannot serve a stale
inclusion.
a failure now returns an error instead of
log.Fatalf, so the deferredcontext.CancelFuncruns on abort instead ofos.Exit. No REST routing change —endpoints and semantics are identical.
0003), idempotency andlifecycle/reorg regression tests, config validation, and
docs/l1_shared_downloader.md.🔄 Review follow-up — 16 Nov 2026
The PR head moved from
ac08f230toe0820f79to address the review threads.Both new commits are signed (SSH ED25519) and GitHub reports them
verified: valid:7990ca1f—fix(sync): address shared downloader review and test lifecycle(7990ca1)
e0820f79—test(bridgeservicefinder): isolate unreachable service fixtures(e0820f7)
Key changes:
backfillTableis now a cancellation-aware, bounded-backoffscheduler; records that are not covered yet are re-selected on the next pass instead of
stopping the backfill permanently, while partial success and the legacy transient path
stay bounded.
and stale-generation protection, so a tick no longer re-reads all completed history
(no new performance number claimed).
dh.mutexis anRWMutex; log reads take the read lock whilecoverage+storage atomicity is preserved under the write lock.
cfg.InitialBlockNumon purpose(reorg replay / historic backfill below
L); only the transaction range uses a separateL+1future floor. The migration re-download cost is not claimed to be eliminated.and keeps the legacy signature; no mutable ctx state is stored.
CopySyncerConfigis the singlenil-safe deep copy.
derive*/Storagemethods were removed and the tests movedto the canonical production primitives.
Testing — local whole-repo CI at the new head
e0820f79:MallocNanoZone=0 go test -count=1 -short -race -p 1 -covermode=atomic -timeout 15m ./...→ exit 0, 88 packages ok, 0 failures (sequential, isolated worktree).
Downloader(legacy|multidownloader) on theL1/L2 bridge and claim syncer config sections. Default
legacyis backwardscompatible.
log.Fatalf→ returned error on REST bind failure only changesthe failure path (process aborts with a non-zero exit and a wrapped error), not the
API surface.
📋 Config Updates
🧾 Diff/Config snippet:
MultiDownloader must be enabled when any L1 consumer selects it.
🔌 API Updates
🔌 Bridge service API
that a failure to bind the public/admin REST listener aborts startup with a returned
error (deferred cancellation runs) instead of
log.Fatalf/os.Exit. Not a breakinginterface change.
🔌 Proxy API
🔌 Others API
✅ Testing
(
cmd, bridgesync, claimsync, multidownloader, sync, config, types). This isnot a full-repo run and not E2E (
GOWORK= externalintegration.go.work,toolchain go1.27.3):
go test -count=1 -short -raceover those 7 trees → all ok(log:
logs/integration-scoped-test.log).golangci-lint run --timeout 5mover the same 7 trees → 0 issues(log:
logs/pr1-lint.log).make build→ exit 0 (log:logs/pr1-build.log; binaries written to thegit-ignored
integration/target/, nothing committed).*_md_integration_test.go), lazyconstruction, constructor validation, storage transaction idempotency/regression,
and cmd startup tests.
separately and is not claimed here.
📊 Performance — legacy vs integrated L1 multidownloader (2026-11-13)
Same fixed window and same historical dataset on both sides:
initialBlock = 4,789,186;fixed
LatestBlocktargets L1InfoTreeSync = 12,119,079 and Bridge/Claim =12,119,148 (inclusive spans: 7,329,894 and 7,329,963 blocks). Stock and
integrated runs share the same source-config digest (
04e82d69…c924, chainId 11155111)and produced identical business data inside the window — L1 info leaves 229,510,
bridges 218,283, claims 63,067 (post-target rows are extra and excluded). One
trial per variant, same Docker/RPC host; real RPC load varies run to run.
v0.12.0-rc4multidownloader in both runs, so its +4.17 % is run-to-run timing variation,
not an integration effect.
header/log fetching, nothing shared); L1 info stayed on MD. The integrated H0 build
runs one shared L1 multidownloader for bridge, claim and transaction inclusion,
with H0 idempotency — and without the H5 trace-prefetch/batching work (separate,
not in this PR).
Evidence (read-only, no writes): the legacy stock DBs re-checked read-only reproduce the
exact window counts (229,510 / 218,283 / 63,067,
block ≤ fixed target); a fullstock-vs-reference comparison over the frozen reference run (
…c10cde59) passed everybusiness/proof/root/history/header check —
bridgeStrictEqual,claimsAllMatch,claimFieldValidityOk,tablesAllEqual,schemaAllMatch,rootHistoryMatch,headerEventOk,expectFromAddressNulls=0— and the H0 integrated validation is afresh PASS. One trial per variant: absolute seconds are load-sensitive and these Cardona
figures are not a guarantee for the current PR head (latest develop + REST adaptation),
which was not re-benchmarked end-to-end on Cardona (the mainnet run below is a separate
measurement and does use the current signed revision). Comparison made 2026-11-13 against the
fixed 2026-11-12 targets, not against the live tip.
📊 Performance — Mainnet — 13–14 November 2026
Same fixed window and same historical dataset for every variant:
initialBlock = 16,896,718;frozen
LatestBlocktargets L1InfoTreeSync = 26,407,018 and Bridge/Claim = 26,407,089(inclusive spans: 9,510,301 and 9,510,372 blocks). The target snapshot was captured
2026-11-13T15:58:41Z on chainId 1 (latest
26,407,089, finalized26,407,018); theseresults are from 2026-11-14. This is a frozen snapshot, not the live chain head today.
All three variants replay the same public Katana source (identical for every variant; the
source URL/token is never recorded) — shared source-config digest
786fcb8660dee2c015b70aefcdf627a2a6cad6c505c987d9f79158de77d69043.v0.12.0-rc4(commit5453f356…, binary sha2565b072065…): legacybridge/claim independent direct-RPC downloaders, L1 info on the default multidownloader.
ac08f230…, binary sha256256c753b…),before the 16 Nov review follow-up (current head
e0820f79…): one shared L1multidownloader for bridge, claim and transaction inclusion, with the canonical-header
idempotency fix and without the H5 trace-prefetch/batching work.
-buildvcs=trueandvcsModified=false(embeddedvcs.revisionmatches the signed HEAD);the stock binary is the immutable signed
v0.12.0-rc4artifact (hard-linked, sha256 unchanged).v0.12.0-rc4"Change in elapsed time" =
(H0 − stock) / stock × 100, computed from the unrounded times;negative = faster.
L1InfoTreeSync is not a legacy-vs-MD comparison: it already ran on the default
multidownloader in all three variants, so its +18.70 % is run-to-run timing variation
and must not be attributed to the integration feature — the observed timing includes the
shared workload and provider conditions, not only the feature. The stock Info run is therefore
never labelled "legacy".
The wall-clock row is the max of the three syncers running in parallel (here the
Claimrow), i.e. the single-controller wall clock until all three complete including the final
validation — not the sum of the three times.
One trial per variant, same provider, the existing Docker container kept active, variable RPC
load and no control run: the differences are not statistically significant and are not a
guarantee for other hosts/conditions.
Compared data (identical in all three comparisons) — from the strict compare
src_count/loc_count/datasets_equalfields, 0 differences: 124,918 L1InfoTree leaves(
l1info_leaf), bridges 265,822 (bridge), claims 114,300 (claim).These are the business rows written inside the fixed window, not scanned block heights and
not overshoot rows.
Comparison strictness: schema match, integrity, claim proof-field formats (32 comma-separated
32-byte hashes), metadata/address fields, root history, event hashes, keys and all columns —
0 diff across
H0-vs-STOCK,H5-vs-STOCKandH5-vs-H0, allok=truewith every componentverdict
true. This is a data/format comparison: it does not claim cryptographic on-chainproof validation or internal RHT payload equality (tree-node tables are not compared;
validated=false).3-variant reference (same frozen inputs; H5 = H0 + prefetch/batching):
5453f356ac08f23003f4c143H5 total vs legacy stock: −70.11 %. (The H5 numbers are the subject of the successor PR2.)
These mainnet percentages are kept separate and are not merged with the Cardona
percentages above.
Evidence (read-only comparisons, no writes):
summary.json,reference-integrity.json,stock-report.json,h0-report.json,h5-report.json,H0-vs-STOCK.json,H5-vs-STOCK.json,H5-vs-H0.json. All three variants:status=completed,nodeExit code=0/SIGINT unexpected=false, harness/reader/fatal = none.Disk / retention (optional note): the mainnet data was produced fresh with no reused cache;
the only reclaim was an authorized Cardona-only cleanup (case
P1, 15 files, ~6.517 GiB)plus removal of the H0 intermediate DBs after all three comparisons — the stock + H5 DBs and all
reports were retained, and the source Docker/config was untouched.
🐞 Issues
🔗 Related PRs
fix: optimize l1 sync with parallel traces and batching 1224 #1953. Stack base = this PR's current head
e0820f79d17c2545f072e8890fad21e5ff71806b.📝 Notes
ac08f2304c2ec70f239ce4551438a67d80c7c1c1(
feat(sync): share multidownloader between l1 bridge and claim syncers),51 files changed, +11788 / −269, plus the 16 Nov review follow-up
7990ca1fande0820f79. Current heade0820f79d17c2545f072e8890fad21e5ff71806b. All signed(SSH ED25519).