Repository navigation
test: e2e: proxy reorg acceptance scenarios on anvil (#1972) - #1991
joanestebanr merged 9 commits into
Conversation
Stop/start/copy helpers for a compose service, a 5s ActivityPollInterval for both aggkit-proxy instances and a host alias for aggkit-proxy-002, used by the resilience scenarios. Co-Authored-By: Claude Fable 5.2 <[email protected]>
Co-Authored-By: Claude Fable 5.2 <[email protected]>
…1972) Co-Authored-By: Claude Fable 5.2 <[email protected]>
…1972) Co-Authored-By: Claude Fable 5.2 <[email protected]>
Co-Authored-By: Claude Fable 5.2 <[email protected]>
Co-Authored-By: Claude Fable 5.2 <[email protected]>
|
@codex review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41caac111d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for { | ||
| select { | ||
| case message, open := <-ws.messages: | ||
| require.True(t, open, "the WebSocket was closed (%v) while waiting for %s; last message: %s", ws.closeErr, what, last) |
There was a problem hiding this comment.
Read the WebSocket close error only after observing closure
When D2 receives its final status and the server closes the WebSocket, this assertion reads ws.closeErr even when open is true, because Go evaluates all arguments before calling require.True. The reader goroutine can concurrently assign ws.closeErr while processing the close frame. Receiving a buffered status message does not synchronize with that later assignment, so the normal completion path contains a data race and can fail under go test -race. Check !open before reading the error, or synchronize access to it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 284689e: waitFor now reads ws.closeErr only inside the !open branch, i.e. after the reader goroutine has closed the channel. The reader also stops on a done channel instead of blocking on a full buffer.
| // then the proxy: the same set, all final, with no duplicate and no orphan entry | ||
| final, err := waitForActivity(ctx, proxyTrackerBaseURL, sender.addr, "exactly the bridges on chain, all final", | ||
| proxyReorgConvergeTimeout, func(a *proxyActivityResponse) bool { |
There was a problem hiding this comment.
Register activity before running the reorg burst
In F4, this is the first activity request for the freshly generated sender, and it happens only after every reorg has completed, all surviving bridges are final, and aggkit serves the corrected set. The activity engine refreshes registered addresses only, so the proxy never observes this sender's pre-reorg state or processes the burst against cached entries. Consequently, the scenario passes even if proxy reorg invalidation is completely broken; it tests only an initial scan of settled history. Register the address before the burst and ensure the proxy has cached an entry that the subsequent reorgs remove or move.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in a09c854: F4 now requests the sender's activity (and requires it empty) before pausing the chain, so the address is registered and refreshed in the background while the burst runs. The last reorg also gets its own baseline, read once aggkit has caught up with the burst.
… store is used (#1972) F4 registers the sender's activity before the burst and reads a fresh reorg baseline for the last reorg. F2 also checks the persisted tracking and that the restarted proxy answers from its store. Each test holds the aggsender (see the next commit). Co-Authored-By: Claude Fable 5.2 <[email protected]>
…view findings (#1972) - B2/C1: the aggoracle does not inject again a GER whose injection a reorg removed (#1992), so the scenarios recover the bridge with a later GER; every pending injection is mined and dropped together. - holdAggsender: aggkit-001's aggsender certifies non-final L2 data, and a certificate settled with a bridge that is then reorged out wedges it for good. The tests keep it from certifying their blocks. - Review findings: B3 waits for both bridges before comparing revisions, the 'after the reorg' instant is taken once aggkit resolved it, the WebSocket reader cannot block or race on its close error, nil guard in the item summary. Co-Authored-By: Claude Fable 5.2 <[email protected]>
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…1972) A changed config is rendered in a temporary directory and the service is recreated with a compose variable that points at it (AGGKIT_001_CONFIG, AGGKIT_PROXY_002_CONFIG), so a killed run leaves the tree clean. Also from the second review: captureGERInjection reads the injections from the mined block, F4 makes the proxy hold entries the burst turns stale and waits for the event of the last reorg itself, and F2 checks the persisted address is marked refreshed. Co-Authored-By: Claude Fable 5.2 <[email protected]>
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1d57fd2
into
feat/445-proxy-reorg-activity-tracker
…, #1969, #1970) (#1986) > This PR brings the content of the integration branch `feat/445-proxy-reorg-wave1` to `develop`: the squash commits of #1981 and #1982, cherry-picked unchanged (no conflicts). Both were already reviewed and approved in those PRs; the finality resolver (#1979) and the API contract (#1980) they build on are already in `develop`. > > On top of them, a9c639c fixes a Codex P2 found on this PR: `reverifyClaim` now also compares `BlockTimestamp`, so a claim re-included at the same height in another block takes the new timestamp (test `TestReverifyStepsClaimReincludedAtSameHeight`). This fix is not in #1982 or in the integration branch. > > Three stacked PRs were then squash-merged into this branch: #1991 (1d57fd2, e2e acceptance scenarios), #1988 (788be65, L2->L1 leaf finality) and #1990 (5859d6f, ETag and `304` for the activity revision). ## 🔄 Changes Summary - **Activity scan (#1981, #1968, #1969)**: `GET /tracker/v1/activity/from/{from_address}` - The scan of the bridge service stops at the first bridge that is known **and final**. Non-final bridges are re-read on every refresh: they are corrected in place if they moved to another tx, block or timestamp, and removed if they are gone. - An entry is settled only when the claim is confirmed and both the bridge block (origin) and the claim block (destination) are final. Until then it keeps being refreshed and can go back from `claimed` to `readyToClaim` or `pending`. - No bridge-service error, including the `503` returned while it resolves a reorg, removes anything. - The activity `revision` is kept in the in-memory cache and in SQLite, with the same value for the same data and across restarts. - New metrics: `bridgetracker_activity_reorgs_detected_total`, `..._entries_corrected_total` and `..._entries_removed_total`. - **Tracker (#1982, #1970)**: a tracking is final only when its `Claimed` step is backed by a claim in a final block. Until then, on every engine tick: - the origin block hash is re-checked, and the bridge tx is resolved again if it changed; - the claim is re-checked with `isClaimed()`, and the tracking goes back to `WaitingClaim` if the claim is gone; - at `readyToClaim`, the GER injection and the L1 info leaf are re-checked. - Backward moves are persisted and pushed to WebSocket subscribers. A failing check (an RPC error or a `503`) never moves a tracking backwards. - **L2->L1 leaf finality ([#1988](#1988), #1987)**: at `readyToClaim` on L2->L1, once `l1-info-tree-index` returns the stored index, the tracker asks the same bridge-service instance for `injected-l1-info-leaf?network_id=0` and marks the `WaitingL1InfoLeafAvailable` step final when `block_finalized` is `true`. After that, the leaf is not re-checked. An omitted `block_finalized`, a `404` or an error keep the previous behaviour (re-checked every tick). L1->L2 and L2->L2 are unchanged. - **ETag and `304` on the activity endpoint ([#1990](#1990), last item of #1971)**: every `200` of `GET /tracker/v1/activity/from/{from_address}` carries the activity `revision` as a strong `ETag`, plus `Cache-Control: no-store`. A request whose `If-None-Match` matches gets `304 Not Modified` with no body. `400`, `500` and `503` keep their status and carry no `ETag`. `CurrentAPIRevision` goes from 7 to 8. CORS is only documented: the proxy default `[REST.CORS].AllowedHeaders = ["*"]` already allows `If-None-Match`, and an explicit list must add it. It also fixes the `API.md` rule on when a claim changes the `revision`: only when its tx changes, not when the same tx moves to another block. - **E2E reorg acceptance tests ([#1991](#1991), #1972)**: scenarios A1-A6, B1-B4, C1, D1-D2, E1-E4 and F1-F4 cause real reorgs on the anvil chains of `anvil-2chains` and check the activity endpoint, the tracker endpoint and its WebSocket before and after. Each scenario first asserts that aggkit reported the reorg (`last-reorg-event`). Test code and test env only (`ActivityPollInterval = "5s"` and optional config overrides in `test/e2e/envs/anvil-2chains`). ##⚠️ Breaking Changes - 🔌 **API/CLI**: no names or values are removed, but behaviour changes: - `claimed` is no longer terminal: it can go back to `readyToClaim` or `pending` after a reorg. The terminal state is `claimed_finalized`. - `tracking_status` only reads `finished`, and the WebSocket only closes, once the claim is final. With the claim done but not yet final, it stays `running`. - `creation_timestamp` of an activity entry can change if a reorg moves the bridge to another block. ## 📋 Config Updates - None. The SQLite migration `bridgetracker0005` adds `activity_bridge.revision_hash`. - For a browser to send `If-None-Match` cross-origin, `[REST.CORS].AllowedHeaders` must allow it: the default `["*"]` does, an explicit list must add it. ## 🔌 API Updates ### 🔌 Bridge service API - No changes. ### 🔌 Proxy API - 🔌 **AProxy API Update**: the `api_revision` 7 fields introduced by #1980 are populated now: `finalized`, `claimed_finalized` and `revision` in the activity endpoint, and `claimed_finalized` and `bridge_status.block_finalized` in the tracker endpoints and WebSocket. No endpoints are added or removed. Clients must accept `claimed_finalized` before this is deployed (agglayer/sdk#47, agglayer/agglayer-dev-ui#64). - 🔌 **AProxy API Update (#1990)**: `api_revision` 8, additive. The activity endpoint accepts the optional `If-None-Match` request header, answers `304 Not Modified` (no body) when it matches, and sends `ETag: "<revision>"` and `Cache-Control: no-store` on `200` and `304`. The body does not change. ## ✅ Testing - 🤖 **Automatic**: unit tests from #1981 and #1982. - Activity, run against both stores: bridge removed, bridge re-included in another block, claim reorged out or moved, `503` keeping the state, final entries not re-read, restart in the middle of a non-final state. - Tracker: every backward transition on L1->L2, L2->L1 and L2->L2, origin reorged out and re-included, `503` and transient errors leaving the state unchanged, final trackings making no calls, WebSocket backward updates. - Run locally on this branch: `go build ./...`, `go vet`, `go test -race ./bridgetracker/...`, `go test ./proxy/...`, `make lint`. - 🤖 **Unit tests from #1988**: `TestReverifyStepsL2ToL1LeafFinality`, `TestReverifyStepsLeafFinalityOnlyWithoutInjectionStep`, `TestGERSourceL1InfoLeafFinalized` and `TestEngineL2ToL1FinalLeafIsNotCheckedAgain`. - 🤖 **Unit tests from #1990**: `bridgetracker/api/activity_etag_test.go` (`If-None-Match` variants, the same `ETag` across filters and pages, errors never turned into `304`, registry without a revision, the `304` through the real router). - 🤖 **E2E against real reorgs (#1991)**: the new CI groups `proxy-reorg-activity`, `proxy-reorg-claims`, `proxy-reorg-tracker-revision` and `proxy-reorg-resilience` in `test-go-e2e.yml` run in this PR's CI. ## 🐞 Issues - Closes #1968 - Closes #1969 - Closes #1970 - Closes #1972 - Closes #1987 - Closes #1971 - Part of agglayer/pm#445 ## 🔗 Related PRs - #1981 and #1982: the original PRs, reviewed and approved against the integration branch. - #1979 (finality resolver) and #1980 (API contract): already merged into `develop`. - #1991 (e2e acceptance scenarios), #1988 (L2->L1 leaf finality) and #1990 (ETag and `304`): stacked PRs, reviewed there and squash-merged into this branch. - agglayer/sdk#47 and agglayer/agglayer-dev-ui#64: clients accepting `claimed_finalized`. ## 📝 Notes These design decisions are still open, recorded as implemented: - (a) `tracking_status` only reads `finished` once the claim is final. - (b) An unclaimed bridge in a final block reads `finalized=true` and goes back to `false` when it is claimed, until the claim is final. - (c) An origin tx that is reorged out and not re-included within `UnresolvedTimeout` (30 s by default) ends in a terminal error. - (d) On L2->L1 the L1 info leaf used to be polled on every tick while the bridge is at `readyToClaim`. #1988 now stops once the leaf is reported final. Its own open point is that the leaf's finality is only asked when the index matches the stored one; if the leaf moved after an L1 reorg, it keeps being re-checked every tick. Issues in aggkit found by the e2e tests of #1991 (none is a proxy problem, none is fixed here): - #1992: after an L2 reorg removes a GER injection, the aggoracle does not inject that GER again, so the bridges that depend on it stay unclaimable until a later GER is injected. The tests work around it by forcing a later GER on L1. - The aggsender certifies non-final L2 blocks when `BridgeL2Sync` is at `LatestBlock`. If a certificate carrying a bridge of a non-final block settles and the bridge is then reorged out, the aggsender cannot build another certificate and the network's settlement stops. The tests hold the aggsender during each scenario. No issue opened yet. Other notes: - With several bridge-service instances behind a load balancer, a lagging instance can make non-final activity entries be removed and re-added until they converge. - The generated swagger carries unrelated drift (the `Duration` enum and `block_finalized` in the bridge-service payloads), as noted in #1982. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
🔄 Changes Summary
aggkit-proxy(test: e2e: proxy reorg acceptance scenarios on anvil #1972, wave 3 of agglayer/pm#445): they cause real reorgs on the anvil chains ofanvil-2chainsand check what the activity endpoint, the tracker endpoint and its WebSocket report before and after. Test code and test env only; no production code changes.GET /bridge/v1/last-reorg-event, asked directly) and only then what the proxy answers, so a failure tells a syncer problem from a proxy one. Mining is controlled by the tests and every wait polls a condition with a timeout.test/e2e/:proxy_reorg_common_test.go(helpers),proxy_reorg_activity_test.go(A1-A6),proxy_reorg_claims_test.go(B1-B4, C1),proxy_reorg_tracker_test.go(D1, D2),proxy_reorg_revision_test.go(E1-E4),proxy_reorg_resilience_test.go(F1-F4).test-go-e2e.yml, each on its own stack:proxy-reorg-activity,proxy-reorg-claims,proxy-reorg-tracker-revision,proxy-reorg-resilience.Scenario status
Local = run on my machine against
aggkit:localbuilt from this branch, onemake test-e2e TEST_RUN=...per CI group, each ending with the post-test bridge check ofTestMainpassing.TestProxyReorgActivityTestProxyReorgActivitydeposit_countTestProxyReorgActivityTestProxyReorgActivityTestProxyReorgActivityTestProxyReorgActivityTestProxyReorgClaimsTestProxyReorgClaimsTestProxyReorgClaimsTestProxyReorgClaimsTestProxyReorgClaimsTestProxyReorgTrackerClaimedgoes back, WebSocket notifiedTestProxyReorgTrackerTestProxyReorgRevisionTestProxyReorgRevisionTestProxyReorgRevisionTestProxyReorgRevision503during the reorgTestProxyReorgResilienceDBPathrestarted, reorg while it is downTestProxyReorgResilienceblock_finalizedTestProxyReorgResilienceTestProxyReorgResilience📋 Config Updates
test/e2e/envs/anvil-2chains):ActivityPollInterval = "5s"in both aggkit-proxy configs (the default is 30s), so the tests wait one short refresh.extra_hosts: host.docker.internal:host-gatewayonaggkit-proxy-002, so the resilience test can put a reverse proxy between it and the bridge service.aggkit-001andaggkit-proxy-002take an optional compose variable (AGGKIT_001_CONFIG,AGGKIT_PROXY_002_CONFIG, defaulting to the tracked files). The tests that need another config render it in a temporary directory and recreate the service with the variable set (Env.RecreateService): they never write to a tracked file, so a killed run leaves the tree clean (checked by interrupting a run with SIGINT).🔌 API Updates
✅ Testing
go build ./...,go vet ./test/e2e/...andmake lintpass. Locally each test takes between 1 and 2.5 minutes, plus the stack start and the post-test check.🐞 Issues
🔗 Related PRs
anvil_reorg, never a rollback to a shorter chain).📝 Notes
NOTEin the code; they test the current behaviour and decide nothing):tracking_statusreadsfinishedonly with a final claim (D2).finalized=true, and a claimed one is not final until its claim is (A6, B1).UnresolvedTimeout, 30 s (D1, F2).aggkit-002syncs network 2 atFinalizedBlock, so it never tracks the reorged blocks. The test asserts that instead (the bridge service never served the bridge andlast-reorg-eventdid not change) and that the proxy forgot the bridge through its RPC window.503or to omitblock_finalizedon demand, so the test runs a reverse proxy on the host betweenaggkit-proxy-002and the bridge service ofaggkit-001. It changes a status code (F1) or removes one JSON field (F3); everything else is aggkit's own answer.DBPathonaggkit-proxy-002and reads the SQLite file out of the stopped container to check the stale entry and tracking really were persisted before the reorg.revision(the hash covers the claim's tx hash, not its block). That matches the "What goes into the hash" table ofdocs/bridgetracker/API.md, while the rule above the table reads as if a claim "re-included in another block" changed it. The test asserts the table.pending, which is right. B2 and C1 work around it: they recover the bridge by forcing a later GER on L1 (forceNewGER).aggkit-001builds certificates from the same non-final L2 data its bridge service serves (BridgeL2SyncatLatestBlock). If a certificate that carries a bridge of a non-final block settles and the bridge is then reorged out, the aggsender cannot build another certificate (failed to get local exit root by hash ...: not found) and the network's settlement stops for good; the post-test bridge check ofTestMainthen fails. Each test here holds the aggsender for its duration (holdAggsender:aggkit-001runs with[AggSender] MaxL2BlockNumberat the current head, and goes back to its own config at the end). Whether aggkit should guard against this combination is not decided here.L2GERSyncatLatestBlock) does not process an injection in the head block until another block is mined on top, soinjected-l1-info-leafonly serves it then. The tests work with it.revisiondo not change. B3 does assert the settled entry is not rechecked (last_updated_timestampstops).claimL1ToL2is split so the claim can be sent without waiting for it to be mined,fetchTrackingDatagets a variant with the proxy URL, and the tracker mirror type gainsblock_finalized.🤖 Generated with Claude Code