Repository navigation
int_proxy: BackendFrontier reads a proxy backend's inputs as of a frontier - #1023
Merged
frankmcsherry merged 1 commit intoOct 24, 2026
Merged
Conversation
…ntier A trace's accumulations are correct only at or beyond its compaction frontier. A reader that starts after the trace has compacted must read every time advanced to a frontier `since`, as `TraceFrontier` does for a cursor. The proxy tactics hand batches to a backend, which reads times as stored. `int_proxy::frontier::BackendFrontier<B, T>` wraps any join or reduce backend over the same batch types and reads what it draws as of `since: Antichain<T>`, advancing with `Lattice::advance_by` and consolidating what then coincides. The tactics, traits and instance types are unchanged. The argument uses only the defining property of `advance_by`, so it holds in any lattice, distributive or not: - A unit whose bound is beyond `since` is left alone, and is exact: a join's output times are already beyond `since`, and a reduce evaluates only at times beyond its lower bound. That is the steady state; only the unit that reads a trace's history pays. - That unit agrees with the as-of reading at every time beyond `since`, and is identical when `since` has one element, when the inner backend presents stored times, or when `since` is beyond the unit's bound. - `since` must be beyond every frontier the trace compacted by. Reading as of the meet of the compaction frontier is not that, and is unsafe in a non-distributive lattice. The tests read a history through the wrapper against the same tactic over the rows advanced by hand. The N5 tests, over the pentagon lattice, run the join and the reduce through the wrapper for every frontier of one or two elements, every bound, and every placement of the records. Co-Authored-By: Claude Fable 5.1 <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #981.
A trace's accumulations are correct only at or beyond its compaction frontier. A reader that starts after the trace has compacted must read every time advanced to a frontier
since, asTraceFrontierdoes for a cursor. The proxy tactics hand batches to a backend, which reads times as stored.int_proxy::frontier::BackendFrontier<B, T>wraps any proxy join or reduce backend over the same batch types. It reads what the backend draws as ofsince: Antichain<T>, advancing withLattice::advance_byand consolidating what then coincides; in a join, a key that nets away on one side also leaves the other. The tactics, traits and instance types are unchanged.The argument uses only the defining property of
advance_by(fort'beyondsince,t ≤ t'exactly whenadvance(t) ≤ t'), so it holds in any lattice, distributive or not:sinceare left alone, and that is exact. A join's output times are already beyondsince, and a reduce evaluates only at times beyond its lower bound. This is the steady state, where the wrapper costs one antichain comparison per unit. Only the unit that reads a trace's history pays.since. It is identical whensincehas one element, when the inner backend presents stored times, or whensinceis beyond the unit's bound.sincemust be beyond every frontier the trace compacted by. Reading as of the meet of a multi-element compaction frontier is not that, and is unsafe in a non-distributive lattice.Tests
tests/int_proxy.rschecks the wrapper in two ways:u64times: a history read through the wrapper, against the same tactic over the rows advanced by hand.sinceotherwise. No case differed exactly. A brute-force search over 9.2 million configurations of small lattices found no exact difference either.cargo test -p differential-dataflowpasses. DDIR's use of it is the next PR.🤖 Generated with Claude Code
https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o