Skip to content

dogs3: a proxy half-join tactic over a backend, built on int_proxy's public surface - #1022

Merged
frankmcsherry merged 2 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:phj-dogs3-proxy-half-join
Oct 24, 2026
Merged

frankmcsherry merged 2 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:phj-dogs3-proxy-half-join

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

Replaces #1019 and #1020: the proxy half-join tactic now lives in dogs3, next to the operator it drives, and is built only from differential's public int_proxy surface, as any outside implementor would build a tactic over proxies.

The half join had no seam between its time logic and its data: dogs3's HalfJoinTactic reads the arrangement's batches itself, so nothing can wrap it, for example to read an arrangement as of a frontier. ProxyHalfJoinTactic implements HalfJoinTactic over a ProxyHalfJoinBackend, in the shape of differential's proxy join and reduce.

  • The tactic owns the comparison of times, strictness, cutting work into blocks of keys, the diff product, and the rule for output times: the update's own time, carrying its data time joined with the arranged time.
  • The backend owns staging the released updates, presenting each block of both sides by key, comparing real keys where the key proxy is a hash, and building output containers.

Notes:

  • The comparison is Ord, which Timestamp promises is a linear extension of the partial order. Nothing needs TotalOrder, so product times work.
  • Arranged times are presented as stored. Advancing them by a frontier can move them in the total order relative to stream times they are incomparable with. For example, with lower = {(1, 0)}, (0, 5) sorts before (1, 0), but advances to (1, 5), which sorts after it. So the backend is given no lower bound to advance by.
  • The arranged side has the join's contract, sorted and consolidated by ((key, val), time). The stream side is in key order only.
  • Matches are pairs of positions into the block's two presented sides, which cross is handed back, so a backend can evaluate the time rule in its own layout.
  • BlobList is public. Every HalfJoinTactic needs a batcher that agrees with it on when an update is released, and BlobList is the one for Vec containers.

What it needed from differential

One change: int_proxy::debug_assert_sorted_bridge is now pub. It checks the bridge contract every proxy backend owes its tactic, so an outside tactic should run the same check rather than a copy that can drift. Everything else was already public:

  • ProxyBridge and KeyPosition;
  • Semigroup, Multiply, Lattice, consolidate_updates and Hashable;
  • ChunkBatch and VecChunk.

Two private helpers weren't worth exposing, so dogs3 has its own: a batch type alias over public types, and an eight-line search.

Tests

tests/proxy_half_join.rs runs the proxy half join over VecHalfJoinBackend in two ways:

  • against the cursor half join in one dataflow, on random inputs, strict and not, at one and two workers;
  • against the definition over Product<u64, u64> times, with several block sizes and colliding hashes.

The dogs3 tests and cargo test -p differential-dataflow pass, and clippy reports nothing in the files touched.

DDIR's port to this tactic is #1021, which builds on this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o

frankmcsherry and others added 2 commits October 24, 2026 00:02
`debug_assert_sorted_bridge` checks the contract every proxy backend owes
its tactic. A tactic built outside differential over the same bridges,
such as dogs3's proxy half join, needs the same check, so it is `pub`
rather than `pub(crate)`. The one change an outside implementor of a
tactic over proxies needed from this module.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
…public surface

The half join had no seam between its time logic and its data: dogs3's
HalfJoinTactic reads the arrangement's batches itself. ProxyHalfJoinTactic
implements HalfJoinTactic and owns the total-order comparison, strictness,
the output time rule (the update's own time, and its data time joined with
the arranged time) and blocking; ProxyHalfJoinBackend owns key matching,
value proxies and output containers, like differential's join and reduce
backends. Matches are pairs of positions into the block's two presented
sides, which `cross` is handed back, so a backend evaluates the time rule
in its own layout.

It lives here, not in differential's int_proxy, and is built from that
module's public surface (ProxyBridge, KeyPosition, the bridge check) the
way any outside implementor would build a tactic over proxies.

Arranged times are presented as stored: advancing them by a frontier moves
them in the total order relative to incomparable stream times, so the
backend is given no lower bound to advance by. The arranged side is a
bridge with the join's contract, sorted and consolidated by
((key, val), time); the stream side is in key order only.

The generic release batcher BlobList is public: every HalfJoinTactic needs
a batcher that agrees with it on when an update is released, and this is
the one for Vec containers. VecHalfJoinBackend is the reference backend.
The tests run the proxy half join against the cursor half join in one
dataflow on random inputs, strict and not, at one and two workers, and
against the definition over product times, with several block sizes and
colliding hashes.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant