Skip to content

DDIR: every import is read as of the time its reader starts - #1024

Merged
frankmcsherry merged 3 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:backend-frontier-ddir
Oct 24, 2026
Merged

frankmcsherry merged 3 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:backend-frontier-ddir

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Replaces #982. Uses #1023's BackendFrontier.

A program installed after a published trace had compacted read the trace's history as stored. Before the compaction frontier, a batch can hold a retraction advanced to the frontier beside the insertion it retracts, which is a count no time had. An iterative scope seeded from such an import need not converge, and tick hung: 3 of 620 differential test seeds with late installs.

Server::import_as_of() is the epoch before the current one, which is also the frontier tick compacts every published trace to. Every import reads as of it, the way Materialize installs a computation "as of" a time:

  • Arrangements: the corgi arrangement Arr carries since. A join or reduce of an import reads through int_proxy: BackendFrontier reads a proxy backend's inputs as of a frontier #1023's BackendFrontier, and a join of two imports reads as of the join of their times.
  • Collections: a collection read from an import has its copied time lanes advanced.
  • Every import path: programs, bindings, snapshots and tails all import through the same path. A tail started late reports the history as one epoch.
  • When there's nothing to do: an arrangement the program built has since at the minimum. Every join and reduce is still wrapped, so each compiles to one operator; at the minimum the wrapper returns after one comparison per unit.

The half join still reads the arrangement as stored. That is correct while the stream is at or beyond the as-of time and published times are at the root. Once #1021 and #1022 land, a follow-up wraps its backend too.

A re-exported import (export "x" = import "y") publishes the producer's arrangement as it is; a reader of x imports it as of its own start.

Tests

  • Regression test: late_install_regressions runs the three seeds that hung (40000075, 40000160, 40000519) under the split, bind and reinstall oracles in every cargo test, in 1.6 s in debug. Without the as-of reads it does not finish.
  • CI sweep: a new step in the release job runs 500 late-install seeds, in about 5 s once built.
  • server_late_import.rs: installs programs late and reads imports as containers, as rows, and as arrangements read by a join, a reduce and a loop.
  • Timing: a probe kept out of the tree times a late install against a million-row history. Install plus first tick takes 83–85 ms, against about 150 ms on master-next, because the reduce of the import now evaluates at one time instead of at every stored epoch.

Checks

  • cargo test -p differential-dataflow -p interactive -p ddir-server: 358 passed.
  • Difftest with all nine oracles: 2,000 passed. Late installs only: 2,500 seeds, including the three that hung, all passed.
  • Release explain suite: 29 passed.
  • Every step of the LDBC CI job: exit 0, with all 41 authored answers matching.

🤖 Generated with Claude Code

https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o

@frankmcsherry
frankmcsherry force-pushed the backend-frontier-ddir branch 3 times, most recently from 80bb185 to 3ad93e1 Compare October 24, 2026 05:08
frankmcsherry and others added 2 commits October 24, 2026 01:14
A program installed after a published trace had compacted read the trace's
history as stored. Before the compaction frontier a batch can hold a
retraction advanced to the frontier beside the insertion it retracts, which
is a count no time had. An iterative scope seeded from such an import need
not converge, and `tick` hung: 3 of 620 differential test seeds with late
installs.

`Server::import_as_of()` is the epoch before the current one, which is also
the frontier `tick` compacts every published trace to. Every import reads as
of it:
- the corgi arrangement `Arr` carries `since`, and a join or reduce of an
  import reads through DD's `BackendFrontier` (a join of two imports reads as
  of the join of their times);
- a collection read from an import has its copied time lanes advanced;
- programs, bindings, snapshots and tails all import through the same path.

The half join reads the arrangement as stored: with the stream at or beyond
the as-of time and published times at the root, the rows it meets and the
times it produces are the same. An arrangement the program built is read as
it is, and an import as of the minimum constructs no wrapper.

The differential tests run with late installs (`LATE_INSTALLS`), and
`reinstall` with them. `late_install_regressions` runs the three seeds that
hung (40000075, 40000160, 40000519) under `split`, `bind` and `reinstall`;
without the as-of reads it does not finish. `server_late_import.rs` reads
imports as containers, as rows, and as arrangements read by a join, a reduce
and a loop, installed late.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
Programs installed late read every import as of the time they start. The
release job now runs 500 late-install seeds under the split, bind and
reinstall oracles, about five seconds once built, beside the regression
test of the three seeds that used to hang, which runs in every cargo test.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
An arrangement the program built has `since` at the minimum, where
`BackendFrontier` returns after one comparison per unit. Wrapping every join
and reduce, rather than branching on the minimum, compiles one operator for
each instead of two.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
@frankmcsherry
frankmcsherry merged commit 476247f into TimelyDataflow:master-next Oct 24, 2026
7 checks passed
frankmcsherry added a commit to frankmcsherry/differential-dataflow that referenced this pull request Oct 24, 2026
…ejoin the results table

Since TimelyDataflow#1024 and TimelyDataflow#1025 a one-off reads every import as of the tick it starts, so the
README no longer calls the one-off runs experimental. The 0.0025 run, rerun on
master-next 7eb7da8, passes again (1,049 ops/s); 0.002 still fails. The one-off row
was split from the results table by a blank line.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u
frankmcsherry added a commit that referenced this pull request Oct 24, 2026
#1010)

* LDBC: an SNB Interactive v2 driver for DDIR, with DuckDB as a baseline

interactive_v2/driver.py reimplements the official Java driver's workload from its
source: per-operation updates on the simulation clock, complex reads on their
frequency grid with the paramgen's parameters, the short-read walk, TCR, and the
per-type 95% on-time audit. systems.py runs it against DDIR (group-commit ticks)
and DuckDB (concurrent connections). snb/operations.py turns the BI CSV inserts and
deletes into single operations with indexed cascades; factors.py rebuilds the
paramgen's inputs so the official paramgen can run. README.md lists what is
reproduced, where it departs, and first SF1 results.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u

* LDBC Interactive v2: per-tick log and server/process CPU time for attribution

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u

* LDBC: IC9 ranks per friend before per request; IC14 iterates costs, then recovers the path; driver --batch-ms

IC9's final top 20 is keyed by request, so every candidate went to one worker; each
slow-path friend now keeps their own 20 first (4 workers: 223 -> 116 ms answer plus
retract). IC14 runs Bellman-Ford on costs alone, keeps every least-cost predecessor,
and walks the smallest next node from the source over the nodes on a cheapest path
(121 -> 74 ms). Answers equal the previous plans on 62 SF1 bindings and the fixture.
The driver can hold DDIR ticks so they are fuller; the client passes DDIR_DIAGNOSTICS.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u

* LDBC Interactive v2: replay and per-read cost tools; README results and attribution

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u

* LDBC Interactive v2: one-off complex reads (--oneoff), from split plans

oneoff/split.py splits IC1, IC9, IC10, IC12, IC13 and IC14 into a standing program
(their binding-independent statements, published) and a one-off program that each
read installs, answers once and drops at the next tick instead of retracting.
Experimental: a one-off installed late reads published traces without #982's
consistent-frontier import.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u

* LDBC Interactive v2: one-off splits for all 14 complex reads; replay --oneoff and --only

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u

* LDBC Interactive v2: delete cascades in the graph program; the driver streams its inputs

graph.py's program takes inserted rows and explicit delete targets and derives the
live relations by snb.updates.State's rules, ahead of the unchanged graph program.
Over the SF1 stream they equal the BI daily journal (three checkpoints); 20,000
operations answer identically either way; test_cascade.py checks the rules in CI.
stream.py reads the snapshot in chunks and the operations a day at a time, and
complex reads are generated lazily: the driver peaks at 181 MB at SF1 (was 8 GB);
the server at 7.5 GB (was 4.7). --cascade client keeps the old path; DuckDB uses it.
Also --sf 0.1 (SF1's mix on SF0.1 data) and factors.py reading plain CSV partitions.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u

* LDBC Interactive v2: sort each relation once when feeding the client-side snapshot

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u

* LDBC Interactive v2 README: one-offs read imports as of their tick; rejoin the results table

Since #1024 and #1025 a one-off reads every import as of the tick it starts, so the
README no longer calls the one-off runs experimental. The 0.0025 run, rerun on
master-next 7eb7da8, passes again (1,049 ops/s); 0.002 still fails. The one-off row
was split from the results table by a blank line.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant