Repository navigation
DDIR: every import is read as of the time its reader starts - #1024
Merged
frankmcsherry merged 3 commits intoOct 24, 2026
Merged
frankmcsherry merged 3 commits into
frankmcsherry merged 3 commits into
Conversation
frankmcsherry
force-pushed
the
backend-frontier-ddir
branch
3 times, most recently
from
October 24, 2026 05:08
80bb185 to
3ad93e1
Compare
A program installed after a published trace had compacted read the trace's history as stored. Before the compaction frontier a batch can hold a retraction advanced to the frontier beside the insertion it retracts, which is a count no time had. An iterative scope seeded from such an import need not converge, and `tick` hung: 3 of 620 differential test seeds with late installs. `Server::import_as_of()` is the epoch before the current one, which is also the frontier `tick` compacts every published trace to. Every import reads as of it: - the corgi arrangement `Arr` carries `since`, and a join or reduce of an import reads through DD's `BackendFrontier` (a join of two imports reads as of the join of their times); - a collection read from an import has its copied time lanes advanced; - programs, bindings, snapshots and tails all import through the same path. The half join reads the arrangement as stored: with the stream at or beyond the as-of time and published times at the root, the rows it meets and the times it produces are the same. An arrangement the program built is read as it is, and an import as of the minimum constructs no wrapper. The differential tests run with late installs (`LATE_INSTALLS`), and `reinstall` with them. `late_install_regressions` runs the three seeds that hung (40000075, 40000160, 40000519) under `split`, `bind` and `reinstall`; without the as-of reads it does not finish. `server_late_import.rs` reads imports as containers, as rows, and as arrangements read by a join, a reduce and a loop, installed late. Co-Authored-By: Claude Fable 5.1 <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
Programs installed late read every import as of the time they start. The release job now runs 500 late-install seeds under the split, bind and reinstall oracles, about five seconds once built, beside the regression test of the three seeds that used to hang, which runs in every cargo test. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
frankmcsherry
force-pushed
the
backend-frontier-ddir
branch
from
October 24, 2026 05:17
3ad93e1 to
4c5cfbe
Compare
An arrangement the program built has `since` at the minimum, where `BackendFrontier` returns after one comparison per unit. Wrapping every join and reduce, rather than branching on the minimum, compiles one operator for each instead of two. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
This was referenced Oct 24, 2026
frankmcsherry
added a commit
to frankmcsherry/differential-dataflow
that referenced
this pull request
Oct 24, 2026
…ejoin the results table Since TimelyDataflow#1024 and TimelyDataflow#1025 a one-off reads every import as of the tick it starts, so the README no longer calls the one-off runs experimental. The 0.0025 run, rerun on master-next 7eb7da8, passes again (1,049 ops/s); 0.002 still fails. The one-off row was split from the results table by a blank line. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u
frankmcsherry
added a commit
that referenced
this pull request
Oct 24, 2026
#1010) * LDBC: an SNB Interactive v2 driver for DDIR, with DuckDB as a baseline interactive_v2/driver.py reimplements the official Java driver's workload from its source: per-operation updates on the simulation clock, complex reads on their frequency grid with the paramgen's parameters, the short-read walk, TCR, and the per-type 95% on-time audit. systems.py runs it against DDIR (group-commit ticks) and DuckDB (concurrent connections). snb/operations.py turns the BI CSV inserts and deletes into single operations with indexed cascades; factors.py rebuilds the paramgen's inputs so the official paramgen can run. README.md lists what is reproduced, where it departs, and first SF1 results. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: per-tick log and server/process CPU time for attribution Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC: IC9 ranks per friend before per request; IC14 iterates costs, then recovers the path; driver --batch-ms IC9's final top 20 is keyed by request, so every candidate went to one worker; each slow-path friend now keeps their own 20 first (4 workers: 223 -> 116 ms answer plus retract). IC14 runs Bellman-Ford on costs alone, keeps every least-cost predecessor, and walks the smallest next node from the source over the nodes on a cheapest path (121 -> 74 ms). Answers equal the previous plans on 62 SF1 bindings and the fixture. The driver can hold DDIR ticks so they are fuller; the client passes DDIR_DIAGNOSTICS. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: replay and per-read cost tools; README results and attribution Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: one-off complex reads (--oneoff), from split plans oneoff/split.py splits IC1, IC9, IC10, IC12, IC13 and IC14 into a standing program (their binding-independent statements, published) and a one-off program that each read installs, answers once and drops at the next tick instead of retracting. Experimental: a one-off installed late reads published traces without #982's consistent-frontier import. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: one-off splits for all 14 complex reads; replay --oneoff and --only Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: delete cascades in the graph program; the driver streams its inputs graph.py's program takes inserted rows and explicit delete targets and derives the live relations by snb.updates.State's rules, ahead of the unchanged graph program. Over the SF1 stream they equal the BI daily journal (three checkpoints); 20,000 operations answer identically either way; test_cascade.py checks the rules in CI. stream.py reads the snapshot in chunks and the operations a day at a time, and complex reads are generated lazily: the driver peaks at 181 MB at SF1 (was 8 GB); the server at 7.5 GB (was 4.7). --cascade client keeps the old path; DuckDB uses it. Also --sf 0.1 (SF1's mix on SF0.1 data) and factors.py reading plain CSV partitions. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: sort each relation once when feeding the client-side snapshot Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2 README: one-offs read imports as of their tick; rejoin the results table Since #1024 and #1025 a one-off reads every import as of the tick it starts, so the README no longer calls the one-off runs experimental. The 0.0025 run, rerun on master-next 7eb7da8, passes again (1,049 ops/s); 0.002 still fails. The one-off row was split from the results table by a blank line. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #982. Uses #1023's
BackendFrontier.A program installed after a published trace had compacted read the trace's history as stored. Before the compaction frontier, a batch can hold a retraction advanced to the frontier beside the insertion it retracts, which is a count no time had. An iterative scope seeded from such an import need not converge, and
tickhung: 3 of 620 differential test seeds with late installs.Server::import_as_of()is the epoch before the current one, which is also the frontiertickcompacts every published trace to. Every import reads as of it, the way Materialize installs a computation "as of" a time:Arrcarriessince. A join or reduce of an import reads through int_proxy: BackendFrontier reads a proxy backend's inputs as of a frontier #1023'sBackendFrontier, and a join of two imports reads as of the join of their times.sinceat the minimum. Every join and reduce is still wrapped, so each compiles to one operator; at the minimum the wrapper returns after one comparison per unit.The half join still reads the arrangement as stored. That is correct while the stream is at or beyond the as-of time and published times are at the root. Once #1021 and #1022 land, a follow-up wraps its backend too.
A re-exported import (
export "x" = import "y") publishes the producer's arrangement as it is; a reader ofximports it as of its own start.Tests
late_install_regressionsruns the three seeds that hung (40000075, 40000160, 40000519) under thesplit,bindandreinstalloracles in everycargo test, in 1.6 s in debug. Without the as-of reads it does not finish.server_late_import.rs: installs programs late and reads imports as containers, as rows, and as arrangements read by a join, a reduce and a loop.Checks
cargo test -p differential-dataflow -p interactive -p ddir-server: 358 passed.🤖 Generated with Claude Code
https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o