Skip to content

Skip irrelevant count scans during lazy progress reporting - #830

Draft
frankmcsherry wants to merge 2 commits into
TimelyDataflow:masterfrom
frankmcsherry:codex/progress-visibility
Draft

frankmcsherry wants to merge 2 commits into
TimelyDataflow:masterfrom
frankmcsherry:codex/progress-visibility

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Lazy progress reporting evaluates visibility before checking whether an update is a subtraction, and is_global scans retained timestamp counts even after an earlier frontier element establishes dominance.

Check the scope-boundary exception and subtraction sign before asking the tracker about visibility, and short-circuit PortInformation::is_global when dominated. The predicates return the same results and add no state. Queries that survive the dominance check still inspect multiplicity.

All three cases occur in the existing Differential BFS example. In one instrumented baseline run (10k vertices, 100k initial edges, ten waves of 1k separately timestamped replacements, four workers), 7,158 lazy-broadcast candidates included 3,373 dominated candidates, 764 nonnegative candidates, and 308 boundary candidates. These disjoint categories avoid scans over 14,986 raw batch entries; the remaining queries scan 9,230 entries. Dominated cases principally came from Leave, Arrange: Reduce, and Feedback, and nonnegative cases from ArrangeByKey. Counts vary with scheduling and are diagnostic, not performance measurements.

Seven interleaved, uninstrumented release runs of existing Timely PageRank, hash-join, barrier, and Differential BFS examples showed median execution times within 2% of baseline. The examples establish that the cases occur, but their small retained progress state does not establish a material application speedup. No benchmark example is added.

A product-timestamp regression test exercises dominance, redundant frontier counts, and visibility after the frontier advances. Workspace tests, doctests, and Clippy passed before removal of the standalone benchmark; production and test code are unchanged. Existing application examples were additionally rebuilt and run against this independent draft.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant