Skip to content

Subgraph lists shutdown candidates once each - #827

Open
bos wants to merge 1 commit into
TimelyDataflow:masterfrom
bos:progress-shutdown-once
Open

bos wants to merge 1 commit into
TimelyDataflow:masterfrom
bos:progress-shutdown-once

Conversation

@bos

@bos bos commented Oct 24, 2026 •

Copy link
Copy Markdown

Problem

Subgraph::propagate_pointstamps pushes the node of every change the tracker pushes into maybe_shutdown, then sorts and deduplicates it. A node appears once per changed port and time, so the vector holds many duplicates, and the sort costs more when the pushed changes are not ordered by location (as with #828).

Change

A Vec<bool> per child records whether the child is already in maybe_shutdown. A child is listed when first seen, and its flag is cleared as the list is drained. The sort and dedup are removed.

Why results and progress semantics are unchanged

The same set of children is considered, each once, against the same tracker state. Children that qualify go into temp_active, a BinaryHeap, so the order of consideration has no effect.

Measurements

With the progress_rounds example from #828, one worker, pinned, perf stat on a c7a.8xlarge, medians of 3 runs:

Workload master this PR #828 #828 and this PR
chain 100 20000 19.33 G instr 19.23 G (−0.5%) 16.88 G 16.17 G (−4.2% vs #828)
chain 10 100000 11.72 G 11.65 G (−0.6%) 11.29 G 11.03 G (−2.4%)
loop 20 20000 18.30 G 18.22 G (−0.4%) 17.87 G 17.43 G (−2.4%)

On master alone the effect is small and within the noise in time, because the pushed changes arrive sorted by location, so the sort is cheap. With #828 the pushed changes arrive in component order, and this change saves 2–4% of instructions and 2–6% of time on the chains. In FlowLog it was measured only together with #828 (see there).

Testing

cargo test --workspace --all-targets and cargo test --doc pass (234 tests). cargo clippy --workspace --all-targets reports the same warnings as master.

… deduplicating

`propagate_pointstamps` collected the node of every pushed change into
`maybe_shutdown` and then sorted and deduplicated it. A per-child flag now
lists each node once, as it is first seen. The candidates feed a binary
heap, so their order does not matter.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant