[vasdommes]

Handle process deaths without scanning unrelated monitors and connections

Problem

On a node with many monitors, links or connections, each process death took
time proportional to all of them, not only to the ones it affects:

• ncEffectDied partitioned all links and monitors of the node
  (splitNotif), for a node death also all watchers, and it rebuilt
  registeredOnNodes.
• cleanupProcess partitioned all of localConnections on every process exit.

Local sends also go through the node controller, so they wait behind slow
deaths. We found this in a task scheduler built on distributed-process, which
keeps ~1,000 monitors and ~600 connections and runs tens of thousands of
short-lived processes; these changes cut its total run time by ~10–20%.

Change

The identifiers that die with an identifier (see impliesDeathOf) form at most
three ranges of keys in the derived Ord instances: for a node, the node
itself, its processes and its typed channels; for a process, the process and
its typed channels.

• Internal.Messaging: deathKey and deathKeys. deathKey ident is a
  monotone projection, and impliesDeathOf a b holds if and only if
  deathKey a b is in deathKeys a.
• Internal.BiMultiMap: partitionOn, partitionOnBy1st and
  partitionOnBy2nd. partitionOn f ps m is
  Map.partitionWithKey (\k _ -> elem (f k) ps) m for a monotone f,
  computed with Map.spanAntitone.
• Internal.Types: IdentifierKind and identifierKind, next to nodeOf.
• Node: splitNotif, the removal of dead watchers, registeredOnNodes and
  cleanupProcess use them.

Behaviour does not change, including the order of notifications and of
forwarded deaths. The code relies on the derived Ord instances of
Identifier, ProcessId and SendPortId; their declarations now say so, and
the new tests check it.

Benchmark

The first commit adds a deaths group to the benchmarks: a process death
(spawn, monitor, exit, notification) while the node holds N unrelated
monitors or connections. Time per death, mean of two runs, GHC 9.10.3, -O2:

Unrelated entries	Before	+ node controller fix	+ process exit fix
none	14 µs	16 µs	16 µs
1,000 monitors	87 µs	20 µs	20 µs
10,000 monitors	2,870 µs	45 µs	43 µs
1,000 connections	91 µs	92 µs	19 µs
10,000 connections	2,920 µs	2,280 µs	37 µs

Differences of a few µs are noise. The remaining growth from 1,000 to 10,000
entries comes from garbage collection of the larger heap: with +RTS -A64m,
both cases take the same time.

To compare, run at the first commit and then at the last one:

cabal bench distributed-process --benchmark-options '-p deaths --csv base.csv'
cabal bench distributed-process --benchmark-options '-p deaths --baseline base.csv'

Tests

• New test-suite TestDeathKeys in distributed-process-tests:
  • for all pairs from a set of 26 identifiers, deathKey agrees with
    impliesDeathOf, and deathKey is monotone;
  • QuickCheck: partitionOn, partitionOnBy1st and partitionOnBy2nd give
    the same results as Map.partitionWithKey, partitionWithKeyBy1st and
    partitionWithKeyBy2nd.
• All distributed-process-tests suites pass.

Notes for reviewers

• TestDeathKeys uses functions that are new in this PR. Its lower bound on
  distributed-process should become the next release when versions are bumped.
• Two scans remain, both out of scope here:
  • every death still scans registeredHere, which is keyed by name;
  • a node disconnect still scans all connections in
    closeImplicitReconnections, because connections are keyed by from,
    not to.

🤖 Generated with Claude Code

[davidsd]

For context, @vasdommes and I encountered performance problems with a distributed-process-based scheduler that we have been working on, and this was one of the fixes.

[LaurentRDC]

Thanks! I mostly have some organizational comments. Nothing showstopping

[LaurentRDC]

Some tests are flaky, I think this is ready to merge.

Thanks @vasdommes !

[LaurentRDC]

distributed-process-0.7.10 has been released on Hackage with this change