gh-155811: Add a seqcount to gc_stats to prevent torn reads

[maurycy]

See #155811 for the context.

tl;dr get_gc_stats reads the stats without pausing the target (by design), so it can return torn data.

The PR adds a seqcount to gc_stats atomically increased by the writer around the stats update on every collection, where odd means it's in progress.

gcmon is the main consumer.

There's no retry, as per #155811 (comment). It's not needed. gc.collect() does not happen that often:

2026-08-15T10:53:48.504627000+0200 maurycy@gimel /Users/maurycy/work/cpython (gc-mon-seq 1291568?) % sudo ./python.exe gc_seq_collision.py 
successes=1966598 failures=191
GC stats changed while being read; retry later

For:

import subprocess
import sys
import time

import _remote_debugging

target = """
import gc
import os
import time

print(os.getpid(), flush=True)
while True:
    gc.collect(0)
    time.sleep(0.01)
"""

p = subprocess.Popen(
    [sys.executable, "-c", target],
    stdout=subprocess.PIPE,
    text=True,
)
try:
    pid = int(p.stdout.readline())
    monitor = _remote_debugging.GCMonitor(pid, debug=True)
    successes = 0
    failures = 0
    messages = set()
    deadline = time.monotonic() + 10.0
    while time.monotonic() < deadline:
        try:
            monitor.get_gc_stats(all_interpreters=False)
            successes += 1
        except RuntimeError as exc:
            failures += 1
            messages.add(str(exc))
    print(f"successes={successes} failures={failures}")
    for message in sorted(messages):
        print(message)
finally:
    p.terminate()
    p.wait()

I'm not sure what's optimal non-hacky test. We haven't had one in #152448. To be honest, it's implicitly tested by the current happy-path tests in test_gc_stats.

• Issue: Torn reads of GC statistics from external process via get_gc_stats #155811

[maurycy]

cc @sergey-miryanov @nascheme

[sergey-miryanov]

I'm concerned that we now need to read memory three times with this approach:

struct gc_stats {
    uint32_t before_update_seq;
    struct gc_young_stats_buffer young;
    struct gc_old_stats_buffer old[2];
    uint32_t after_update_seq;
};

If before_update_seq and after_update_seq don't match, we have a torn read.

What do you think?

[maurycy]

@sergey-miryanov I'd push back. :-)

successes=1966598 failures=191

is around ~200KHz.

I'm not sure if there's a guarantee about copy-order and monotonicity of process_vm_readv etc.

Also, assuming these guarantees, before_update_seq and after_update_seq might be racy? What if:

1. Reader read before_update_seq (0)
2. Writer incremented before_update_seq (1)
3. Reader read after_update_seq (0)
4. Writer incremented after_update_seq (2)

[sergey-miryanov]

Yes, my variant will not work, I thought more about it - so drop it :)

[sergey-miryanov]

What if:
Writer:

1. Move update_seq to the begin of struct
2. Write to update_seq

Reader:

1. Read gc_stats (update_seq + stats)
2. Read update_seq
3. Compare update_seq(1) and update_seq(2)

[sergey-miryanov]

Even better, we could pack both counters into a single 32-bit integer — using the lower 16 bits for before and the upper 16 bits for after. This would let us read both values with a single atomic load.

What do you think?

[maurycy]

@sergey-miryanov The latter requires a full atomic snapshot read of the whole struct. Otherwise there's a race risk?

S = { (before=0, after=0), stats }

1. Reader read (before=0, after=0)...
2. Writer incremented `before=1`
3. Writer changed the stats
4. Reader read ...stats
5. Writer incremented `after=1`

Atomicity is not guaranteed:

       The data transfers performed by process_vm_readv() and
       process_vm_writev() are not guaranteed to be atomic in any way.

The former (ie: S = { update_seq, stats }) depends on a guarantee that update_seq is read before stats (copy-order / monotonicity, at least: target[0] is read before target[n > 0]), and I'm not sure if ReadProcessMemory or mach_vm_read_overwrite provide it. Windows does not document it, and macOS Mach VM seems to be sparsely documented.

Perhaps we could do _Py_RemoteDebug_BatchedReadRemoteMemory, since iovec provides such guarantee:

 Buffers are processed in array order.  This means that
 process_vm_readv() completely fills local_iov[0] before proceeding
 to local_iov[1], and so on.  Likewise, remote_iov[0] is completely
 read before proceeding to remote_iov[1], and so on.

That said, target[0] is read before target[n > 0] is likely how it works right now, but that's an implementation detail. I think that even memcpy on Linux/ARM breaks strong target[n] is read before target[n+1] and exactly-once read:

• https://github.com/torvalds/linux/blob/66edb901bf874d9e0787326ba12d3548b2da8700/arch/arm64/lib/memcpy.S#L57
• https://github.com/torvalds/linux/blob/66edb901bf874d9e0787326ba12d3548b2da8700/arch/arm64/lib/memcpy.S#L68-L74

To be honest, I don't think that one additional syscall is that bad. Measured 200KHz is solid (way more than gcmon needs), it's not a hot path, and is trivial to wrap one's head around. In practice (implementation detail), I remember measuring it and small reads are much faster.

I also added update_seq at the end to avoid breaking clients. Not sure if this matters but I remember this comment.

The thing that I think is worth measuring - and I'm not an expert here - is the probability of collision. What's the maximum observable GC? The time.sleep(0.1) is 100Hz. Can we expect the rate of 100Khz GC collections?

Sorry for so much text :) I keep posting essays here but wanted to explain myself, since maybe I'm making a fatal blunder here. Does it make sense to you?

[sergey-miryanov]

I tested on this script:

import gc
import time

ts_start = time.perf_counter_ns()
stats_start = gc.get_stats()
while True:
    try:
        gc.collect(2)
    except KeyboardInterrupt:
        break

ts_stop = time.perf_counter_ns()
stats_stop = gc.get_stats()

dt = (ts_stop - ts_start) / 1e9
dc = (stats_stop[2]["collections"] - stats_start[2]["collections"])
dd = (stats_stop[2]["duration"] - stats_start[2]["duration"])

print("collections", dc)
print(f"durations: {dd:.4f}, rate: {dc/dd:.2f}, avg: {dd/dc:.6f}")
print(f"time: {dt:.4f}, rate: {dc/dt:.2f}, avg: {dt/dc:.6f}")

Average duration for gc.collect(2) is about 0.210 ms on my machine (11th Gen Intel(R) Core(TM) i5-11600K @ 3.90GHz, Windows 11)

> python .\empty_gc.py
collections 17523
durations: 3.7099, rate: 4723.37, avg: 0.000212
time: 3.7173, rate: 4713.90, avg: 0.000212

Average duration of _remote_debugging.get_gc_stats is about 0.560 ms.

| PID:IID | Metric      |     Count |              Sum |   Avg |   P50 |   P90 |   P95 |   P99 |    Cov |      F |
|---------|-------------|-----------|------------------|-------|-------|-------|-------|-------|--------|--------|
| Total   | GC Pause(0) |         3 |            0.286 | 0.095 | 0.094 | 0.102 | 0.103 | 0.104 | 100.0% |  1.000 |
|         | GC Pause(2) | 804/19676 | 181.871/4180.404 | 0.226 | 0.225 | 0.236 | 0.241 | 0.294 |   4.1% | 22.985 |
|---------|-------------|-----------|------------------|-------|-------|-------|-------|-------|--------|--------|
|         | Read Time   |       268 |          148.984 | 0.556 | 0.546 | 0.576 | 0.622 | 0.790 |        |        |

I don't think we can achieve 100Khz rate for gc.collect :)

[maurycy]

@colesbury I know that we've just removed _PySeqLock but I'd love your sanity check here! Thank you.

[sergey-miryanov]

I don't think we should retry more than once if the GC stats read is inconsistent.

I mean one more try if we fail on the first attempt. @maurycy, would you mind adding a single retry pass?

[maurycy]

I don't think we should retry more than once if the GC stats read is inconsistent.

I mean one more try if we fail on the first attempt. @maurycy, would you mind adding a single retry pass?

@sergey-miryanov It introduces some complexity. Do you think it could be avoided with gcmon retrying?

[sergey-miryanov]

@maurycy It seems cheaper to do the retry inside the call rather than from gcmon.

A GCMonitor.get_gc_stats call costs about 30μs on my machine right now. But with the new GC, we can have pauses as short as ~200ns with ~100μs intervals between them.

[pablogsal]

LGTM!

[maurycy]

That was faster than light. Thank you, @pablogsal. 🖤

[miss-islington-app]

Thanks @maurycy for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

[miss-islington-app]

Sorry, @maurycy and @pablogsal, I could not cleanly backport this to 3.15 due to a conflict.

Please backport manually with cherry_picker, see the devguide for more information.

cherry_picker 5fecd448bb120378978a37dde65dfce233d88c0d 3.15

[sergey-miryanov]

Thanks all!

[sergey-miryanov]

@maurycy Do you want to take a look at the backport? If not, I can handle it myself.

[bedevere-app]

GH-158829 is a backport of this pull request to the 3.15 branch.

[maurycy]

@sergey-miryanov #158829

[bedevere-app]

GH-158829 is a backport of this pull request to the 3.15 branch.