Repository navigation
_pickle crashes when read() returns a bytes subclass for large payloads #158841
Description
Crash report
Loading a pickle containing a large string (>= 1MB) using a custom file-like object that returns a bytes subclass from read() causes a deterministic segmentation fault in CPython 3.15.
PoC
import pickle
import gc
# GC-tracked heap type
class MyBytes(bytes):
def __init__(self, *a):
self.tag = "meta"
class ReaderSub:
def __init__(self, data):
self.d = data
self.p = 0
def read(self, n):
c = self.d[self.p:self.p + n]
self.p += len(c)
return MyBytes(c)
def readline(self):
i = self.d.find(b'\n', self.p)
line = self.d[self.p:i+1] if i >= 0 else self.d[self.p:]
self.p = len(self.d) if i < 0 else i + 1
return line
doc = pickle.dumps({'v': 'B' * (3 * 1024 * 1024)})
obj = pickle.load(ReaderSub(doc))
print("Successfully loaded.")Code Output
Running on Python 3.15.0b1+:
realloc(): invalid old sizeRunning with Python 3.14.7:
Successfully loaded.Root Cause
The crash occurs in Modules/_pickle.c, specifically within the new chunked read loop in _Unpickler_ReadFromFile (around line 1445).
The loop does this:
data = _Pickle_FastCall(self->read, len);
/* ... */
if (_PyBytes_Resize(&data, cursize) < 0)_PyBytes_Resize is being called directly on the foreign object returned by self->read().
Because MyBytes is a subclass of bytes, it passes PyBytes_Check(). However, because it is a Python-level subclass, it is a heap type and is therefore tracked by the Garbage Collector.
This means the actual allocation includes a PyGC_Head before the object data. When _PyBytes_Resize calls PyObject_Realloc(v, PyBytesObject_SIZE + newsize), the pointer v is an interior pointer (offset by 32 bytes from the true malloc address). Reallocating an interior pointer corrupts the heap and immediately crashes the interpreter.
CPython versions tested on:
3.15
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.15.0rc2 (main, Sep 29 2026, 15:02:39) [Clang 22.1.3 ]
Activity
[-]Crash in _pickle: _PyBytes_Resize on bytes subclass causes SIGSEGV[/-][+]_pickle crashes when read() returns a bytes subclass for large payloads[/+]encukou commentedon Oct 5, 2026
cc @vstinner
I guess
_PyBytes_IsMutableneedsPyBytes_CheckExact?vstinner commentedon Oct 5, 2026
Oh. It seems like it's a bad idea to accept bytes subclass in
_PyBytes_Resize(). For example, if it's refcount is greater than 1,_PyBytes_Resize()returns a bytes object, not an instance of the subclass.And so, yes, _pickle should not call
_PyBytes_Resize()if PyBytes_CheckExact() is false.encukou commentedon Oct 8, 2026
Well,
_PyBytes_Resizeis documented, so it might be better to keep the documented behaviour (“creating a new bytes object and destroying the old one, only more efficiently” -- of course, without the optimization in this case).IMO, that's fine.
BytesSubclass(...)[:]and most other operations will do the same thing.@drakeo338, instead of your fix in
pickle.c, could you add a!PyBytes_CheckExact(v)to_PyBytes_ResizeKeepOnError(just before the!_PyObject_IsUniquelyReferenced)?