Skip to content

perf: share repeated source message IDs when reading PO catalogs - #1395

Open
SAY-5 wants to merge 1 commit into
python-babel:masterfrom
SAY-5:fix/po-message-id-sharing-20261018
Open

SAY-5 wants to merge 1 commit into
python-babel:masterfrom
SAY-5:fix/po-message-id-sharing-20261018

Conversation

@SAY-5

@SAY-5 SAY-5 commented Oct 18, 2026

Copy link
Copy Markdown

Reading the same source messages from several PO catalogs currently allocates a separate message-ID string for each catalog. Share normalized singular IDs and each plural component through sys.intern() when the parser retains a message. Translations, contexts and mutable message metadata remain independent; ignored obsolete IDs are not interned.

Enable this only on conventional CPython builds. Free-threaded CPython keeps interned strings until shutdown even when its GIL is enabled, so those builds and other Python implementations keep the existing behavior.

Refs #571.

For eight catalogs with 128 long IDs each, fresh-process tracemalloc measurements on conventional CPython 3.13 gave:

Input Upstream retained bytes Patched retained bytes
Shared IDs 984,192 742,272
Shared Unicode IDs 1,033,344 748,416
Unique IDs per catalog 984,192 1,399,280

CPython 3.14 reproduced the same differences. Shared IDs went from 1,024 distinct string objects to 128. This favors overlapping catalogs: the unique-ID case adds 415,088 bytes of interning bookkeeping, and that difference remained after releasing catalogs. In 100 successive batches of 128 unique IDs, the retained difference plateaued at the same value; this is an observation, not a bound for all workloads. These are traced Python allocations, not total RSS or a speedup claim.

Inputs were prepared before tracing, with one warm-up parse and locale=None. Separate cold-import measurements included the build guard: the median retained-allocation difference was 709 bytes on 3.13 and 763 bytes on 3.14 over three pairs. The baseline already initializes sysconfig through zoneinfo on those runtimes; this does not establish the same cost on every supported runtime.

Validation: the original sharing assertions fail on upstream and pass with sharing enabled; the free-threaded/other-implementation guard assertions fail before the guard and pass afterward. All 129 targeted parser/catalog cases pass on CPython 3.13 and 3.14, and 33 writer/MO compatibility cases pass on 3.13, using the complete imported CLDR 48.2 data. The configured Ruff 0.14.10 check passes. Runtime branch tests simulate build configuration; no real free-threaded or PyPy runtime was available locally.

Shared/unique/Unicode memory reproducer

Run this in a fresh process for each case (shared, unique, or unicode) on upstream and the PR branch. The source checkout must be importable as babel.

import gc
import io
import json
import sys
import tracemalloc
from babel.messages.pofile import read_po

case = sys.argv[1]
assert case in {"shared", "unique", "unicode"}
prefix = "Ω 日本語 café " if case == "unicode" else "Long source message "
ids = [[f"{prefix}{batch if case == 'unique' else 0:03d}:{i:03d}: "
        + prefix * 10 for i in range(128)] for batch in range(8)]
texts = ["".join("msgid " + json.dumps(value, ensure_ascii=False) + "\n"
                 + f'msgstr "translation {batch}:{i}"\n\n'
                 for i, value in enumerate(values))
         for batch, values in enumerate(ids)]

def read(text):
    return read_po(io.StringIO(text), locale=None, abort_invalid=True)

read('msgid "Warm-up long source identifier"\nmsgstr "value"\n')
gc.collect()
tracemalloc.start()
catalogs = [read(text) for text in texts]
gc.collect()
print("retained, peak:", tracemalloc.get_traced_memory())
print("distinct ID objects:", len({id(catalog[value].id)
      for catalog, values in zip(catalogs, ids) for value in values}))

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant