Skip to content

gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 - #155293

Merged
encukou merged 7 commits into
python:mainfrom
sethmlarson:unicode-stringprep
Aug 18, 2026
Merged

gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454#155293
encukou merged 7 commits into
python:mainfrom
sethmlarson:unicode-stringprep

Conversation

@sethmlarson

@sethmlarson sethmlarson commented Aug 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

This PR fixes makeunicodedata.py and re-runs mkstringprep.py with the new Unicode database. In the future mkstringprep.py should be run as a part of makeunicodedata.py.

I'll take on backporting this PR manually, as we'll need to apply the patch and then re-execute makeunicodedata.py for each Python version as each contains a different Unicode version.

@sethmlarson @StanFromIreland
Don't consider Unicode codepoint attributes outside RFC 3454
6aeb041 
Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

Co-authored-by: Stan Ulbrych <[email protected]>
@bedevere-app bedevere-app Bot mentioned this pull request Aug 6, 2026
Closed
@sethmlarson

sethmlarson commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

@StanFromIreland @malemburg Now that GHA service has recovered, requesting a review on this PR.

encukou
encukou reviewed Aug 11, 2026
View reviewed changes
Comment thread Lib/test/test_codecs.py Outdated
@sethmlarson @encukou
Apply suggestion from @encukou
d978d13 
Co-authored-by: Petr Viktorin <[email protected]>
@sethmlarson
sethmlarson requested a review from encukou August 11, 2026 17:04
@encukou encukou mentioned this pull request Aug 12, 2026
Merged
@encukou

encukou commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

I filed a suggestion as a PR: sethmlarson#2. Otherwise, LGTM -- I didn't find another issue.

@malemburg would be the best person to review this, of course. Marc-André, should we wait for you?

malemburg
malemburg reviewed Aug 13, 2026
View reviewed changes

@malemburg malemburg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have time for a review, but from casual inspection, this looks good to me. Thanks.

@encukou
encukou merged commit 7e109d0 into python:main Aug 18, 2026
54 of 55 checks passed
@encukou

encukou commented Aug 18, 2026

Copy link
Copy Markdown
Member

Thanks for the fix!

I'll take on backporting this PR manually

OK; let me know if you want to delegate.

@sethmlarson
sethmlarson deleted the unicode-stringprep branch August 18, 2026 12:58
@encukou encukou added the needs backport to 3.15 bugs and security fixes label Aug 18, 2026
@miss-islington-app

miss-islington-app Bot commented Aug 18, 2026

Copy link
Copy Markdown

Thanks @sethmlarson for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

@bedevere-app

bedevere-app Bot commented Aug 18, 2026

Copy link
Copy Markdown

GH-156017 is a backport of this pull request to the 3.15 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Aug 18, 2026
encukou added a commit to encukou/cpython that referenced this pull request Aug 18, 2026
@sethmlarson @StanFromIreland @encukou
pythongh-155292: Don't consider Unicode codepoint attributes outside …
e2ad986 
…RFC 3454 (pythonGH-155293)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
(cherry picked from commit 7e109d0)
@bedevere-app

bedevere-app Bot commented Aug 18, 2026

Copy link
Copy Markdown

GH-156020 is a backport of this pull request to the 3.14 branch.

pablogsal pushed a commit that referenced this pull request Aug 18, 2026
@miss-islington @sethmlarson
@StanFromIreland @encukou
[3.15] gh-155292: Don't consider Unicode codepoint attributes outside…
5181304 
… RFC 3454 (GH-155293) (#156017)

gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 (GH-155293)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)

Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
@stdmnpkg stdmnpkg mentioned this pull request Aug 19, 2026
Merged
6 of 8 tasks
encukou added a commit that referenced this pull request Sep 4, 2026
@encukou @sethmlarson @StanFromIreland
[3.14] gh-155292: Don't consider Unicode codepoint attributes outside…
1e54caa 
… RFC 3454 (GH-155293) (GH-156020)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)
The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14
and below, so unassigned characters don't have bidi values.

Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
Yhg1s pushed a commit that referenced this pull request Sep 8, 2026
@encukou
[3.12] gh-155292: Don't consider Unicode codepoint attributes outside…
c016c25 
… RFC 3454 (GH-155293) (GH-156020) (#156930)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)
The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14
and below, so unassigned characters don't have bidi values.
(cherry picked from commit 1e54caa)
Also, add Unicode_3_2_0_FunctionsTest as in the later versions,
to make the new test work.

Co-authored-by: Seth Larson [email protected]
Co-authored-by: Stan Ulbrych [email protected]
Co-authored-by: Petr Viktorin [email protected]
encukou added a commit that referenced this pull request Sep 9, 2026
@miss-islington @encukou
@sethmlarson @StanFromIreland
[3.13] gh-155292: Don't consider Unicode codepoint attributes outside…
c28b121 
… RFC 3454 (GH-155293) (GH-156020) (GH-156922)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)
The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14
and below, so unassigned characters don't have bidi values.
(cherry picked from commit 1e54caa)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>

---------

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
pablogsal pushed a commit that referenced this pull request Oct 1, 2026
@encukou @sethmlarson @StanFromIreland
[3.10] gh-155292: Don't consider Unicode codepoint attributes outside…
d397a49 
… RFC 3454 (GH-155293) (GH-156020) (#156932)

* [3.10] gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 (GH-155293) (GH-156020)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)
The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14
and below, so unassigned characters don't have bidi values.
(cherry picked from commit 1e54caa)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>

* Add Unicode_3_2_0_FunctionsTest

* Regenerate Unicode data

---------

Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
pablogsal pushed a commit that referenced this pull request Oct 1, 2026
@encukou @sethmlarson @StanFromIreland
[3.11] gh-155292: Don't consider Unicode codepoint attributes outside…
69f92eb 
… RFC 3454 (GH-155293) (GH-156020) (#156931)

* [3.11] gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 (GH-155293) (GH-156020)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)
The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14
and below, so unassigned characters don't have bidi values.

Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
(cherry picked from commit 1e54caa)

* Add Unicode_3_2_0_FunctionsTest

* Regenerate Unicode data

---------

Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@malemburg malemburg malemburg left review comments
@StanFromIreland StanFromIreland Awaiting requested review from StanFromIreland
@encukou encukou Awaiting requested review from encukou

Assignees

No one assigned

Labels

topic-unicode type-security A security issue

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sethmlarson @encukou @malemburg