Repository navigation
ShareableList cannot safely handle multibyte utf-8 characters #88336
Description
| BPO | 44170 |
|---|---|
| Nosy | @akulakov, @huwcbjones |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2021-05-18.11:12:09.372>
labels = ['3.8', 'type-bug', 'library']
title = 'ShareableList cannot safely handle multibyte utf-8 characters'
updated_at = <Date 2021-12-09.06:21:18.650>
user = 'https://github.com/huwcbjones'bugs.python.org fields:
activity = <Date 2021-12-09.06:21:18.650>
actor = 'andrei.avk'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-05-18.11:12:09.372>
creator = 'huwcbjones'
dependencies = []
files = ['50049']
hgrepos = []
issue_num = 44170
keywords = ['patch']
message_count = 4.0
messages = ['393868', '393869', '408074', '408075']
nosy_count = 3.0
nosy_names = ['python-dev', 'andrei.avk', 'huwcbjones']
pr_nums = ['26328']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44170'
versions = ['Python 3.8']
Activity
huwcbjones commentedon May 18, 2021
I've experienced a UnicodeDecodeError when adding unicode strings that contain multibye utf-8 characters into a shareable list.
My observation is that ShareableList chunks the list of strings before sending it over the process boundary, however this chunking process is not multibyte aware and will chunk in the middle of multibyte characters.
On the other end, this results in the ShareableList throwing a UnicodeDecodeError when it fails to decode not-a-full multibyte utf-8 character.
From running the attached MWE, I see that the string is sent in two chunks, the first being b'Boom \xf0\x9f\x92\xa5 \xf0\x9f\x92\xa5 \xf0' which clearly splits the 4 bytes of the 💥 character into the first byte and remaining 3 bytes.
huwcbjones commentedon May 18, 2021
The workaround I am using is to manually encode/decode.
For the MWE, this means encoding on the creation side
and decoding before using the string
akulakov commentedon Dec 9, 2021
We classify 'crash' type as seg faults etc, so changing this to 'behavior' type.
akulakov commentedon Dec 9, 2021
I've confirmed this issue is still present in 3.11.
gh-88336: Fix UnicodeDecodeError with multibyte utf8 characters in Sh…
[3.14] gh-88336: Fix UnicodeDecodeError with multibyte utf8 character…
[3.15] gh-88336: Fix UnicodeDecodeError with multibyte utf8 character…