[peter-nu]

Lua 5.1's string library classifies bytes with the C library (isspace(), isalpha(), tolower(), …), and that follows LC_CTYPE. Python sets LC_CTYPE from the environment at startup. In a UTF-8 locale macOS counts byte 0xA0 as a space and 0xE0 as a letter, and Latin-1 or Windows code-page locales lower-case bytes above 0x7F. "à" is C3 A0 and "ム" is E3 83 A0, so %s, %a and string.lower() cut or change them. Later mw.ustring/mw.text calls then fail:

[string "mw_text"]:81: bad argument #1 for 'gsub' (string is not UTF-8)

This is the error in #301 (Windows, string.lower() in a fr.wikipedia module). On macOS it hits ordinary en.wiktionary pages. 17 sample pages gave 179 such errors, 122 of them on à. Linux in a UTF-8 locale, like MediaWiki's servers, classifies only ASCII, so it never sees them.

from wikitextprocessor import Wtp
wtp = Wtp()
wtp.add_page("Module:test", 828, """
local export = {}
function export.trim(frame)
  return mw.text.trim((frame.args[1]:gsub("%s+$", "")))
end
return export
""", model="Scribunto")
wtp.start_page("test")
print(wtp.expand("{{#invoke:test|trim|umanità}}"))  # macOS: Lua error above

Change. A new lua/_c_locale_string.lua, loaded once in initialize_lua(), wraps string.find/match/gmatch/gsub. Each wrapper rewrites the pattern so that every class (%a %c %d %l %p %s %u %w %x and their complements, inside or outside [...]) becomes an explicit ASCII byte set. The pattern is read the way lstrlib.c reads it. %bxy, %f[...], back-references and malformed patterns are kept, so Lua still reports the error. Rewritten patterns are cached. string.lower/upper change only A-Z/a-z when the string has bytes above 0x7F. The string metatable shares the table, so s:find() is covered.

This doesn't make string Unicode-aware. On any machine it gives exactly what the C locale gives, which is what MediaWiki returns. So modules behave as on the wiki, and Linux results don't change.

Tests. tests/test_lua_locale.py checks all 256 bytes against every class (bare, in a set, negated) and lower/upper of every byte. It runs once in the current locale and once in a locale that misclassifies bytes, if the system has one (macOS: C.UTF-8; Linux: a Latin-1 locale; skipped otherwise). It also checks module calls that trim, case-change and split UTF-8 text, and that other pattern syntax survives. Before the change the new tests fail on macOS and on Linux with a Latin-1 locale (Debian, de_DE.ISO-8859-1). After it the suite passes (1009 tests) on macOS in the default locale and in LC_ALL=C, and on Linux. On Linux in C.UTF-8, results are the same with and without the change.

Checks.

• 200,000 random patterns gave the same results as unwrapped Lua under LC_ALL=C.
• The 17 en.wiktionary pages expand to the same text in the default locale as under LC_ALL=C.
• Cost: about 0.1 ms in initialize_lua() and 0.05–0.5 µs per string call. Page expansion times were within noise.

Worth your eye:

• Argument errors such as string.find(nil, "x") are still raised by the original functions, but the message may name the wrapper differently.
• Bytes above 0x7F in a pattern's own text ([à], ranges like [\128-\255]) are copied unchanged, as before.
• A range ending in % inside a set ([a-%]) is left untranslated, because lstrlib reads it oddly. Such a pattern keeps the locale behaviour.

🤖 Generated with Claude Code

[kristian-clausal]

This looks fine (although I won't pretend to get everything at a fine detail), but if the purpose of this is to emulate how stuff works on Linux, then we do not want to replace the original implementation when it's not necessary. That is, there should probably be a check here if we're running on Windows or MacOS, and only run the compatibility code if so. This should also prevent any problems on the Linux side if the implementation is not completely airtight, and reduce any performance penalties (even if very small).

[kristian-clausal]

This looks fine at a quick glance. Only thing I'd like to ask is to gate whether to use this compatibility code behind a check of if the system that is running is not compatible. Otherwise, if we're on Linux it is unnecessary.