Repository navigation
Conversation
The default name of an imported model contains U+00B7, which CP932 cannot encode, so on Japanese Windows the import failed while writing job.json. Reading falls back to the platform encoding for job.json files written by earlier versions. Fixes huggingface#131
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #131.
register_imported()names a modelImported 路 <label>when no name is given, and_write_meta()wrotejob.jsonin the platform's default encoding. On Japanese Windows that is CP932, which has no路, so the import failed with'cp932' codec can't encode character '\xb7'.What changed
_write_meta()and the output-dir migration writejob.jsonas UTF-8._read_job_meta(), which reads UTF-8 and falls back to the platform encoding. Without the fallback, ajob.jsonthat an earlier version wrote in CP932 with non-ASCII text (a Japanese job name, say) would fail to parse, and the job would drop out of the list.How this was tested
tests/test_jobs.pymakePath.read_text/write_textdefault to CP932. One imports a model with the default name and loads it back; it fails on main with the error from the issue. The other loads a CP932job.jsonwritten the old way; it fails if the fallback is removed.pytest tests/test_jobs.py: 23 passed. The rest of the suite needslerobot, which I didn't install locally.PYTHONUTF8=0 LC_ALL=ja_JP.SJIS,register_imported()on a local model raisesUnicodeEncodeErroron main and registers the model on this branch.ruff checkandruff format --check(0.14.1, the pre-commit version) pass on both files.Other text reads in
jobs.py(the training log,config.json) still use the default encoding. I left them out to keep this tojob.json.Written with Claude Code (Claude Opus 5.5).