Skip to content

fix(remote-helper): push silently drops or corrupts files containing non-ASCII text - #59

Merged
aloth merged 1 commit into
aloth:mainfrom
am009:patch-1
Oct 9, 2026
Merged

aloth merged 1 commit into
aloth:mainfrom
am009:patch-1

Conversation

@am009

@am009 am009 commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Summary

git push through git-remote-overleaf silently drops files whose contents contain
non-ASCII characters, e.g. any UTF-8 LaTeX source with Chinese comments. In a 12-file
test tree the first push reported success, but only 6 files reached the project:
main.tex, ref.bib, build.sh, .gitignore and two section files never arrived, and
the file deleted in that commit was not deleted either. Content that does get through is
mangled, because the same code path re-encodes it as latin1.

Root cause

parseFastExport() reads the fast-export stream line by line and re-joins the lines of a
data <N> section (src/remote-helper.ts:189):

const lineBytes = Buffer.from(line + '\n', 'binary');   // latin1
dataBuffer = Buffer.concat([dataBuffer, lineBytes]);
if (dataBuffer.length >= pendingDataBytes) { ... }

data <N> counts UTF-8 bytes, but 'binary' (latin1) encodes every character as a
single byte. For a blob containing non-ASCII text the accumulated buffer is systematically
short, so the >= pendingDataBytes check fires late: the parser stays in the data state
and swallows the following M <mode> :<mark> <path> / blob / mark lines as blob
content. Files whose M line is consumed that way are never added to files and never
uploaded.

Deficits measured on the test tree (UTF-8 length vs. latin1 length):

file UTF-8 latin1 deficit
.gitignore 305 209 96
build.sh 1,792 1,462 330
main.tex 4,686 3,704 982
ref.bib 58,376 58,322 54

git pull / git fetch are not affected: the import path writes entry.getData()
buffers directly to stdout.

Fix

One line, so the byte count matches data <N> again:

-      const lineBytes = Buffer.from(line + '\n', 'binary');
+      const lineBytes = Buffer.from(line + '\n', 'utf8');

Verification

  • Reproduced on a 12-file project: first push uploaded 6 files, left the deleted file in
    place, and olcli diff still reported 6 missing files.
  • After the change, cleared .git/overleaf/marks, .git/overleaf/state.json and
    refs/overleaf/main and pushed again. The debug log (GIT_REMOTE_OVERLEAF_DEBUG=1)
    shows all 12 files uploaded and the deleted path removed.
  • Downloaded the project zip afterwards and compared every file byte for byte with the
    local working tree: 12/12 identical, including the files with Chinese text, and no
    remote-only files left. olcli diff reports no differences.

Repro

mkdir repro && cd repro && git init
printf '%% 中文注释\nhello\n' > main.tex
git add main.tex && git commit -m init
git remote add origin overleaf::https://www.overleaf.com/project/<project-id>
git push -u origin main
# main.tex is missing from the project (or arrives mangled)

Scope

This fixes text content (valid UTF-8), which is what Overleaf projects contain. Genuinely
binary blobs are still decoded by readline as UTF-8, so PNG/PDF payloads remain
unsupported on push; fixing that properly means reading data <N> as raw bytes instead of
going through readline. I left that out to keep this change minimal — happy to follow up
if you want it.

There is no automated test here: parseFastExport() is not exported, and the existing
tests cover the client/config/ignore layers. I can add a regression test if you are open
to exporting the function (or extracting the stream parser).

Change buffer encoding from binary to UTF-8 for data accumulation.
@aloth
aloth merged commit 2275f7d into aloth:main Oct 9, 2026
aloth added a commit that referenced this pull request Oct 9, 2026
Fixes git push through git-remote-overleaf for non-ASCII text (#59) and binary files.
@aloth

aloth commented Oct 9, 2026

Copy link
Copy Markdown
Owner

Thank you for the precise report and fix, including the byte-count analysis and the repro. This shipped in v0.13.1. Building on it, the remote helper now reads data <N> payloads as raw bytes instead of going through readline, so binary files such as images and PDFs push correctly as well, and parseFastExport moved to src/fast-export.ts with regression tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants