Skip to content

fix(csv-parse): wait for full CRLF before auto-detecting record delimiter - #539

Open
chuanghiduoc wants to merge 1 commit into
adaltas:masterfrom
chuanghiduoc:fix/csv-parse-utf16le-crlf-lookahead
Open

chuanghiduoc wants to merge 1 commit into
adaltas:masterfrom
chuanghiduoc:fix/csv-parse-utf16le-crlf-lookahead

Conversation

@chuanghiduoc

Copy link
Copy Markdown

Summary

Streaming csv-parse can mis-detect a UTF-16LE CRLF record delimiter when a chunk boundary splits the four-byte \r\n after a closing quote. The parser permanently selects \r, then emits a bogus ["\n"] record (silent corruption) or throws CSV_INVALID_OPENING_QUOTE on valid input. parseSync and a single-write parse handle the same bytes correctly.

const input = Buffer.from('"b"\r\n"c"\r\n', "utf16le");
const parser = parse({ encoding: "utf16le" }, (err, records) => {
  // expected [["b"], ["c"]]
});
for (let i = 0; i < input.length; i++) parser.write(input.subarray(i, i + 1));
parser.end();
// actual: Invalid Opening Quote ... value is "\n"

A single record ('"b"\r\n') silently becomes [["b"], ["\n"]]. Two-byte and five-byte writes happen to work; one-, three- and nine-byte writes fail, so the result depends on chunk alignment.

This is present on current HEAD and on published [email protected]. It reproduces with explicit encoding: "utf16le" and with BOM auto-detection, and is not fixed by #537.

Fix

When the record delimiter is still undiscovered, __needMoreData used recordDelimiterMaxLength (0) for the lookahead after a closing quote. It therefore proceeded with only part of the encoded CRLF, and auto-discovery accepted \r because the full \r\n was not buffered.

The fix computes an encoding-aware CRLF lookahead once and uses it for both the general delimiter check and the post-quote check. Auto-discovery now waits for the complete \r\n and selects it over \r.

Tests

Added two cases to packages/csv-parse/test/option.encoding.ts for one and two quoted records written one byte at a time.

  • npx mocha 'test/**/*.{js,ts}' --ignore test/samples.js -> 562 passing, 2 pending
  • npx tsc --noEmit -> clean
  • Prettier/ESLint clean for the changed files

test/samples.js is ignored locally because its new URL(".", import.meta.url).pathname produces an invalid D:\D:\... path on Windows; it runs normally on Linux CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant