Repository navigation
fix(csv-parse): wait for full CRLF before auto-detecting record delimiter - #539
Open
chuanghiduoc wants to merge 1 commit into
Open
chuanghiduoc wants to merge 1 commit into
chuanghiduoc wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Streaming
csv-parsecan mis-detect a UTF-16LE CRLF record delimiter when a chunk boundary splits the four-byte\r\nafter a closing quote. The parser permanently selects\r, then emits a bogus["\n"]record (silent corruption) or throwsCSV_INVALID_OPENING_QUOTEon valid input.parseSyncand a single-writeparsehandle the same bytes correctly.A single record (
'"b"\r\n') silently becomes[["b"], ["\n"]]. Two-byte and five-byte writes happen to work; one-, three- and nine-byte writes fail, so the result depends on chunk alignment.This is present on current HEAD and on published
[email protected]. It reproduces with explicitencoding: "utf16le"and with BOM auto-detection, and is not fixed by #537.Fix
When the record delimiter is still undiscovered,
__needMoreDatausedrecordDelimiterMaxLength(0) for the lookahead after a closing quote. It therefore proceeded with only part of the encoded CRLF, and auto-discovery accepted\rbecause the full\r\nwas not buffered.The fix computes an encoding-aware CRLF lookahead once and uses it for both the general delimiter check and the post-quote check. Auto-discovery now waits for the complete
\r\nand selects it over\r.Tests
Added two cases to
packages/csv-parse/test/option.encoding.tsfor one and two quoted records written one byte at a time.npx mocha 'test/**/*.{js,ts}' --ignore test/samples.js-> 562 passing, 2 pendingnpx tsc --noEmit-> cleantest/samples.jsis ignored locally because itsnew URL(".", import.meta.url).pathnameproduces an invalidD:\D:\...path on Windows; it runs normally on Linux CI.