Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automatic delimiter discovery crashes on non-ASCII field content and misreads UTF-16 input:
The frequency table contains candidates 0 through 126, as the delimiter_auto documentation specifies, but counting indexes it with every field character. Skip character codes outside that table so Unicode field data can be parsed while scoring the supported delimiters.
Decode the discovery sample using the configured encoding or a detected BOM, then encode the selected delimiter in that encoding. Preserve that delimiter when BOM handling renormalizes the parser options. Keep the original bytes for parsing, and wait for a complete BOM when its bytes arrive in separate chunks. A final UTF-16LE BOM without records is also handled.
Added 19 regression/control tests covering Unicode values, the 126/127 candidate boundary, custom scoring, UTF-16LE and its
ucs2alias, BOM detection, binary output, and chunk boundaries.Validation against
9fabcbewith Node.js 24.15.0:npm test -- --reporter dotpasses, 648 passed, 2 existing pending, including the sample tests and TypeScript check.npm test -- --reporter dot --ignore test/samples.jswith Git Bash as the command-local script shell. The existing sample collection path problem is tracked in fix: resolve sample file URLs as native paths #512.git diff --checkpassed.The other workspace packages were not tested. The build emits its existing circular-dependency warning.