Skip to content

fix(csv-parse): handle Unicode delimiter discovery - #540

Open
ice-pring wants to merge 2 commits into
adaltas:masterfrom
ice-pring:fix/discover-delimiter-with-unicode-fields
Open

ice-pring wants to merge 2 commits into
adaltas:masterfrom
ice-pring:fix/discover-delimiter-with-unicode-fields

Conversation

@ice-pring

@ice-pring ice-pring commented Oct 23, 2026 •

Copy link
Copy Markdown

Automatic delimiter discovery crashes on non-ASCII field content and misreads UTF-16 input:

parse("é,東京,😀\nß,北京,🚀", { delimiter_auto: true });
// TypeError: Cannot read properties of undefined (reading 'lines')

The frequency table contains candidates 0 through 126, as the delimiter_auto documentation specifies, but counting indexes it with every field character. Skip character codes outside that table so Unicode field data can be parsed while scoring the supported delimiters.

Decode the discovery sample using the configured encoding or a detected BOM, then encode the selected delimiter in that encoding. Preserve that delimiter when BOM handling renormalizes the parser options. Keep the original bytes for parsing, and wait for a complete BOM when its bytes arrive in separate chunks. A final UTF-16LE BOM without records is also handled.

Added 19 regression/control tests covering Unicode values, the 126/127 candidate boundary, custom scoring, UTF-16LE and its ucs2 alias, BOM detection, binary output, and chunk boundaries.

Validation against 9fabcbe with Node.js 24.15.0:

  • Ubuntu 24.04 under WSL: the complete csv-parse npm test -- --reporter dot passes, 648 passed, 2 existing pending, including the sample tests and TypeScript check.
  • Windows: TypeScript and parser tests pass, 579 passed, 2 existing pending, using npm test -- --reporter dot --ignore test/samples.js with Git Bash as the command-local script shell. The existing sample collection path problem is tracked in fix: resolve sample file URLs as native paths #512.
  • Before the original Unicode fix, focused tests have 7 failures. The added encoding tests have 9 failures against the previous API implementation; all pass afterward.
  • Package build passed. Fourteen checks passed across source, rebuilt CommonJS and rebuilt browser ESM sync bundles (the browser bundle was executed in Node).
  • Changed-file ESLint/Prettier and git diff --check passed.

The other workspace packages were not tested. The build emits its existing circular-dependency warning.

@ice-pring ice-pring changed the title fix(csv-parse): discover delimiters in Unicode field data fix(csv-parse): handle Unicode delimiter discovery Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant