Skip to content

fix(scan): escape control characters and throw SqlError on lexer errors - #185

Closed
pyramation wants to merge 1 commit into
mainfrom
devin/1792899503-scan-json-escape
Closed

pyramation wants to merge 1 commit into
mainfrom
devin/1792899503-scan-json-escape

Conversation

@pyramation

Copy link
Copy Markdown
Collaborator

Summary

scan/scanSync (v18) threw a raw JSON SyntaxError in two cases, in both the published 18.2.1 and 18.2.0:

  • Valid SQL containing control characters (SELECT 'a\x01b', \b, \x1f): build_scan_json only escaped " \ \n \r \t, so the JSON was invalid. Other bytes < 0x20 are now written as \u00XX, and the buffer budget goes from 2 to 6 bytes per input byte.
  • Input the lexer rejects (SELECT 'unterminated, SELECT $$x, SELECT /* x, ''): wasm_scan returns the error text, but the wrapper only checked for syntax error/ERROR prefixes, so the error text went to JSON.parse. Any non-JSON result now throws SqlError:
    if (!resultStr.startsWith('{')) throw new SqlError(resultStr);

Edits are in templates/full/; versions/18/src/ was regenerated with copy:templates.

Evidence:

  • The 2 new tests in versions/18/test/scan.test.js fail against the published 18.2.1 wasm/ and pass on this build. The full v18 suite passes (145/145).
  • I ran 200k corpus inputs (half of them mutated) through this build and through published 18.2.1. 147,027 gave byte-identical scan JSON on both, with no diffs. 13,049 inputs that hit SyntaxError before now scan, and 39,212 now throw SqlError instead of SyntaxError.

Link to Devin session: https://app.devin.ai/sessions/b32426e30a71409594f003cf94c9e560
Open in Devin Desktop: https://app.devin.ai/desktop/session/b32426e30a71409594f003cf94c9e560?variant=devin
Requested by: @pyramation

@devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@tenki-reviewer

tenki-reviewer Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Review complete. No blocking issues — approved ✅; 1 nitpick below.

🧹 Nitpicks (1) — 🟢 1 low
  • 🟢 scan/scanSync now throw SqlError instead of Error (index.ts:547) — scan and scanSync previously threw plain Error on lexer failure; they now throw SqlError, whose constructor sets name = 'SqlError' (versions/18/src/index.ts:547-549, and the same in scanSync and templates/full/index.ts).

This PR hardens the parse-result JSON pipeline: the C WASM wrappers now escape control characters (0x00-0x1f) as backslash-u escape sequences before the result string is parsed in JS, and scan/scanSync replace the prefix-matching error heuristic with a startsWith('{') check that throws SqlError on parse failure. New tests cover control-character escaping round-trips and the error behavior.

Files Change
templates/full/wasm_wrapper.c, versions/18/src/wasm_wrapper.c Escape control characters in token text as \\u00XX into an over-allocated buffer before returning the JSON result string.
templates/full/index.ts, versions/18/src/index.ts Change scan error detection to a startsWith('{') JSON check and throw SqlError with the original message.
versions/18/test/scan.test.js Add tests asserting control-character escaping round-trips and the new SqlError failure path.

Reviewed commit: 0e33eb7

@devin-ai-integration

Copy link
Copy Markdown

Closing in favor of #183, which consolidates the v18 scan/fingerprint/normalize error fixes, including control-character escaping in build_scan_json and lexer errors as SqlError.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant