Add source positional information to the AST

[kyleconroy]

Every AST node now carries source position information — StartOffset, FragmentLength, StartLine, and StartColumn — matching what Microsoft's ScriptDom emits for the same input.

Approach

ScriptDom already computes positions for every TSqlFragment; TsqlAstParser was explicitly discarding them via a SkipProperties set. Removing those four properties from that set gave us authoritative expected values, so no golden file was hand-authored. All 957 regenerable ast.json files were regenerated with the official parser using the loop documented in CLAUDE.md.

Regeneration is reproducible: re-running it over the committed goldens produces zero diffs, and git diff of parser/testdata between the regeneration commit and the branch tip is empty — every subsequent commit touched only Go code.

Changes

TsqlAstParser/Program.cs — emits the four position properties for fragments with real tokens (StartOffset >= 0); still skips ScriptTokenStream and token indices, which would bloat the JSON with ScriptDom-internal numbering.

ast/fragment.go (new) — a Fragment struct embedded in all 860 node structs, giving every node uniform position fields and accessors. A Pin() mechanism marks spans that reproduce ScriptDom quirks narrower than a node's full token extent, so enclosing parse functions don't widen them.

parser/source_map.go (new) — converts the lexer's byte offsets into ScriptDom's coordinate system: UTF-16 code-unit offsets and 1-based line/column. This is load-bearing rather than theoretical — 149 test inputs contain non-ASCII text where Go byte offsets and .NET string indices diverge. Lone \r starts a new line, matching ScriptDom.

parser/parser.go — span-recording helpers (spanned, spanFrom, tokSpan, spanTokens, spanFromChild, unspan, trimTrailingSemicolon, and others) plus token-end tracking.

parser/parse_*.go, parser/marshal.go — parse functions record spans; the marshaler emits them and derives spans for wrapper nodes synthesized during marshaling.

parser/parser_test.go — the 75 handwritten PhaseOne_* goldens can't be regenerated (the official parser rejects their intentionally partial SQL), so tests whose golden lacks positions are compared with position fields stripped from both sides.

Positional quirks

Most of the work was matching ScriptDom's spans, which are not simply "first token through last token":

• Statements include their terminating semicolon, but a CREATE VIEW body's SELECT does not
• EXISTS predicates exclude the EXISTS keyword; ORDER bulk-insert options exclude UNIQUE
• BUCKET_COUNT index hints span only their value; COMPRESSION_DELAY excludes its time unit
• Index definitions exclude trailing CLUSTERED/COLUMNSTORE keywords
• Some synthesized nodes (defaulted options, normalized keyword identifiers) carry no position at all

Beyond positions

Reaching exact golden parity required a few output corrections worth reviewing separately, since they change AST output beyond position fields:

• SQLCMD $(var) database names now emit SqlCommandIdentifier
• ALTER ENDPOINT and CREATE MESSAGE TYPE always emit their defaulted enum fields
• CONNECTION is treated as a user data type rather than a SQL data type

These also fixed 5 pre-existing PhaseOne test failures.

Testing

• go test ./parser/ passes (fresh run, 1023 tests)
• All 882 tests with positional goldens match exactly, including position fields
• All 75 legacy PhaseOne tests pass — 5 were failing before this branch
• Spot-checked outputs against goldens as parsed JSON: identical, including the Unicode test that exercises UTF-16 offset conversion

🤖 Generated with Claude Code

---

Generated by Claude Code