Skip to content

fix(v18): pin libpg_query with checked scan allocations - #188

Open
pyramation wants to merge 1 commit into
devin/1792898986-fingerprint-normalize-errorsfrom
fix/v18-pin-scan-oom
Open

pyramation wants to merge 1 commit into
devin/1792898986-fingerprint-normalize-errorsfrom
fix/v18-pin-scan-oom

Conversation

@pyramation

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #183. Fixes the ~50 MB scan abort.

Cause. pg_query_scan in the fork never checked its mallocs. At about 50 MB (18.6M tokens) one of them returns NULL inside the 1 GiB heap, and the token protobuf gets packed at address 0. That overwrites static data: the call aborts with Assertion failed: <protobuf bytes> (several MB written to stderr), and the next sync call in the same tick fails too.

Changes.

  • Pin: versions/18 (Makefile LIBPG_QUERY_COMMIT and x-publish.libpgQueryCommit) moves from b0f5f33 to 8076f09. That is the head of fix/scan-out-of-memory-18, Report out of memory from pg_query_scan instead of writing through NULL libpg_query#12, which checks each allocation, frees what was allocated and returns result.error = "out of memory".
  • Wrapper: readDetailedResult (templates/full/index.ts) maps libpg_query's plain "out of memory" error to the same out of WASM memory error that a NULL result already throws. So scan reports one message no matter where the allocation failed.
scanSync("SELECT a, b FROM t WHERE c = 1;\n".repeat(1690000))
// before: Aborted(Assertion failed: …binary…); next scanSync("SELECT 1") throws
// after:  Error: libpg_query returned no result: out of WASM memory …; next sync and async calls return 2 tokens

Regression test

New test in versions/18/test/scan.test.js: "throws out of WASM memory at 50 MB, then sync and async calls work". It runs in a child process, because before this fix the abort corrupts the instance.

Merge order

  1. Dan merges Report out of memory from pg_query_scan instead of writing through NULL libpg_query#12 into 18-constructive with a merge commit, not squash. The Makefile clones 18-constructive and checks out 8076f09, so that commit has to stay reachable on the branch.
  2. Re-run this PR's CI. Until step 1, the v18 WASM job fails with fatal: reference is not a tree, as fix: stack-depth guard and shadow-stack restore for deep recursion (follow-up to #179) #180 did.
  3. Merge fix(v18): throw SqlError from fingerprint/normalize/scan on every libpg_query error #183, then this PR.

Link to Devin session: https://app.devin.ai/sessions/68876ce060664609ab6b63c8bed91e78
Open in Devin Desktop: https://app.devin.ai/desktop/session/68876ce060664609ab6b63c8bed91e78?variant=devin
Requested by: @pyramation

@devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@tenki-reviewer

tenki-reviewer Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Review complete. 🟡 1 medium

📍 Findings outside the diff (1) — 🟡 1 medium — defects on lines GitHub can't attach comments to

🟡 Medium — parse and parseSync miss the new out of memory remap · index.ts:308–326 · unchanged line

// versions/18/src/index.ts
308	    if (errorPtr) {
309	      // Read PgQueryError struct
310	      const messagePtr = wasmModule.getValue(errorPtr, 'i32');
311	      const funcnamePtr = wasmModule.getValue(errorPtr + 4, 'i32');
312	      const filenamePtr = wasmModule.getValue(errorPtr + 8, 'i32');
313	      const lineno = wasmModule.getValue(errorPtr + 12, 'i32');
314	      const cursorpos = wasmModule.getValue(errorPtr + 16, 'i32');
315	      
316	      const message = messagePtr ? wasmModule.UTF8ToString(messagePtr) : 'Unknown error';
317	      const funcname = funcnamePtr ? wasmModule.UTF8ToString(funcnamePtr) : undefined;
318	      const filename = filenamePtr ? wasmModule.UTF8ToString(filenamePtr) : undefined;
319	      

The new message === 'out of memory' heap-limit remap was added only in readDetailedResult (versions/18/src/index.ts:266), so scan/fingerprint/normalize report malloc failures as the descriptive WASM heap-limit error. parse and parseSync throw from their own errorPtr branch and still surface the same malloc failure as a bare SqlError('out of memory') with line/position metadata. Callers that branch on the heap-limit message (e.g. to retry in smaller pieces) get inconsistent behavior across the APIs.

The same gap exists in templates/full/index.ts at lines 301-319.


This PR upgrades the binding to libpg-query v18 and improves how malloc failures are reported: when the parser error message is exactly out of memory, the new code throws a descriptive WASM heap-limit error instead of a bare SQL error, and a regression test exercises the OOM path in a child process.

Files Change
templates/full/index.ts, versions/18/src/index.ts Add out of memory message remap in readDetailedResult (and related plumbing) so heap-limit failures surface a dedicated error.
versions/18/Makefile, versions/18/package.json Pin/build the v18 libpg-query artifact and bump package metadata.
versions/18/test/scan.test.js New child-process regression test covering the OOM error path.

Reviewed commit: b30efdb

@devin-ai-integration

Copy link
Copy Markdown

Folded into #190

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant