Skip to content

fix: composite keys, computed/atomic UPDATE, batch UPDATE-as-DELETE, CHECK and JOIN/subquery ORDER BY - #518

Open
CastleSoft wants to merge 1 commit into
masterfrom
fix/composite-keys-update-atomicity-orderby
Open

CastleSoft wants to merge 1 commit into
masterfrom
fix/composite-keys-update-atomicity-orderby

Conversation

@CastleSoft

Copy link
Copy Markdown
Collaborator

Summary

Works through the four open items from #516/#517. The probes found four more defects on the way, including a data-loss bug in ExecuteBatchSQL. All of them are fixed here. Each fix has a regression test that was run red on the unfixed source and green after.

Items asked for

Item What was wrong Fix
Composite UNIQUE (a, b) Never checked on INSERT or UPDATE Enforced on INSERT, multi-row INSERT (atomic) and every UPDATE path. A hash index on the constraint's first column narrows the check. NULLs never conflict. The upsert conflict lookup now finds composite conflicts too.
Composite PRIMARY KEY (directory) Worse than recorded. A table-level PRIMARY KEY (a, b) was read as a column called PRIMARY (with CONSTRAINT pk …, a column called CONSTRAINT). That column became the key and was always NULL, so the table could hold one row. Inline PRIMARY KEY on several columns (EF Core EnsureCreated) kept only the last column. Enforced as SQLite enforces it in a rowid table: a UNIQUE constraint over all key columns, with the row id kept as the storage key. PRIMARY KEY (id) on one column makes that column the key. Named table constraints (CONSTRAINT x …) are parsed, including FOREIGN KEY.
Partly-failing multi-row UPDATE NOT NULL and CHECK were checked as each row was written, so a failure on row 2 kept row 1 changed. Confirmed with CHECK (n > id) and SET n = 2. Every matched row is validated before the first write. The NOT NULL check reads through the SET list without allocating; only tables with a CHECK build the updated row.
ORDER BY on JOIN / subquery queries The JOIN path read only the first token of the first key: LENGTH(p.name) DESC sorted by nothing, and c.score * -1 sorted ascending by score. It also ignored OFFSET, and LIMIT 0 returned every row. The AST path read each key as a bare name (UPPER(name) → "Unexpected trailing content '(name)'"). Its subquery executor used the first key only. The JOIN path now goes through the single-table path's SqlOrderBy and handles all three LIMIT forms. The AST parser keeps the expression text of each item, and both AST executors sort on every key.

Found on the way

Defect Fix
🔴 ExecuteBatchSQL ran some UPDATEs as DELETEs. When the UPDATE fast path declined a statement, the DELETE classifier tried next and accepted the canonical UPDATE shape. Example: a quoted SET column, SET "n" = 5, as linq2db and EF Core write it. The batch then deleted the rows it was asked to update, on columnar and page-based tables, and they stayed deleted after reopen. The scanner's result is checked: a statement with a SET column is never a DELETE, and the reverse.
UPDATE … SET n = n + 5 failed ("Invalid value 'n + 5'"). SET label = name on a TEXT column stored the text name. A SET value that reads a column or calls a function is computed per row from the row as it was before the statement, so SET a = b, b = a swaps. This works on directory and single-file tables and through ExecuteBatchSQL. Constant arithmetic keeps its old literal parse and is evaluated only when that parse fails (SET n = 2 * 3).
UPDATE t SET … without WHERE threw "Invalid UPDATE syntax". A SET literal containing where was cut at that word. The SET list and the WHERE clause are split at the first WHERE outside quotes and parentheses.
A table-level CHECK (…) rejected every row: it was stored with its keyword and could not be evaluated. A NULL value failed a CHECK, and <> was not an operator. CHECK uses SQLite semantics: a NULL result passes, and <> and functions work. Syntax the new evaluator doesn't cover falls back to the previous evaluator.

Deliberately unchanged

  • Single-file storage keeps its composite-key and UNIQUE behaviour. Changing it needs a file-format decision from you (known issues §10/§11).
  • Directory tables created before this change keep their old key. The other key columns were never recorded, so they can't be recovered; re-creating the table drops the false violations. Documented in §11.
  • On a TEXT column, SET name = 2 * 3 still stores the text 2 * 3. Constant expressions keep their old literal reading so existing unquoted values (dates, for example) don't change meaning. SQLite would store 6.

Tests

New test classes: CompositeKeyTests, UpdateSetExpressionTests, BatchUpdateClassificationTests, TableCheckConstraintTests, JoinAndSubqueryOrderByTests. They hold 57 test cases; 51 fail on the unfixed source. The six that already passed there are expected:

  • The two ReleasesTheKey cases check against over-enforcement, and the old code enforced nothing.
  • The four CheckFails…/NotNullFails… cases couldn't detect a partial update on the old code: it rejected the expression before writing any row. Atomicity itself is proven by Update_LiteralFailingAColumnCheckOnALaterRow_…, which fails on the old code with row 1 left at 2.
Suite Total Failed Skipped
Core 2449 0 6 (the documented single-file cases)
VectorSearch 255 0 0
EF Core 116 0 0
linq2DB 96 0 0
Regression canaries (AGENTS.md list) 51 0 0

SharpCoreDB.CI.slnf builds with 0 errors.

Performance

Write-path gate (--gate, Release): GATE PASSED, with every metric ok.

Metric Best-of-reps ratio
raw INSERT 1.02×
raw UPDATE 1.24×
raw DELETE 1.16×
default INSERT 1.04×
default UPDATE 1.20×

The limit is 1.50×; ratio = baseline ÷ current, so above 1 is slower. On the hot UPDATE path the new SET-value check is an allocation-free literal test. Validating rows before writing replaces the per-row validation that was already there.

…TE, CHECK and JOIN/subquery ORDER BY

- Composite PRIMARY KEY and multi-column UNIQUE (directory storage): table-level PRIMARY KEY (a, b) was read as a
  phantom column PRIMARY/CONSTRAINT that became an always-NULL key; PRIMARY KEY on several columns (EF Core
  EnsureCreated) kept only the last one; UNIQUE (a, b) was never checked. A composite key is now a UNIQUE constraint
  over its columns with the row id as storage key, enforced on INSERT, multi-row INSERT and every UPDATE path.
- ExecuteBatchSQL ran an UPDATE its fast path declined (e.g. a quoted SET column) as a DELETE of the same WHERE.
- UPDATE SET values that read a column or call a function are computed per row from the pre-statement row
  (directory, single-file, batch); every matched row is validated before the first write (statement atomicity).
- UPDATE without WHERE is accepted; a SET literal containing "where" is no longer cut.
- CHECK: table-level constraints no longer reject every row; NULL passes; <> and functions work.
- ORDER BY on the JOIN and AST/subquery paths reads every key and expression keys; JOIN honours LIMIT/OFFSET.
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant