Skip to content

fix: linq2db adapter, scalar functions over columns, ORDER BY/LIMIT, PK-update and UNIQUE enforcement - #516

Merged
CastleSoft merged 7 commits into
masterfrom
fix/linq2db-adapter-and-scalar-functions
Nov 16, 2026
Merged

CastleSoft merged 7 commits into
masterfrom
fix/linq2db-adapter-and-scalar-functions

Conversation

@CastleSoft

@CastleSoft CastleSoft commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes the defects and improvements (#1–#13) found in a review of the linq2db adapter, plus two engine defects the review surfaced. Every fix has a regression test that was run red against the previous code and green after.

1. Scalar functions over a column were never evaluated (engine) — 6ece045f

Only a SELECT without FROM evaluated functions. Over a table:

  • the SELECT list returned the row unchanged when it held an expression — SELECT upper(name) FROM t gave an ordinal reader the id column;
  • every WHERE evaluator read the left operand as a column name, so strftime('%Y', at) = '2026', length(name) = 4, upper(name) = 'X' matched no row, and UPDATE/DELETE with such a WHERE silently changed nothing.

linq2db's SQLite dialect wraps every DateTime comparison in strftime(…), so every date filter through the adapter returned nothing (x.At == d, x.At > d, x.At.Year == 2026, DeleteWhereAsync(x => x.At < d)).

Fixed in all five places with the defect — Table.EvaluateWhere (directory mode), SingleFileTable.EvaluateSingleCondition and its compiled fast path, SqlParser.EvaluateJoinWhere (UPDATE/DELETE/parallel scan), the single-table SELECT list, and the AST executor — through one shared evaluator:

  • SqlScalarExpression — literals, columns, SQLite scalar functions, CAST, ||, arithmetic and WHERE conditions. It only takes a condition that has a computed operand; plain column op literal conditions keep their existing evaluators, collations and fast paths.
  • SqlDateTimeFunctions — date, time, datetime, julianday, unixepoch, strftime over values, with SQLite's modifiers, month-overflow rule, ±HH:MM handling and NULL on invalid input. SELECT without FROM now uses it instead of its own text-based copies.

2–6. linq2db adapter — 1b36000d

# Defect Fix
2 GetByIdAsync ignored its keys and returned the first row; swallowed every exception into None Looks up the mapped primary key (composite keys, numeric key conversion); ArgumentException on key-count mismatch; errors propagate
3 AddSharpCoreDBLinq2Db(Action<DataOptions>) discarded the configuration (DataOptions is immutable) Now Func<DataOptions, DataOptions>; expression lambdas such as o => o.UseSharpCoreDB(cs) compile unchanged
4 The DataOptions constructor and UseSharpCoreDB replaced the caller's mapping schema (fluent mappings silently lost) Schemas are combined; the caller's has priority
5 Every DI scope shared one live ADO.NET connection (and its transaction state), never disposed Options carry a connection factory; each DataConnection owns and disposes its own (database shared via SharpCoreDBInstancePool)
6 Guid N vs D mismatch; dates parsed with the current culture (th-TH read 2026 as 1483), UTC kind lost D everywhere; invariant parse with RoundtripKind; ConfigureCustomMapping marked [Obsolete]

7–13. linq2db adapter contracts, dependency and package text — 9769fc5e

# Change
7 TransactionAsync rolled back with the caller's token, so a cancelled token replaced the action's own error with "The operation was canceled" — now rolls back with CancellationToken.None. New overload TransactionAsync((ctx, ct) => …) hands the action the context and token.
8 Cancellation always surfaces as OperationCanceledException (a token cancelled during a Fin call used to come back as Fail). Seven try/catch copies → one helper. Contract documented: Option/Seq throw, Fin captures, cancellation throws.
9 InsertBatchAsync(entities, BulkCopyOptions) overload (ProviderSpecific → MultipleRows); a row count above int.MaxValue is a Fail instead of a wrapped negative number.
10 AddSharpCoreDBLinq2Db also registers FunctionalLinq2DbContext (TryAdd, same lifetime, over the scope's connection).
11 Depend on Microsoft.Data.Sqlite.Core instead of the Microsoft.Data.Sqlite metapackage — linq2db only reflects over the managed types; the metapackage shipped native e_sqlite3 for 29 RIDs (66 MB) into every consumer. Clean build verified: no e_sqlite3 in the output, all tests pass. Adds a Microsoft.Data.Sqlite.Core entry to Directory.Packages.props at the existing 10.0.14.
12 README/package text: no stale v1.9.5 line or pinned --version 2.0.0.0, no unverified "zero-dependency" / Native AOT claims (aot tag removed); documents the error and transaction contract. Version and release notes untouched.
13 FindOneAsync uses FirstOrDefaultAsync; Seq reads materialise once; SharpCoreDBDataProvider.cs renamed after its type; stale SQLite -wal/-shm cleanup removed from the test fixture.

Remaining engine items, found and fixed on the way — commit 4

Area Defect Fix
ORDER BY Single-table path read only the first token: ORDER BY grp, name (linq2db ThenBy, EF composite orderings) came back unsorted; expression/alias/qualified keys unsorted; text ordered by machine culture New SqlOrderBy: every key + direction, expressions, aliases, ordinals, COLLATE, NULLS FIRST/LAST; stable SQLite sort before OFFSET/LIMIT
LIMIT LIMIT n OFFSET m and LIMIT m, n threw (every linq2db Skip/Take); LIMIT 0 returned all rows SQLite's forms; negative = no limit; LIMIT 0 = none
ExecuteQueryFast Rows keyed col0…: any WHERE returned 0 rows, or all rows when unparseable Evaluated by column name with the table's own WHERE evaluator; public CompileWhereClause no longer accepts rows it cannot evaluate
Single-file NOT NULL Never enforced; constructor also reset the flags Enforced on INSERT, batch INSERT, UPDATE; multi-row INSERT now atomic (batch path)
Shared evaluator Two text values compared as numbers ('01' = '1') Text coerced only against a real number
SqlFunctions Date family only for DateTime args / threw; StrFTime culture-dependent Routed to SqlDateTimeFunctions; legacy shape kept; invariant
Dead code / coverage Unused scalar evaluators and col0 predicate cache; linq2DB suite absent from coverage Removed; coverage measures all four CI suites

Recorded, not fixed (docs/sharpcoredb-known-issues.md §8–§11, each reproduced): PK-changing UPDATE loses a row (directory) / duplicates it (single-file); column UNIQUE unenforced on both modes; single-file ignores COLLATE; single-file enforces no PK uniqueness, and composite keys (EF Core declares one inline PRIMARY KEY per key column) are modelled as their last column on both modes. Single-file PK enforcement was implemented and backed out: the EF Core CompositeKey_* tests showed existing databases would reject legitimate inserts. §8/§9 need the write-path perf gate; §10/§11 need an owner decision on the column-entry format.

Culture-invariant SQL literals (engine) — 1b36000d

The four copies of the value→SQL-literal switch (ParameterBinder.FormatParameter, CompiledQueryExecutor.FormatParameterValue, SqlFunctions.Quote, Table.FormatValue) used the current culture: th-TH stored 2026 as 2569, DateTimeOffset became '1/1/2026 12:30:15 PM +10:00', three copies dropped milliseconds, sv-SE wrote negative numbers with U+2212, comma-decimal cultures wrote 1,5. All four now share SqlLiteralFormatter.

Behaviour changes for callers

  • AddSharpCoreDBLinq2Db configure overload: Action<DataOptions> → Func<DataOptions, DataOptions> (the old form never worked).
  • GetByIdAsync throws where it used to return None silently.
  • SharpCoreDBMappingSchema.ConfigureCustomMapping is obsolete.
  • Fin-returning methods now throw OperationCanceledException when cancelled mid-call instead of returning Fail.
  • The package no longer brings native SQLite binaries; an app that relied on the adapter to supply e_sqlite3 for its own Microsoft.Data.Sqlite usage must reference Microsoft.Data.Sqlite itself.
  • FunctionalLinq2DbContextGuardTests: the null-action case now names its overload (two TransactionAsync overloads accept a lambda).

Tests

New, each proven red before the fix:

  • ScalarFunctionOverColumnTests — 51 cases, directory + single-file storage (48 red before)
  • DateAndStringFilterTests (linq2db) — 9 LINQ shapes end to end (9/9 red before)
  • SqlLiteralCultureTests — 72 cases, 4 cultures × 4 formatters (54 red before)
  • GetByIdAsyncTests (9), DependencyInjectionTests (5), MappingSchemaTests (7), FunctionalContextContractTests (10, the two behaviour defects run red first)

Results with Category!=Debug&Category!=Manual&Category!=Performance:

  • core 2360 / 0 failed (5 skipped: the documented single-file cases), VectorSearch 255 / 0, EF Core 116 / 0, linq2DB 96 / 0
  • regression canaries (AGENTS.md list) 51 / 0
  • SharpCoreDB.CI.slnf builds with 0 errors
  • coverage (tools/measure-coverage.ps1, coverage-job filter): 46.64%, floor 18% — PASS

Not covered

See Recorded, not fixed above (known issues §8–§11).

CHANGELOG entries are under [Unreleased]. No version or package metadata changed.

Update — now includes #517

…lture-invariant SQL literals

linq2db adapter (SharpCoreDB.Functional.Linq2DB):
- GetByIdAsync ignored its keys and returned the first row of the table; it now
  looks the row up by the entity's mapped primary key (composite keys, type
  conversion), throws on a key-count mismatch, and no longer swallows errors.
- Every DI scope shared one live ADO.NET connection that was never disposed; the
  options now carry a connection factory, so each DataConnection owns its own.
- AddSharpCoreDBLinq2Db(Action<DataOptions>) discarded the configuration
  (DataOptions is immutable); the overload now takes Func<DataOptions, DataOptions>.
- The DataOptions constructor and UseSharpCoreDB replaced a caller's mapping
  schema; it is now combined, with the caller's schema taking priority.
- Guid -> string used "N" while parameters store "D"; dates were parsed with the
  current culture (th-TH read 2026 back as 1483) and lost their UTC kind.
- SharpCoreDBMappingSchema.ConfigureCustomMapping is obsolete (it mutated the
  process-wide Instance).

Engine: the four copies of the value-to-SQL-literal switch (ParameterBinder,
CompiledQueryExecutor, SqlFunctions.Quote, Table.FormatValue) formatted dates,
offsets and numbers with the current culture. They now share SqlLiteralFormatter
(SQLite's canonical yyyy-MM-dd HH:mm:ss.fff, invariant numbers).

Tests: GetByIdAsyncTests, DependencyInjectionTests, MappingSchemaTests,
SqlLiteralCultureTests - each red before, green after. Suites: core 2237/0,
VectorSearch 255/0, EF Core 116/0, linq2DB 71/0.
…time, upper, length, CAST)

Only a SELECT without FROM evaluated functions. Over a table the SELECT list
returned the row unchanged when it held an expression, and every WHERE evaluator
read the left operand as a column name, so strftime('%Y', at) = '2026' matched no
row. linq2db's SQLite dialect wraps every DateTime comparison in strftime, so every
date filter through the adapter returned nothing, and UPDATE/DELETE with such a
WHERE changed nothing.

Fixed in all five places that had the defect - Table.EvaluateWhere (directory
mode), SingleFileTable.EvaluateSingleCondition and its compiled fast path,
SqlParser.EvaluateJoinWhere (UPDATE/DELETE/parallel scan), the single-table SELECT
list, and the AST executor - through one shared evaluator:

- SqlScalarExpression: literals, columns, SQLite scalar functions, CAST, ||,
  arithmetic, and WHERE conditions. It only takes a condition with a computed
  operand; plain column-op-literal conditions keep their existing paths.
- SqlDateTimeFunctions: date, time, datetime, julianday, unixepoch and strftime
  over values, with SQLite's modifiers, month overflow, +HH:MM and NULL on bad
  input. SELECT without FROM now uses it instead of its own text-based copies.

Tests: ScalarFunctionOverColumnTests (51 cases, directory and single-file; 48 red
before) and linq2db DateAndStringFilterTests (9 LINQ shapes; 9 red before).
Suites: core 2288/0, VectorSearch 255/0, EF Core 116/0, linq2DB 80/0, canaries 51/0.
…rop native SQLite dependency

- TransactionAsync rolled back with the caller's token, so a cancelled token
  replaced the action's own error with "The operation was canceled"; it now rolls
  back with CancellationToken.None. New overload hands the action the context and
  the token.
- Cancellation always surfaces as OperationCanceledException: a token cancelled
  during a Fin-returning call used to come back as Fail. The seven copies of that
  try/catch are one helper; the Option/Seq-throw, Fin-capture contract is documented.
- InsertBatchAsync: new overload taking BulkCopyOptions (ProviderSpecific mapped to
  MultipleRows); a row count above Int32 is a Fail instead of a wrapped number.
- AddSharpCoreDBLinq2Db also registers FunctionalLinq2DbContext (TryAdd, same
  lifetime, over the scope's connection).
- Depend on Microsoft.Data.Sqlite.Core instead of the Microsoft.Data.Sqlite
  metapackage: linq2db only reflects over the managed types, and the metapackage
  shipped native e_sqlite3 for 29 RIDs into every consumer. Clean build verified:
  no e_sqlite3 in output, all tests pass.
- README/package text: no stale version line, pinned install, or unverified
  zero-dependency / Native AOT claims; documents the error and transaction contract.
- FindOneAsync uses FirstOrDefaultAsync; Seq reads materialise once;
  SharpCoreDBDataProvider.cs renamed after its type; stale SQLite cleanup removed
  from the test fixture.

Tests: FunctionalContextContractTests (10) plus a guard case; the two behaviour
defects were run red before the fix. Suites: core 2288/0, VectorSearch 255/0,
EF Core 116/0, linq2DB 91/0, canaries 51/0.
@CastleSoft CastleSoft changed the title fix: linq2db adapter defects, culture-invariant SQL literals, and scalar functions over columns fix: linq2db adapter defects and contracts, culture-invariant SQL literals, and scalar functions over columns Nov 16, 2026
…OT NULL; record constraint gaps

- ORDER BY on the single-table path read only its first token: a second key was
  ignored (linq2db ThenBy, EF composite orderings came back unsorted), expression,
  alias and qualified keys left rows unsorted, and text was ordered with the
  machine's culture. New SqlOrderBy reads the whole list (directions, expressions,
  aliases, ordinals, COLLATE, NULLS FIRST/LAST) and sorts stably with SQLite's rules
  before OFFSET/LIMIT.
- LIMIT n OFFSET m and LIMIT m, n threw "Invalid LIMIT clause" (every linq2db
  Skip/Take); a negative count now means no limit and LIMIT 0 returns no rows.
- ExecuteQueryFast keyed row values col0..colN, so any WHERE matched no row, or
  every row when its compiled predicate could not parse it. It now evaluates by
  column name with the table's own evaluator; the public CompileWhereClause no
  longer accepts rows it cannot evaluate.
- Single-file storage never enforced NOT NULL and its constructor reset the flags;
  INSERT, batch INSERT and UPDATE now enforce it, and a single-file multi-row INSERT
  takes the atomic batch path.
- Shared evaluator compared two text values as numbers ('01' = '1'); fixed.
- SqlFunctions.EvaluateFunction routes SQLite's date family to SqlDateTimeFunctions
  (legacy DateTime shape kept); StrFTime is culture-invariant.
- Removed dead code (unused scalar evaluators, the col0 predicate cache and
  LruCache); coverage now measures the linq2db suite.
- Recorded in docs/sharpcoredb-known-issues.md 8-11, not fixed: PK-changing UPDATE
  loses/duplicates a row, column UNIQUE unenforced, single-file ignores COLLATE,
  single-file PK uniqueness and composite keys. Single-file PK enforcement was tried
  and backed out: EF Core CompositeKey tests showed existing databases would reject
  legitimate inserts.

Suites: core 2360/0 (5 skipped, the documented single-file cases), VectorSearch
255/0, EF Core 116/0, linq2DB 96/0, canaries 51/0. Coverage 47.34% (floor 18%).
…UE constraints (directory storage)

Known issue 8: an UPDATE that moved a primary key onto an existing key was
accepted; the PK index was re-pointed at the moved row and the original row became
unreachable (silent data loss). Every UPDATE path - UpdateAffectedCount (SQL),
UpdateMultipleCore (ExecuteBatchSQL, structured batch API) and UpdateByPrimaryKey -
now validates new key values before any row is written, tracking what earlier
operations of a batch claim and release; a SET of the key matching several rows is
rejected too.

Known issue 9: a UNIQUE column constraint was stored but never checked (the
INSERT-time check was a stub). A single-column UNIQUE now gets an automatic unique
hash index (as SQLite's sqlite_autoindex_*), registered at CREATE TABLE and on
every open. One validator covers single-row INSERT, both batch critical sections
and every UPDATE path; NULLs never conflict.

On the way: a UNIQUE index was enforced only by a single-row INSERT into a columnar
table (not page-based, not multi-row, not UPDATE), all three HashIndex add paths
rejected a second NULL, and a table with a unique index gave up the atomic batched
multi-row INSERT for a per-row loop. The batch core enforces unique indexes now, so
CanUseBatchedInsert no longer excludes them; the test that pinned the old fallback
is replaced by one asserting the batch path is used and still rejects a duplicate.

Every check starts with an allocation-free test that returns at once when a table
or statement has no key to protect.

Not covered: single-file storage (its persisted Unique flag is also written for
every inline PRIMARY KEY column, so it cannot tell a declared UNIQUE from an EF Core
composite key's column - tried and backed out when the EF CompositeKey tests failed;
known issues 10/11) and composite UNIQUE (a, b) on INSERT.

Tests: PrimaryKeyUpdateTests, UniqueConstraintTests (columnar, page-based), red
before. Suites: core 2392/0 (6 skipped), VectorSearch 255/0, EF Core 116/0,
linq2DB 96/0, canaries 51/0. Write-path gate: PASSED, every metric ok.
…-constraints

fix: reject PK-changing UPDATEs onto an existing key and enforce UNIQUE constraints
@CastleSoft CastleSoft changed the title fix: linq2db adapter defects and contracts, culture-invariant SQL literals, and scalar functions over columns fix: linq2db adapter, scalar functions over columns, ORDER BY/LIMIT, PK-update and UNIQUE enforcement Nov 16, 2026
@CastleSoft
CastleSoft merged commit 4419551 into master Nov 16, 2026
13 checks passed
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant