Repository navigation
fix: linq2db adapter, scalar functions over columns, ORDER BY/LIMIT, PK-update and UNIQUE enforcement - #516
Merged
Conversation
…lture-invariant SQL literals linq2db adapter (SharpCoreDB.Functional.Linq2DB): - GetByIdAsync ignored its keys and returned the first row of the table; it now looks the row up by the entity's mapped primary key (composite keys, type conversion), throws on a key-count mismatch, and no longer swallows errors. - Every DI scope shared one live ADO.NET connection that was never disposed; the options now carry a connection factory, so each DataConnection owns its own. - AddSharpCoreDBLinq2Db(Action<DataOptions>) discarded the configuration (DataOptions is immutable); the overload now takes Func<DataOptions, DataOptions>. - The DataOptions constructor and UseSharpCoreDB replaced a caller's mapping schema; it is now combined, with the caller's schema taking priority. - Guid -> string used "N" while parameters store "D"; dates were parsed with the current culture (th-TH read 2026 back as 1483) and lost their UTC kind. - SharpCoreDBMappingSchema.ConfigureCustomMapping is obsolete (it mutated the process-wide Instance). Engine: the four copies of the value-to-SQL-literal switch (ParameterBinder, CompiledQueryExecutor, SqlFunctions.Quote, Table.FormatValue) formatted dates, offsets and numbers with the current culture. They now share SqlLiteralFormatter (SQLite's canonical yyyy-MM-dd HH:mm:ss.fff, invariant numbers). Tests: GetByIdAsyncTests, DependencyInjectionTests, MappingSchemaTests, SqlLiteralCultureTests - each red before, green after. Suites: core 2237/0, VectorSearch 255/0, EF Core 116/0, linq2DB 71/0.
…time, upper, length, CAST)
Only a SELECT without FROM evaluated functions. Over a table the SELECT list
returned the row unchanged when it held an expression, and every WHERE evaluator
read the left operand as a column name, so strftime('%Y', at) = '2026' matched no
row. linq2db's SQLite dialect wraps every DateTime comparison in strftime, so every
date filter through the adapter returned nothing, and UPDATE/DELETE with such a
WHERE changed nothing.
Fixed in all five places that had the defect - Table.EvaluateWhere (directory
mode), SingleFileTable.EvaluateSingleCondition and its compiled fast path,
SqlParser.EvaluateJoinWhere (UPDATE/DELETE/parallel scan), the single-table SELECT
list, and the AST executor - through one shared evaluator:
- SqlScalarExpression: literals, columns, SQLite scalar functions, CAST, ||,
arithmetic, and WHERE conditions. It only takes a condition with a computed
operand; plain column-op-literal conditions keep their existing paths.
- SqlDateTimeFunctions: date, time, datetime, julianday, unixepoch and strftime
over values, with SQLite's modifiers, month overflow, +HH:MM and NULL on bad
input. SELECT without FROM now uses it instead of its own text-based copies.
Tests: ScalarFunctionOverColumnTests (51 cases, directory and single-file; 48 red
before) and linq2db DateAndStringFilterTests (9 LINQ shapes; 9 red before).
Suites: core 2288/0, VectorSearch 255/0, EF Core 116/0, linq2DB 80/0, canaries 51/0.
…rop native SQLite dependency - TransactionAsync rolled back with the caller's token, so a cancelled token replaced the action's own error with "The operation was canceled"; it now rolls back with CancellationToken.None. New overload hands the action the context and the token. - Cancellation always surfaces as OperationCanceledException: a token cancelled during a Fin-returning call used to come back as Fail. The seven copies of that try/catch are one helper; the Option/Seq-throw, Fin-capture contract is documented. - InsertBatchAsync: new overload taking BulkCopyOptions (ProviderSpecific mapped to MultipleRows); a row count above Int32 is a Fail instead of a wrapped number. - AddSharpCoreDBLinq2Db also registers FunctionalLinq2DbContext (TryAdd, same lifetime, over the scope's connection). - Depend on Microsoft.Data.Sqlite.Core instead of the Microsoft.Data.Sqlite metapackage: linq2db only reflects over the managed types, and the metapackage shipped native e_sqlite3 for 29 RIDs into every consumer. Clean build verified: no e_sqlite3 in output, all tests pass. - README/package text: no stale version line, pinned install, or unverified zero-dependency / Native AOT claims; documents the error and transaction contract. - FindOneAsync uses FirstOrDefaultAsync; Seq reads materialise once; SharpCoreDBDataProvider.cs renamed after its type; stale SQLite cleanup removed from the test fixture. Tests: FunctionalContextContractTests (10) plus a guard case; the two behaviour defects were run red before the fix. Suites: core 2288/0, VectorSearch 255/0, EF Core 116/0, linq2DB 91/0, canaries 51/0.
…OT NULL; record constraint gaps
- ORDER BY on the single-table path read only its first token: a second key was
ignored (linq2db ThenBy, EF composite orderings came back unsorted), expression,
alias and qualified keys left rows unsorted, and text was ordered with the
machine's culture. New SqlOrderBy reads the whole list (directions, expressions,
aliases, ordinals, COLLATE, NULLS FIRST/LAST) and sorts stably with SQLite's rules
before OFFSET/LIMIT.
- LIMIT n OFFSET m and LIMIT m, n threw "Invalid LIMIT clause" (every linq2db
Skip/Take); a negative count now means no limit and LIMIT 0 returns no rows.
- ExecuteQueryFast keyed row values col0..colN, so any WHERE matched no row, or
every row when its compiled predicate could not parse it. It now evaluates by
column name with the table's own evaluator; the public CompileWhereClause no
longer accepts rows it cannot evaluate.
- Single-file storage never enforced NOT NULL and its constructor reset the flags;
INSERT, batch INSERT and UPDATE now enforce it, and a single-file multi-row INSERT
takes the atomic batch path.
- Shared evaluator compared two text values as numbers ('01' = '1'); fixed.
- SqlFunctions.EvaluateFunction routes SQLite's date family to SqlDateTimeFunctions
(legacy DateTime shape kept); StrFTime is culture-invariant.
- Removed dead code (unused scalar evaluators, the col0 predicate cache and
LruCache); coverage now measures the linq2db suite.
- Recorded in docs/sharpcoredb-known-issues.md 8-11, not fixed: PK-changing UPDATE
loses/duplicates a row, column UNIQUE unenforced, single-file ignores COLLATE,
single-file PK uniqueness and composite keys. Single-file PK enforcement was tried
and backed out: EF Core CompositeKey tests showed existing databases would reject
legitimate inserts.
Suites: core 2360/0 (5 skipped, the documented single-file cases), VectorSearch
255/0, EF Core 116/0, linq2DB 96/0, canaries 51/0. Coverage 47.34% (floor 18%).
…UE constraints (directory storage) Known issue 8: an UPDATE that moved a primary key onto an existing key was accepted; the PK index was re-pointed at the moved row and the original row became unreachable (silent data loss). Every UPDATE path - UpdateAffectedCount (SQL), UpdateMultipleCore (ExecuteBatchSQL, structured batch API) and UpdateByPrimaryKey - now validates new key values before any row is written, tracking what earlier operations of a batch claim and release; a SET of the key matching several rows is rejected too. Known issue 9: a UNIQUE column constraint was stored but never checked (the INSERT-time check was a stub). A single-column UNIQUE now gets an automatic unique hash index (as SQLite's sqlite_autoindex_*), registered at CREATE TABLE and on every open. One validator covers single-row INSERT, both batch critical sections and every UPDATE path; NULLs never conflict. On the way: a UNIQUE index was enforced only by a single-row INSERT into a columnar table (not page-based, not multi-row, not UPDATE), all three HashIndex add paths rejected a second NULL, and a table with a unique index gave up the atomic batched multi-row INSERT for a per-row loop. The batch core enforces unique indexes now, so CanUseBatchedInsert no longer excludes them; the test that pinned the old fallback is replaced by one asserting the batch path is used and still rejects a duplicate. Every check starts with an allocation-free test that returns at once when a table or statement has no key to protect. Not covered: single-file storage (its persisted Unique flag is also written for every inline PRIMARY KEY column, so it cannot tell a declared UNIQUE from an EF Core composite key's column - tried and backed out when the EF CompositeKey tests failed; known issues 10/11) and composite UNIQUE (a, b) on INSERT. Tests: PrimaryKeyUpdateTests, UniqueConstraintTests (columnar, page-based), red before. Suites: core 2392/0 (6 skipped), VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, canaries 51/0. Write-path gate: PASSED, every metric ok.
…-constraints fix: reject PK-changing UPDATEs onto an existing key and enforce UNIQUE constraints
…float equality (Sonar S1244)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
Fixes the defects and improvements (#1–#13) found in a review of the linq2db adapter, plus two engine defects the review surfaced. Every fix has a regression test that was run red against the previous code and green after.
1. Scalar functions over a column were never evaluated (engine) —
6ece045fOnly a
SELECTwithoutFROMevaluated functions. Over a table:SELECT upper(name) FROM tgave an ordinal reader theidcolumn;strftime('%Y', at) = '2026',length(name) = 4,upper(name) = 'X'matched no row, and UPDATE/DELETE with such a WHERE silently changed nothing.linq2db's SQLite dialect wraps every
DateTimecomparison instrftime(…), so every date filter through the adapter returned nothing (x.At == d,x.At > d,x.At.Year == 2026,DeleteWhereAsync(x => x.At < d)).Fixed in all five places with the defect —
Table.EvaluateWhere(directory mode),SingleFileTable.EvaluateSingleConditionand its compiled fast path,SqlParser.EvaluateJoinWhere(UPDATE/DELETE/parallel scan), the single-table SELECT list, and the AST executor — through one shared evaluator:SqlScalarExpression— literals, columns, SQLite scalar functions,CAST,||, arithmetic and WHERE conditions. It only takes a condition that has a computed operand; plaincolumn op literalconditions keep their existing evaluators, collations and fast paths.SqlDateTimeFunctions—date,time,datetime,julianday,unixepoch,strftimeover values, with SQLite's modifiers, month-overflow rule,±HH:MMhandling and NULL on invalid input.SELECTwithoutFROMnow uses it instead of its own text-based copies.2–6. linq2db adapter —
1b36000dGetByIdAsyncignored its keys and returned the first row; swallowed every exception intoNoneArgumentExceptionon key-count mismatch; errors propagateAddSharpCoreDBLinq2Db(Action<DataOptions>)discarded the configuration (DataOptionsis immutable)Func<DataOptions, DataOptions>; expression lambdas such aso => o.UseSharpCoreDB(cs)compile unchangedDataOptionsconstructor andUseSharpCoreDBreplaced the caller's mapping schema (fluent mappings silently lost)DataConnectionowns and disposes its own (database shared viaSharpCoreDBInstancePool)NvsDmismatch; dates parsed with the current culture (th-TH read 2026 as 1483), UTC kind lostDeverywhere; invariant parse withRoundtripKind;ConfigureCustomMappingmarked[Obsolete]7–13. linq2db adapter contracts, dependency and package text —
9769fc5eTransactionAsyncrolled back with the caller's token, so a cancelled token replaced the action's own error with "The operation was canceled" — now rolls back withCancellationToken.None. New overloadTransactionAsync((ctx, ct) => …)hands the action the context and token.OperationCanceledException(a token cancelled during aFincall used to come back asFail). Seven try/catch copies → one helper. Contract documented:Option/Seqthrow,Fincaptures, cancellation throws.InsertBatchAsync(entities, BulkCopyOptions)overload (ProviderSpecific→MultipleRows); a row count aboveint.MaxValueis aFailinstead of a wrapped negative number.AddSharpCoreDBLinq2Dbalso registersFunctionalLinq2DbContext(TryAdd, same lifetime, over the scope's connection).Microsoft.Data.Sqlite.Coreinstead of theMicrosoft.Data.Sqlitemetapackage — linq2db only reflects over the managed types; the metapackage shipped nativee_sqlite3for 29 RIDs (66 MB) into every consumer. Clean build verified: noe_sqlite3in the output, all tests pass. Adds aMicrosoft.Data.Sqlite.Coreentry toDirectory.Packages.propsat the existing 10.0.14.v1.9.5line or pinned--version 2.0.0.0, no unverified "zero-dependency" / Native AOT claims (aottag removed); documents the error and transaction contract. Version and release notes untouched.FindOneAsyncusesFirstOrDefaultAsync;Seqreads materialise once;SharpCoreDBDataProvider.csrenamed after its type; stale SQLite-wal/-shmcleanup removed from the test fixture.Remaining engine items, found and fixed on the way — commit 4
ORDER BY grp, name(linq2dbThenBy, EF composite orderings) came back unsorted; expression/alias/qualified keys unsorted; text ordered by machine cultureSqlOrderBy: every key + direction, expressions, aliases, ordinals,COLLATE,NULLS FIRST/LAST; stable SQLite sort before OFFSET/LIMITLIMIT n OFFSET mandLIMIT m, nthrew (every linq2dbSkip/Take);LIMIT 0returned all rowsLIMIT 0= noneExecuteQueryFastcol0…: any WHERE returned 0 rows, or all rows when unparseableCompileWhereClauseno longer accepts rows it cannot evaluate'01' = '1')SqlFunctionsDateTimeargs / threw;StrFTimeculture-dependentSqlDateTimeFunctions; legacy shape kept; invariantcol0predicate cache; linq2DB suite absent from coverageRecorded, not fixed (
docs/sharpcoredb-known-issues.md§8–§11, each reproduced): PK-changing UPDATE loses a row (directory) / duplicates it (single-file); columnUNIQUEunenforced on both modes; single-file ignoresCOLLATE; single-file enforces no PK uniqueness, and composite keys (EF Core declares one inlinePRIMARY KEYper key column) are modelled as their last column on both modes. Single-file PK enforcement was implemented and backed out: the EF CoreCompositeKey_*tests showed existing databases would reject legitimate inserts. §8/§9 need the write-path perf gate; §10/§11 need an owner decision on the column-entry format.Culture-invariant SQL literals (engine) —
1b36000dThe four copies of the value→SQL-literal switch (
ParameterBinder.FormatParameter,CompiledQueryExecutor.FormatParameterValue,SqlFunctions.Quote,Table.FormatValue) used the current culture: th-TH stored 2026 as 2569,DateTimeOffsetbecame'1/1/2026 12:30:15 PM +10:00', three copies dropped milliseconds, sv-SE wrote negative numbers with U+2212, comma-decimal cultures wrote1,5. All four now shareSqlLiteralFormatter.Behaviour changes for callers
AddSharpCoreDBLinq2Dbconfigure overload:Action<DataOptions>→Func<DataOptions, DataOptions>(the old form never worked).GetByIdAsyncthrows where it used to returnNonesilently.SharpCoreDBMappingSchema.ConfigureCustomMappingis obsolete.Fin-returning methods now throwOperationCanceledExceptionwhen cancelled mid-call instead of returningFail.e_sqlite3for its own Microsoft.Data.Sqlite usage must referenceMicrosoft.Data.Sqliteitself.FunctionalLinq2DbContextGuardTests: the null-action case now names its overload (twoTransactionAsyncoverloads accept a lambda).Tests
New, each proven red before the fix:
ScalarFunctionOverColumnTests— 51 cases, directory + single-file storage (48 red before)DateAndStringFilterTests(linq2db) — 9 LINQ shapes end to end (9/9 red before)SqlLiteralCultureTests— 72 cases, 4 cultures × 4 formatters (54 red before)GetByIdAsyncTests(9),DependencyInjectionTests(5),MappingSchemaTests(7),FunctionalContextContractTests(10, the two behaviour defects run red first)Results with
Category!=Debug&Category!=Manual&Category!=Performance:AGENTS.mdlist) 51 / 0SharpCoreDB.CI.slnfbuilds with 0 errorstools/measure-coverage.ps1, coverage-job filter): 46.64%, floor 18% — PASSNot covered
See Recorded, not fixed above (known issues §8–§11).
CHANGELOG entries are under
[Unreleased]. No version or package metadata changed.Update — now includes #517
8c43fe67): PK-changing UPDATE no longer loses a row (known issue §8) and UNIQUE constraints are enforced on directory storage (§9). Details in fix: reject PK-changing UPDATEs onto an existing key and enforce UNIQUE constraints #517. Folding it in meansmaster— and the NuGet publish — is pushed once, not twice.77be4b2a): the NUMERIC cast tested integrality withd == Math.Floor(d); it now usesdouble.IsInteger(d)— same semantics, no exact float equality. Core suite 2392 / 0 failed (6 skipped) after the change.