diff --git a/Directory.Packages.props b/Directory.Packages.props index 14ec3b00..ce908d51 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -70,6 +70,7 @@ + diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index af562111..86ba22aa 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -10,9 +10,50 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 > **Where this bucket ends.** `2.1.4` was cut from this line's `HEAD` on 2026-11-13, so everything below the > `[2.1.4]` heading shipped in that patch. New entries start here, above `[2.1.4]`. +### Added + +- **linq2db adapter: `TransactionAsync((context, token) => …)`**, an overload that hands the action the `FunctionalLinq2DbContext` and the cancellation token, so the work inside is visibly bound to the transaction's connection. The original `TransactionAsync(() => …)` delegates to it. +- **linq2db adapter: `InsertBatchAsync(entities, BulkCopyOptions)`** takes the caller's linq2db bulk-copy options (batch size, parameters, row-copied notifications) instead of a fixed `MaxBatchSize = 1000`. `BulkCopyType.ProviderSpecific` is mapped to `MultipleRows`, because the provider-specific path needs a native SQLite connection the adapter never has; the caller's options instance is not modified. +- **linq2db adapter: `AddSharpCoreDBLinq2Db` also registers `FunctionalLinq2DbContext`**, with the same lifetime and over the scope's own connection. A registration the caller already made is kept (`TryAdd`). Previously every caller wired the context by hand — often as a singleton over one shared connection. + +### Changed + +- **linq2db adapter: the package depends on `Microsoft.Data.Sqlite.Core` instead of the `Microsoft.Data.Sqlite` metapackage.** linq2db's SQLite dialect only reflects over the managed Microsoft.Data.Sqlite types when it builds its provider; every command runs on a `SharpCoreDBConnection`. The metapackage also pulled in `SQLitePCLRaw.bundle_e_sqlite3` — the native SQLite library for 29 runtime identifiers (66 MB in the package cache) — into every consumer, for nothing. Verified with a clean build of the adapter's test suite on `Microsoft.Data.Sqlite.Core`: no `e_sqlite3` binary in the output, and all 91 tests pass. `Directory.Packages.props` gains a `Microsoft.Data.Sqlite.Core` entry at the same 10.0.14 as `Microsoft.Data.Sqlite`. +- **linq2db adapter: cancellation always surfaces as `OperationCanceledException`.** The adapter's error contract is now written down and uniform: methods returning `Option`/`Seq` throw on a failure, methods returning `Fin` capture it as `Fail`, and cancellation throws whether the token was cancelled before the call or during it. Up to 2.1.4 every `Fin` method caught everything, so a token cancelled before the call threw but one cancelled during the call came back as a `Fail`. The seven copies of that try/catch are now one helper. +- **linq2db adapter: package text.** The README no longer carries a stale `v1.9.5` version line or a pinned `--version 2.0.0.0` install, and neither it nor the package description claims "zero-dependency" or Native AOT, which were never verified (the `aot` package tag is removed). The README now documents the error/cancellation/transaction contract and the batch options. No version or release-notes metadata changed. + +- **Coverage measures the linq2db adapter suite.** `tools/measure-coverage.ps1` defaults to the four suites CI runs; `SharpCoreDB.Functional.Linq2DB.Tests` was left out, so the adapter assembly never entered the coverage figure. +- **Removed dead code.** `SqlParser.EvaluateScalarFunction` / `EvaluateFunctionCall` (no caller — the live evaluators are the AST executor's and `SqlScalarExpression`), and `Database.GetOrCompileWhereClause` / `ClearWhereClauseCache` / `LruCache`, which served only the `ExecuteQueryFast` predicate path replaced below. + +### Deprecated + +- **`SharpCoreDBMappingSchema.ConfigureCustomMapping`** (linq2db adapter) is `[Obsolete]`: it changed the + process-wide `Instance` that every connection shares. Register converters on your own `MappingSchema` and add it + with `DataOptions.UseAdditionalMappingSchema` — since this release the adapter keeps a caller's schema and gives it + priority (see *Fixed*). + ### Fixed - **`ExecuteBatchSQL` reordered a mixed batch, so `[DELETE …, INSERT …]` lost the row it had just inserted.** The dispatcher classifies a batch into INSERT / UPDATE / DELETE buckets and runs the buckets in a fixed order (inserts, then updates, then deletes) — a pure win for a homogeneous batch, and a behaviour change for a mixed one, because SQLite applies a batch's statements in the order it was given. Measured on a two-statement batch of exactly the shape `SharpCoreDbProjectionCheckpointStore.SaveCheckpointAsync` uses: `DELETE FROM t WHERE k = 'a'` followed by `INSERT INTO t VALUES ('a', 20)` left the table **empty** (the INSERT ran first and the DELETE removed it), while the same two statements issued one at a time left the single row `seq = 20`. Through the store the symptom is worse than an empty table: overwriting an existing checkpoint kept returning the **old** sequence, so a projection would re-process events from a checkpoint that never advanced. The fix keeps the grouped fast path for a homogeneous batch — all INSERTs, all UPDATEs or all DELETEs, which is what the bulk-insert benchmarks use — and applies a mixed batch statement by statement in the caller's order, on both the sync and the async dispatcher (they share the classifier, so they now share the ordering contract). Pinned by `tests/SharpCoreDB.Tests/BatchStatementOrderTests.cs` (5 cases: delete-then-insert, insert-then-delete, update-then-delete, all-inserts fast path, and the async twin; 2 of them fail with the guard disabled) and by `SharpCoreDbProjectionCheckpointStoreBehaviorTests.SaveCheckpointAsync_WhenCalledTwice_KeepsLatestSequence` at the store layer. Core suite 2270/2270, projections suite 44/44. Found by mutation testing: the store's overwrite path had one test asserting only `GlobalSequence`, so a green suite covered a batch application that deleted the checkpoint it was writing. +- **linq2db adapter: `FunctionalLinq2DbContext.GetByIdAsync` ignored its keys and returned the first row of the table.** The method ran `GetTable().Take(1)` and never read `keyValues`, so `GetByIdAsync([2])` and `GetByIdAsync([999])` both answered with row 1 — a lookup for one user could hand back another — and it caught every exception into `None`, so a failing query read as "not found". It now reads the entity's mapped primary key from the mapping schema (`GetEntityDescriptor`, columns ordered by `PrimaryKeyOrder`), builds `row.Key1 == k1 && …` with the values bound as parameters (one cached query per entity type, not per key), converts a value of another numeric type to the member's type (`[3]` for a `long` key), and throws `ArgumentException` on a key-count mismatch or an unconvertible value and `InvalidOperationException` when the entity has no mapped key; query errors propagate as they do from `FindOneAsync`. Pinned by `GetByIdAsyncTests` (9 cases, all red before — including composite keys — and green after). +- **linq2db adapter: every DI scope shared one ADO.NET connection, the `Action` registration could not work, and a caller's mapping schema was discarded.** Three defects in how the adapter builds `DataOptions`, each reproduced before the fix: (1) `CreateOptions`/`UseSharpCoreDB` captured one live `SharpCoreDBConnection`, so `AddSharpCoreDBLinq2Db(connectionString)` gave every scope — every web request — the *same* connection and its transaction state for the application's lifetime, and nothing disposed it (linq2db only disposes a connection it created); the options now carry a connection **factory**, so each `SharpCoreDBDataConnection` opens and disposes its own connection, sharing the database through `SharpCoreDBInstancePool`. (2) `AddSharpCoreDBLinq2Db(Action)` discarded what the callback built — `DataOptions` is immutable — and resolving the connection threw *"Configuration string is not provided"*; the overload now takes `Func` (an expression lambda such as `o => o.UseSharpCoreDB(cs)` compiles unchanged). (3) The `DataOptions` constructor and `UseSharpCoreDB` applied the SharpCoreDB schema with `UseMappingSchema`, which linq2db documents as *"replaces all previous registrations"*: a fluent mapping of an entity to `custom_table` silently queried `[Entity]`. The SharpCoreDB schema is now combined with the caller's, once, with the caller's schema first so its converters win. Pinned by `DependencyInjectionTests` (5 cases) and the first four `MappingSchemaTests`, all red before and green after; the adapter README gains a *Dependency injection, connections and mapping schemas* section. +- **Values rendered into SQL text were formatted with the current culture — a date stored under th-TH read back as year 1483, and a `DateTimeOffset` was stored as `'1/1/2026 12:30:15 PM +10:00'`.** Found through the linq2db adapter and fixed at both layers. In the adapter's `SharpCoreDBMappingSchema`, stored dates were parsed with `DateTime.Parse(str)`: under a Buddhist-calendar culture (th-TH) `2026-01-01` read back as 1483, and a round-trip `…Z` string came back as local time; dates are now parsed with the invariant culture and `RoundtripKind`, and `Guid → string` uses the same hyphenated `D` form as the Guid parameter (it used `N`, which never matched a stored value). In the engine, the four copies of the value-to-SQL-literal switch — `ParameterBinder.FormatParameter` (the ADO.NET binder), `CompiledQueryExecutor.FormatParameterValue`, `SqlFunctions.Quote` and `Table.FormatValue` — formatted dates with the current culture's calendar (th-TH stored 2026 as 2569), rendered `DateTimeOffset`/`DateOnly`/`TimeOnly` through their culture-dependent `ToString()` (without milliseconds), three of them dropped a `DateTime`'s milliseconds, sv-SE wrote a negative integer with U+2212 instead of `-`, and `Table.FormatValue` wrote a `double` as `1,5` under a comma-decimal culture. All four now go through one `SqlLiteralFormatter`: SQLite's canonical `yyyy-MM-dd HH:mm:ss.fff` (plus `±HH:MM` for an offset), invariant numbers. Pinned by `SqlLiteralCultureTests` (72 cases across en-US, th-TH, sv-SE and de-DE and all four formatters; 54 failed before, 72 pass after) and the date cases in `MappingSchemaTests`. Suites: core **2237 / 0 failed**, VectorSearch 255 / 0, EF Core 116 / 0, linq2DB 71 / 0, regression canaries 51 / 0; `SharpCoreDB.CI.slnf` builds with 0 errors. +- **A scalar function applied to a column was never evaluated, so every DateTime filter written through linq2db matched no row.** Only a SELECT without FROM evaluated functions. Over a table, the single-table SELECT path returned the scanned row unchanged when the list held an expression — `SELECT upper(name) FROM t` handed an ordinal-based reader the row's first column (its `id`) — and every WHERE evaluator read the left operand as a column name, so `strftime('%Y', at) = '2026'`, `length(name) = 4` and `upper(name) = 'X'` matched nothing. linq2db's SQLite dialect wraps every `DateTime` comparison in `strftime('%Y-%m-%d %H:%M:%f', …)` (and `.Year` in `CAST(strftime('%Y', …) AS int)`), so `Where(x => x.At == d)`, `x.At > d`, `x.At.Year == 2026`, `x.Name.ToUpper() == …` and `x.Name.Length == …` all returned no rows, and an UPDATE or DELETE with such a WHERE silently changed nothing. The cause was in five places, all fixed: the directory-mode evaluator (`Table.EvaluateWhere`), the single-file evaluator (`SingleFileTable.EvaluateSingleCondition`, whose compiled fast path also "compiled" `strftime('%Y', at) = …` into a lookup of a column literally named `strftime('%Y', at)`), the UPDATE/DELETE/parallel-scan evaluator (`SqlParser.EvaluateJoinWhere`, which split a function call apart at its spaces), the single-table SELECT list (`ApplySelectListToRows`), and the AST executor, which mapped a function operand in WHERE to NULL and did not know the date functions. They now share one evaluator, `SqlScalarExpression` — literals, column references, SQLite's scalar string/numeric functions, `CAST`, `||` and arithmetic — and one implementation of SQLite's date and time functions, `SqlDateTimeFunctions` (`date`, `time`, `datetime`, `julianday`, `unixepoch`, `strftime` over values, with SQLite's modifiers, its month-overflow rule, `±HH:MM` handling and NULL for an invalid date). A SELECT without FROM uses the same code, which adds `date`, `time` and `datetime` there and replaces its own text-based copies of the other three. A condition is only taken by the shared evaluator when one of its operands is computed, so plain `column op literal` conditions keep their existing evaluators, collations and fast paths. Pinned by `ScalarFunctionOverColumnTests` (51 cases on directory and single-file storage — SELECT list, WHERE with AND/OR/LIKE/IS NULL/CAST, COUNT(\*), UPDATE, DELETE, the AST path and SELECT without FROM; 48 failed before) and the linq2db adapter's `DateAndStringFilterTests` (9 LINQ shapes run end to end, including a date-filtered `DeleteWhereAsync`; 9 of 9 failed against the previous engine). Suites: core **2288 / 0 failed**, VectorSearch 255 / 0, EF Core 116 / 0, linq2DB 80 / 0, regression canaries 51 / 0. Not covered: `ORDER BY` an expression and the `ExecuteQueryStruct` path, which reads rows by ordinal rather than by name. + +- **linq2db adapter: `TransactionAsync` replaced the action's error with "The operation was canceled".** It rolled back with the caller's token; once that token was cancelled the rollback itself threw, and the outer catch returned that exception instead of the action's. The rollback now runs with `CancellationToken.None`. Pinned by `FunctionalContextContractTests.TransactionAsync_WhenActionThrowsAfterTokenIsCancelled_ReportsTheActionsError`, red before and green after, with `TransactionAsync_WhenCancelledDuringTheAction_ThrowsOperationCanceled` for the cancellation contract above. +- **linq2db adapter: `InsertBatchAsync` cast its row count from `long` to `int`**, so a count above `int.MaxValue` came back as a wrapped negative number. It is now a `Fail` carrying an `OverflowException` (`ToRowCount`, unit-tested on both sides of the limit). +- **linq2db adapter: smaller defects.** `FindOneAsync` read one row into a list (`Take(1).ToListAsync()`) and now calls linq2db's `FirstOrDefaultAsync` explicitly, which avoids the BCL `AsyncEnumerable` ambiguity that led to the workaround; `QueryAsync`/`GetAllAsync` read rows into a list that `toSeq` then copied, and now read straight into the array the `Seq` wraps; `SharpCoreDBDataProvider.cs` is renamed after the type it holds (`SharpCoreDBProviderExtensions.cs`) and loses its unused usings; the test fixture no longer retries deletion of SQLite `-wal`/`-shm` files that the adapter stopped producing in 2.1.3. Suites: core 2288 / 0 failed, VectorSearch 255 / 0, EF Core 116 / 0, linq2DB **91 / 0**, regression canaries 51 / 0. + +- **ORDER BY on the single-table SELECT path read only its first token.** A second key was ignored — `ORDER BY grp, name` came back unsorted, which is what linq2db emits for `OrderBy(…).ThenBy(…)` and EF Core for composite orderings — an expression or alias key (`ORDER BY length(name)`, `ORDER BY u`) left the rows unsorted, a qualified key (`[x].[name]`) matched no row value, and a single text column was ordered with the machine's culture (case-insensitively under en-US) instead of SQLite's BINARY collation. A new `SqlOrderBy` reads the whole list — every key with its own direction, expressions through `SqlScalarExpression`, result-column aliases and ordinals, `COLLATE`, `NULLS FIRST|LAST` — and sorts stably with SQLite's rules (NULLs first ascending, BINARY unless the key or the column's declared collation says otherwise) before OFFSET/LIMIT. A list it cannot read (`vec_distance_*`, which keeps its vector-index route) keeps the previous handling. Pinned by `SingleTableOrderByTests` (directory + single-file) and the adapter's `OrderingTests`. +- **`LIMIT n OFFSET m` and `LIMIT m, n` threw "Invalid LIMIT clause"; `LIMIT 0` returned every row.** `ParseLimitClause` accepted only `LIMIT n`, so every linq2db `Skip()`/`Take()` failed on the single-table path; a negative count now means no limit (linq2db writes `LIMIT -1 OFFSET m` for `Skip` alone) and `LIMIT 0` returns no rows, as in SQLite. +- **`Database.ExecuteQueryFast` with a WHERE returned no rows — or all of them.** It keyed each row's values `col0`, `col1`, … before evaluating the WHERE, so a condition naming a column matched nothing (`grp = 2` → 0 rows), and a condition its compiled predicate could not parse fell back to "accept every row" (`length(name) = 3` → the whole table). It now evaluates by column name with the table's own WHERE evaluator — the one `ExecuteQuery` uses. The public `SqlParserPerformanceOptimizations.CompileWhereClause` no longer accepts a row it cannot evaluate: a computed operand or an unparseable condition goes to the shared evaluator. Pinned by `ExecuteQueryFastWhereTests`. +- **Single-file storage did not enforce NOT NULL.** Its INSERT paths only applied defaults, so a NULL went into a NOT NULL column in any session, and its constructor reset the NOT NULL and AUTOINCREMENT flags it had just copied. INSERT, batch INSERT and UPDATE now reject the NULL (UPDATE only when a row matches, as in SQLite), and a single-file multi-row INSERT takes the batch path, which validates every row before keeping any — one failing row used to leave the rows before it in place. Pinned by `ColumnNotNullConstraintTests` and `PrimaryKeyUniquenessTests.MultiRowInsert_WithANotNullViolation_KeepsNoRow`. +- **Two text values compared as numbers in the shared expression evaluator** (from the scalar-function fix above): `substr(at, 6, 2) = '1'` matched `'01'`. Text is now read as a number only against an actual number, as SQLite's affinity rules do. +- **`SqlFunctions.EvaluateFunction` answered SQLite's date functions only for a `DateTime` argument** (`STRFTIME`, `DATE`; anything else was NULL) and threw for `TIME`, `DATETIME`, `JULIANDAY`, `UNIXEPOCH`. They now go through `SqlDateTimeFunctions` with SQLite's argument order and format language; the legacy `(DateTime, .NET format)` shape keeps its behaviour. `SqlFunctions.StrFTime` formatted with the current culture (th-TH printed 2026 as 2569) and is now invariant. Pinned by `SqlFunctionsDateFamilyTests`. +- **Recorded, not fixed (`docs/sharpcoredb-known-issues.md` §8–§11):** an UPDATE that moves a primary key onto an existing key loses a row on directory storage and duplicates it on single-file storage; a column-level `UNIQUE` is enforced on neither mode; single-file storage ignores a declared `COLLATE`; and single-file storage enforces no PRIMARY KEY uniqueness, which cannot be switched on safely until composite keys are modelled — EF Core declares one inline `PRIMARY KEY` per key column and both modes keep only the last. Turning single-file enforcement on was tried and backed out: `CompleteExampleIntegrationTests.CompositeKey_*` showed existing EF Core databases would reject legitimate inserts. §8 and §9 need the write-path performance gate; §10 and §11 need an owner decision on the column-entry format. Suites: core **2360 / 0 failed (5 skipped: the documented single-file cases)**, VectorSearch 255 / 0, EF Core 116 / 0, linq2DB **96 / 0**, regression canaries 51 / 0. + +- **An UPDATE that moved a PRIMARY KEY onto an existing key lost a row (directory storage).** `UPDATE t SET id = 1 WHERE id = 3` with a row 1 present was accepted: the primary-key index was re-pointed at the moved row and the original row 1 became unreachable. Every UPDATE path — `Table.UpdateAffectedCount` (SQL UPDATE), `UpdateMultipleCore` (`ExecuteBatchSQL`, the structured batch API) and `UpdateByPrimaryKey` — now validates new key values before any row is written, counting what earlier operations of a batch claim and release; a SET of the key matching more than one row is rejected too. The check starts with a test that allocates nothing and returns at once for an UPDATE that does not touch a key. Pinned by `PrimaryKeyUpdateTests` (columnar and page-based, in the session and after a reopen). Known issue §8; its single-file half stays with §11. +- **A UNIQUE constraint was never enforced; a UNIQUE index only partly.** A column declared `UNIQUE` (or `UNIQUE (col)`) accepted duplicates on every storage mode: the INSERT-time check was a stub. A UNIQUE *index* was checked only by a single-row INSERT into a columnar table — not on page-based tables, not by a multi-row INSERT, not by an UPDATE — and rejected a second NULL as a duplicate. A single-column UNIQUE now gets an automatic unique hash index on directory storage, as SQLite's `sqlite_autoindex_…` (registered at CREATE TABLE and on every open, so existing tables are covered; rows stored earlier never block an open). One validator covers INSERT, both batch critical sections and every UPDATE path; NULLs never conflict; and a table with a unique index keeps the batched — atomic — multi-row INSERT, which it used to give up for a per-row loop that left earlier rows behind. Pinned by `UniqueConstraintTests` (columnar, page-based) and `MultiRowInsertBatchingTests.CanUseBatchedInsert_IsTrue_WithUniqueSecondaryIndex_BecauseTheBatchCoreEnforcesIt`, which replaces the test that pinned the old per-row fallback. Known issue §9. Still open: single-file storage, whose persisted `ColumnFlags.Unique` is also written for every inline PRIMARY KEY column, so a reopened file cannot tell a declared UNIQUE from an EF Core composite key's column (the format decision of §10/§11; enforcing it was tried and backed out when `CompleteExampleIntegrationTests.CompositeKey_*` failed); and a composite `UNIQUE (a, b)` on INSERT. Suites: core **2392 / 0 failed (6 skipped: documented single-file cases)**, VectorSearch 255 / 0, EF Core 116 / 0, linq2DB 96 / 0, regression canaries 51 / 0; write-path gate: GATE PASSED on a NOISY box (WSearch and Defender need elevation), every metric ok, worst best-of-reps ratio 1.22x on raw UPDATE against a 1.50x limit while its median ratio was 0.60x (faster). ## [2.1.4] - 2026-11-13 diff --git a/docs/sharpcoredb-known-issues.md b/docs/sharpcoredb-known-issues.md index 16ea7fd4..09abf5ba 100644 --- a/docs/sharpcoredb-known-issues.md +++ b/docs/sharpcoredb-known-issues.md @@ -421,6 +421,105 @@ attribution before merge. --- +## 8. ✅ FIXED (directory storage) — An UPDATE that moves a PRIMARY KEY onto an existing key lost a row + +> ✅ **FIXED (2026-11-17, `fix/pk-update-and-unique-constraints`) on directory storage.** Every UPDATE path — +> `Table.UpdateAffectedCount` (SQL UPDATE), `UpdateMultipleCore` (`ExecuteBatchSQL` and the structured batch API) and +> `UpdateByPrimaryKey` — now validates a new key before any row is written: a key another row holds, or a SET of the key +> that matches more than one row, is rejected and nothing changes. Pinned by `PrimaryKeyUpdateTests` (columnar and +> page-based, in the session and after a reopen). The single-file half — a duplicated key, not a lost row — is the same +> composite-key question as §11 and is tracked there. + +Found 2026-11-17. Neither storage mode checks the new key of an UPDATE that changes the primary key: + +```sql +CREATE TABLE t (id INTEGER PRIMARY KEY, email TEXT); +INSERT INTO t (id, email) VALUES (1, 'a@x'); +INSERT INTO t (id, email) VALUES (3, 'c@x'); +UPDATE t SET id = 1 WHERE id = 3; -- SQLite: UNIQUE constraint failed: t.id +SELECT id FROM t; +``` + +- **Directory storage:** the UPDATE is accepted and the original row `1` is **lost** (silent data loss). +- **Single-file storage:** the UPDATE is accepted and two rows carry id `1`. + +**Why it is open:** the directory-mode fix belongs in `Table.Update`, the write path the performance gate in +`docs/performance/AUTONOMOUS_AGENT_BRIEF.md` guards; it needs its own change with a gate run. + +--- + +## 9. ✅ FIXED (directory storage) — A column-level `UNIQUE` constraint was not enforced + +> ✅ **FIXED (2026-11-17, `fix/pk-update-and-unique-constraints`) on directory storage.** A single-column UNIQUE +> (column-level or table-level `UNIQUE (col)`) gets an automatic unique hash index, as SQLite's `sqlite_autoindex_…` — +> registered at CREATE TABLE and on every open, so existing tables are covered. INSERT, multi-row INSERT (now atomic on +> tables with a unique index) and every UPDATE path are checked; NULLs never conflict. On the way: a UNIQUE *index* was +> enforced only by a single-row INSERT into a columnar table, and a second NULL was rejected as a duplicate. Pinned by +> `UniqueConstraintTests` (columnar and page-based). **Still open:** single-file storage (its persisted +> `ColumnFlags.Unique` is also written for every inline PRIMARY KEY column, so a reopened file cannot tell a declared +> UNIQUE from an EF Core composite key's column — the format decision of §10/§11; tried and backed out when +> `CompleteExampleIntegrationTests.CompositeKey_*` failed) and a composite `UNIQUE (a, b)` on INSERT. + +Found 2026-11-17. A column declared `UNIQUE` accepts duplicates on both storage modes: + +```sql +CREATE TABLE t (id INTEGER PRIMARY KEY, email TEXT UNIQUE); +INSERT INTO t (id, email) VALUES (1, 'a@x'); +INSERT INTO t (id, email) VALUES (2, 'a@x'); -- SQLite: UNIQUE constraint failed: t.email; SharpCoreDB: accepted +``` + +A UNIQUE *index* is enforced on directory storage; the column constraint is parsed and, on single-file storage, +persisted as `ColumnFlags.Unique`, but no write path checks it. **Why it is open:** same write-path gate as §8. + +--- + +## 10. 🟡 OPEN (owner decision) — Single-file storage ignores a column's declared `COLLATE` + +> ⚠️ **STILL TRUE (verified 2026-11-17), pending an owner decision** on the column-entry format extension below. + +Found 2026-11-17. On single-file storage `CREATE TABLE t (name TEXT COLLATE NOCASE)` behaves as BINARY: `WHERE +name = 'X'` does not match `'x'`, and `ORDER BY name` sorts case-sensitively. Directory storage honours it. + +Three gaps make it up: the single-file `CREATE TABLE` parser (`DatabaseExtensions`) does not read `COLLATE`; the +single-file column entry (`ColumnDefinitionEntry`, 48 bytes) has no collation field; and the single-file WHERE +evaluator (`SingleFileTable.EvaluateSingleCondition`) is static and has no collation support. + +**Decision needed:** persisting the collation means claiming spare bits of `ColumnDefinitionEntry.Flags` (bits 8–11 +would hold `CollationType`; a file written before reads as 0 = BINARY, today's behaviour). That is compatible with +`docs/manual/upgrade-and-downgrade.md` — old files keep working, downgrade is already unsupported — but it is an +on-disk format extension. A `Locale(…)` collation would still need a name field. Pinned by the skipped +`SingleTableOrderByTests.OrderBy_ColumnDeclaredNoCase_OnSingleFile_SortsCaseInsensitively`. + +--- + +## 11. 🔴 OPEN (owner decision) — PRIMARY KEY uniqueness and composite keys + +> ⚠️ **STILL TRUE (verified 2026-11-17), pending an owner decision** on modelling composite keys. Single-file +> enforcement was implemented and backed out in the same change because of the compatibility hazard below. + +Found 2026-11-17 while fixing single-file constraint enforcement. + +- **Single-file storage does not enforce PRIMARY KEY uniqueness.** `INSERT … VALUES (2, 'y'), (1, 'dup')` into a + table already holding id 1 leaves two rows with id 1, and an UPDATE that moves a key onto an existing one (§8's + single-file half) duplicates it. +- **Neither mode models a composite key.** EF Core's `EnsureCreated` (`SharpCoreDBDatabaseCreator`) declares one + inline `PRIMARY KEY` per key column; both storage modes keep only the *last* such column as `PrimaryKeyIndex`. + Directory storage then enforces uniqueness on that one column — a false violation as soon as two rows share it + (order lines numbered 1, 2, … per order) — and single-file storage enforces nothing. SQLite rejects the + declaration outright ("table has more than one primary key"). + +**Why single-file enforcement was not simply switched on:** its format persists `ColumnFlags.PrimaryKey` on that one +column only, so a file cannot tell a single-column key from a composite key that lost its other columns. Enforcing +single-column uniqueness would make existing EF Core single-file databases with composite keys reject legitimate +inserts after an upgrade — exactly what `CompleteExampleIntegrationTests.CompositeKey_*` showed when it was tried. + +**Decision needed:** model composite keys (persist the `PrimaryKey` flag on every key column and enforce uniqueness +on the tuple — for files written before, there is no way to recover the lost key columns, so they would stay +unenforced), and only then enforce single-column uniqueness on single-file storage. Pinned by the skipped +single-file cases in `PrimaryKeyUniquenessTests`. + +--- + ## Priority summary | # | Issue | Severity | Status in ServiceMq.SharpCoreDb | In SharpCoreDB 2.1.0 | @@ -431,4 +530,8 @@ attribution before merge. | 4 | No read-after-write flush | 🟡 Low | Flush calls in provider | **Fixed** — `ExecuteQuery` flushes pending batch writes | | 5 | SQL validator `@` prefix | 🟡 Low | Keys without `@` prefix | **Fixed** — `NormalizeKey` strips `@` / `:` | | 6 | INTEGER → Int32 | 🟡 Low | `BIGINT` columns | **Still the default** — opt in with `UseSqliteIntegerAffinity` | -| 7 | Bound text re-parsed as SQL | 🔴 High | Write through `ITable.Insert` instead of SQL text | **Fixed** — one `SqlLiteralScanner` (SQL/SQLite: a doubled quote is one quote, a backslash is data) behind every scanner; `ParseValue` strips one pair and un-collapses doubling; a tuple with too many values is rejected. `ParameterLiteralRoundTripTests` 74/74 | \ No newline at end of file +| 7 | Bound text re-parsed as SQL | 🔴 High | Write through `ITable.Insert` instead of SQL text | **Fixed** — one `SqlLiteralScanner` (SQL/SQLite: a doubled quote is one quote, a backslash is data) behind every scanner; `ParseValue` strips one pair and un-collapses doubling; a tuple with too many values is rejected. `ParameterLiteralRoundTripTests` 74/74 | +| 8 | PK-changing UPDATE loses/duplicates a row | 🔴 High | — | **Fixed on directory storage** — every UPDATE path validates the new key first; single-file part tracked in §11 | +| 9 | Column `UNIQUE` not enforced | 🟠 Medium | — | **Fixed on directory storage** — automatic unique index; single-file (format decision, §10/§11) and composite `UNIQUE (a, b)` still open | +| 10 | Single-file ignores `COLLATE` | 🟡 Low | — | **Open, owner decision** — needs a column-entry format extension (§10) | +| 11 | PK uniqueness / composite keys | 🔴 High | — | **Open, owner decision** — single-file enforces no PK uniqueness; composite keys are modelled as their last column (§11) | \ No newline at end of file diff --git a/src/SharpCoreDB.Functional.Linq2DB/Extensions.cs b/src/SharpCoreDB.Functional.Linq2DB/Extensions.cs index 24091052..94658294 100644 --- a/src/SharpCoreDB.Functional.Linq2DB/Extensions.cs +++ b/src/SharpCoreDB.Functional.Linq2DB/Extensions.cs @@ -1,6 +1,6 @@ using LinqToDB; -using LinqToDB.Data; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; namespace SharpCoreDB.Functional.Linq2DB; @@ -21,12 +21,13 @@ public static DataOptions UseSharpCoreDB(this DataOptions options, string connec ArgumentNullException.ThrowIfNull(options); ArgumentException.ThrowIfNullOrWhiteSpace(connectionString); - var (provider, connection) = SharpCoreDBDataConnection.CreateProviderAndConnection(connectionString); - return SharpCoreDBDataConnection.Bind(options, provider, connection); + return SharpCoreDBDataConnection.Bind(options, connectionString); } /// - /// Adds SharpCoreDB linq2db DataConnection (via DataOptions) to the service collection. + /// Adds a SharpCoreDB linq2db and a + /// over it to the service collection, both with . Each resolved connection opens its own + /// SharpCoreDB connection. A registration the caller already made is kept. /// public static IServiceCollection AddSharpCoreDBLinq2Db( this IServiceCollection services, @@ -39,33 +40,54 @@ public static IServiceCollection AddSharpCoreDBLinq2Db( var options = new DataOptions() .UseSharpCoreDB(connectionString); - services.Add(new ServiceDescriptor( - typeof(SharpCoreDBDataConnection), - _ => new SharpCoreDBDataConnection(options), - lifetime)); - - return services; + return Register(services, options, lifetime); } /// - /// Adds SharpCoreDB DataConnection with custom DataOptions configuration. + /// Adds SharpCoreDB DataConnection with custom DataOptions configuration, for example + /// options => options.UseSharpCoreDB(connectionString).UseAdditionalMappingSchema(mySchema). + /// + /// is immutable — every Use… call returns a new instance — so the callback + /// returns the configured options. Up to 2.1.4 this overload took an Action<DataOptions>, discarded + /// what the callback built, and resolving the connection threw "Configuration string is not provided". + /// /// + /// The service collection. + /// Builds the options from an empty . + /// Lifetime of the registered connection. + /// The service collection. public static IServiceCollection AddSharpCoreDBLinq2Db( this IServiceCollection services, - Action configureOptions, + Func configureOptions, ServiceLifetime lifetime = ServiceLifetime.Scoped) { ArgumentNullException.ThrowIfNull(services); ArgumentNullException.ThrowIfNull(configureOptions); - var options = new DataOptions(); - configureOptions(options); + var options = configureOptions(new DataOptions()) + ?? throw new InvalidOperationException( + "The configureOptions callback returned null; return the configured DataOptions."); + return Register(services, options, lifetime); + } + + /// + /// Registers the connection and the functional context over it. Up to 2.1.4 only the connection was registered, + /// so every caller wired by hand — often as a singleton over one shared + /// connection. + /// + private static IServiceCollection Register(IServiceCollection services, DataOptions options, ServiceLifetime lifetime) + { services.Add(new ServiceDescriptor( typeof(SharpCoreDBDataConnection), _ => new SharpCoreDBDataConnection(options), lifetime)); + services.TryAdd(new ServiceDescriptor( + typeof(FunctionalLinq2DbContext), + sp => new FunctionalLinq2DbContext(sp.GetRequiredService()), + lifetime)); + return services; } diff --git a/src/SharpCoreDB.Functional.Linq2DB/FunctionalLinq2DbContext.cs b/src/SharpCoreDB.Functional.Linq2DB/FunctionalLinq2DbContext.cs index f17e7ac6..b6952d03 100644 --- a/src/SharpCoreDB.Functional.Linq2DB/FunctionalLinq2DbContext.cs +++ b/src/SharpCoreDB.Functional.Linq2DB/FunctionalLinq2DbContext.cs @@ -1,12 +1,8 @@ -using System; -using System.Collections.Generic; using System.Linq.Expressions; -using System.Threading; -using System.Threading.Tasks; +using System.Runtime.InteropServices; using LinqToDB; -using LinqToDB.Data; using LinqToDB.Async; -using SharpCoreDB.Functional; +using LinqToDB.Data; using static SharpCoreDB.Functional.Prelude; namespace SharpCoreDB.Functional.Linq2DB; @@ -15,6 +11,11 @@ namespace SharpCoreDB.Functional.Linq2DB; /// Functional adapter over linq2db operations with Option/Fin return types. /// Provides railway-oriented programming patterns for SharpCoreDB queries. /// C# 15: Uses primary constructor for immutable dependency injection. +/// +/// The error contract: methods returning or throw when the query fails; +/// methods returning capture the failure as Fail; and cancellation always surfaces as +/// — before the call or during it — never as a failure. +/// /// /// The underlying SharpCoreDB linq2db data connection public sealed class FunctionalLinq2DbContext(SharpCoreDBDataConnection connection) @@ -22,42 +23,121 @@ public sealed class FunctionalLinq2DbContext(SharpCoreDBDataConnection connectio private readonly SharpCoreDBDataConnection _connection = connection ?? throw new ArgumentNullException(nameof(connection)); /// - /// Gets an entity by its primary key. + /// Gets an entity by its mapped primary key. + /// + /// Up to 2.1.4 this ignored and returned the first row of the table, and it turned + /// every exception into None. A query that fails now throws, as does. + /// /// /// Entity type - /// Primary key value(s). For single-column PKs pass a single-element array or use overloads in future versions. + /// + /// One value per primary-key column, in PrimaryKey(order) order. A value of another numeric type (an + /// for a key) is converted to the member's type. + /// /// Cancellation token - /// Some entity when found; none otherwise. For production, prefer with explicit predicates for complex keys. + /// Some entity when a row has that key; none otherwise. + /// + /// The number of values differs from the number of key columns, or a value cannot be converted to its column's type. + /// + /// has no mapped primary key. public async Task> GetByIdAsync( object[] keyValues, CancellationToken cancellationToken = default) where T : class { ArgumentNullException.ThrowIfNull(keyValues); - if (keyValues.Length == 0) - return None(); - cancellationToken.ThrowIfCancellationRequested(); - // Production implementation note: Full PK metadata lookup (via MappingSchema or ITable metadata) is complex for arbitrary entities. - // For now, we provide a simple first-match for single-PK tables (common case). Users should use FindOneAsync with explicit predicates for robustness. - // This avoids reflection overhead and mapping schema inspection in the hot path. - try + var predicate = BuildPrimaryKeyPredicate(keyValues); + return await FindOneAsync(predicate, cancellationToken).ConfigureAwait(false); + } + + /// + /// Builds row => row.Key1 == keyValues[0] && … from the entity's mapped primary-key columns. + /// The values are read through a holder object, the same shape as a compiler-generated closure, so linq2db binds + /// them as parameters and caches one query per entity type instead of one per key. + /// + private Expression> BuildPrimaryKeyPredicate(object[] keyValues) + where T : class + { + var keyColumns = _connection.MappingSchema + .GetEntityDescriptor(typeof(T), _connection.Options.ConnectionOptions.OnEntityDescriptorCreated) + .Columns + .Where(column => column.IsPrimaryKey) + .OrderBy(column => column.PrimaryKeyOrder) + .ToArray(); + + if (keyColumns.Length == 0) { - IQueryable q = _connection.GetTable(); - var list = await q.Take(1).ToListAsync(cancellationToken).ConfigureAwait(false); // Limit to avoid full table scan in fallback - var entity = list.FirstOrDefault(); + throw new InvalidOperationException( + $"{typeof(T).Name} has no mapped primary key; use FindOneAsync with an explicit predicate."); + } + + if (keyValues.Length != keyColumns.Length) + { + throw new ArgumentException( + $"{typeof(T).Name} has {keyColumns.Length} primary-key column(s) " + + $"({string.Join(", ", keyColumns.Select(column => column.MemberName))}), " + + $"but {keyValues.Length} key value(s) were supplied.", + nameof(keyValues)); + } + + var holder = new KeyValueHolder(new object?[keyColumns.Length]); + for (var i = 0; i < keyColumns.Length; i++) + { + holder.Values[i] = ConvertKeyValue(keyValues[i], keyColumns[i].MemberType, keyColumns[i].MemberName); + } + + var row = Expression.Parameter(typeof(T), "row"); + var values = Expression.Field(Expression.Constant(holder), nameof(KeyValueHolder.Values)); + Expression? body = null; + for (var i = 0; i < keyColumns.Length; i++) + { + var equal = Expression.Equal( + Expression.MakeMemberAccess(row, keyColumns[i].MemberInfo), + Expression.Convert(Expression.ArrayIndex(values, Expression.Constant(i)), keyColumns[i].MemberType)); + body = body is null ? equal : Expression.AndAlso(body, equal); + } + + return Expression.Lambda>(body!, row); + } + + private static object? ConvertKeyValue(object? value, Type memberType, string memberName) + { + var targetType = Nullable.GetUnderlyingType(memberType) ?? memberType; + if (value is null) + { + return !memberType.IsValueType || targetType != memberType + ? null + : throw new ArgumentException($"Key value for {memberName} cannot be null.", "keyValues"); + } + + if (targetType.IsInstanceOfType(value)) + { + return value; + } - return Optional(entity); + try + { + return targetType.IsEnum + ? Enum.ToObject(targetType, value) + : Convert.ChangeType(value, targetType, System.Globalization.CultureInfo.InvariantCulture); } - catch (Exception) + catch (Exception ex) when (ex is InvalidCastException or FormatException or OverflowException + or ArgumentException) { - // In production GetByIdAsync, we could log but here we return None to maintain Option semantics (no exception leakage). - // Advanced users can catch via TransactionAsync or use raw linq2db. - return None(); + throw new ArgumentException( + $"Key value '{value}' ({value.GetType().Name}) cannot be converted to {targetType.Name} for {memberName}.", + "keyValues", + ex); } } + private sealed class KeyValueHolder(object?[] values) + { + public readonly object?[] Values = values; + } + /// /// Gets the first entity matching the predicate. /// @@ -73,9 +153,11 @@ public async Task> FindOneAsync( ArgumentNullException.ThrowIfNull(predicate); cancellationToken.ThrowIfCancellationRequested(); - IQueryable q = _connection.GetTable().Where(predicate); - var list = await q.Take(1).ToListAsync(cancellationToken).ConfigureAwait(false); // Explicit Take(1) + ToListAsync for optimal linq2db translation and C# 15 overload resolution - var entity = list.FirstOrDefault(); + // Called through linq2db's own extension class: since .NET 10 the BCL's System.Linq.AsyncEnumerable offers a + // FirstOrDefaultAsync for the same receiver, and that ambiguity is why this used Take(1).ToListAsync(), which + // allocated a list for one row. + var entity = await AsyncExtensions.FirstOrDefaultAsync(_connection.GetTable().Where(predicate), cancellationToken) + .ConfigureAwait(false); return Optional(entity); } @@ -87,7 +169,7 @@ public async Task> FindOneAsync( /// Query builder callback /// Cancellation token /// A functional sequence of results - public async Task> QueryAsync( + public Task> QueryAsync( Func, IQueryable> queryBuilder, CancellationToken cancellationToken = default) where T : class @@ -95,10 +177,7 @@ public async Task> QueryAsync( ArgumentNullException.ThrowIfNull(queryBuilder); cancellationToken.ThrowIfCancellationRequested(); - IQueryable q = queryBuilder(_connection.GetTable()); - var result = await q.ToListAsync(cancellationToken).ConfigureAwait(false); - - return toSeq(result); + return ToSeqAsync(queryBuilder(_connection.GetTable()), cancellationToken); } /// @@ -108,7 +187,7 @@ public async Task> QueryAsync( /// Filter predicate /// Cancellation token /// A functional sequence of matching results - public async Task> QueryAsync( + public Task> QueryAsync( Expression> predicate, CancellationToken cancellationToken = default) where T : class @@ -116,10 +195,7 @@ public async Task> QueryAsync( ArgumentNullException.ThrowIfNull(predicate); cancellationToken.ThrowIfCancellationRequested(); - IQueryable q = _connection.GetTable().Where(predicate); - var result = await q.ToListAsync(cancellationToken).ConfigureAwait(false); - - return toSeq(result); + return ToSeqAsync(_connection.GetTable().Where(predicate), cancellationToken); } /// @@ -128,15 +204,12 @@ public async Task> QueryAsync( /// Entity type /// Cancellation token /// A functional sequence of all entities - public async Task> GetAllAsync(CancellationToken cancellationToken = default) + public Task> GetAllAsync(CancellationToken cancellationToken = default) where T : class { cancellationToken.ThrowIfCancellationRequested(); - IQueryable q = _connection.GetTable(); - var result = await q.ToListAsync(cancellationToken).ConfigureAwait(false); - - return toSeq(result); + return ToSeqAsync(_connection.GetTable(), cancellationToken); } /// @@ -146,7 +219,7 @@ public async Task> GetAllAsync(CancellationToken cancellationToken = d /// Entity to insert /// Cancellation token /// Success when persisted; failure with error otherwise - public async Task> InsertAsync( + public Task> InsertAsync( T entity, CancellationToken cancellationToken = default) where T : class @@ -154,53 +227,76 @@ public async Task> InsertAsync( ArgumentNullException.ThrowIfNull(entity); cancellationToken.ThrowIfCancellationRequested(); - try + return RunAsync(async () => { await _connection.InsertAsync(entity, token: cancellationToken).ConfigureAwait(false); - return FinSucc(unit); - } - catch (Exception ex) - { - return FinFail(Error.New(ex)); - } + return unit; + }); } /// - /// Inserts multiple entities in a batch using linq2db BulkCopy for high performance. + /// Inserts multiple entities in a batch using linq2db BulkCopy (multi-row SQL, up to 1000 rows per statement). /// /// Entity type /// Entities to insert /// Cancellation token /// Success with row count; failure with error otherwise + public Task> InsertBatchAsync( + IEnumerable entities, + CancellationToken cancellationToken = default) + where T : class + => InsertBatchAsync(entities, DefaultBulkCopyOptions, cancellationToken); + + /// + /// Inserts multiple entities in a batch using linq2db BulkCopy with the caller's options — batch size, + /// parameters, row-copied notifications. + /// + /// is replaced by : the + /// provider-specific path needs a native Microsoft.Data.Sqlite connection, and the adapter's connection is + /// SharpCoreDB's. The caller's instance is not modified. + /// + /// + /// Entity type + /// Entities to insert + /// Bulk copy options + /// Cancellation token + /// + /// Success with the row count; failure with error otherwise, including a count above + /// (up to 2.1.4 the count was cast and wrapped to a negative number). + /// public async Task> InsertBatchAsync( IEnumerable entities, + BulkCopyOptions options, CancellationToken cancellationToken = default) where T : class { ArgumentNullException.ThrowIfNull(entities); + ArgumentNullException.ThrowIfNull(options); cancellationToken.ThrowIfCancellationRequested(); - try - { - var options = new BulkCopyOptions - { - MaxBatchSize = 1000, // Tuned for typical SharpCoreDB workloads - UseParameters = true - }; + var effective = options.BulkCopyType == BulkCopyType.ProviderSpecific + ? options with { BulkCopyType = BulkCopyType.MultipleRows } + : options; - var result = await _connection.BulkCopyAsync( - options, - entities, - cancellationToken).ConfigureAwait(false); + var copied = await RunAsync(async () => + (await _connection.BulkCopyAsync(effective, entities, cancellationToken).ConfigureAwait(false)).RowsCopied) + .ConfigureAwait(false); - return FinSucc((int)result.RowsCopied); - } - catch (Exception ex) - { - return FinFail(Error.New(ex)); - } + return copied.Bind(ToRowCount); } + /// + /// Converts a bulk-copy row count to the the batch API returns: a count that does not fit is a + /// failure, not a wrapped negative number. + /// + /// The row count linq2db reported. + /// The count, or a failure carrying an . + internal static Fin ToRowCount(long rowsCopied) => + rowsCopied <= int.MaxValue + ? FinSucc((int)rowsCopied) + : FinFail(Error.New(new OverflowException( + $"{rowsCopied} rows were copied, more than an Int32 count can report; split the batch."))); + /// /// Updates an existing entity. /// @@ -208,7 +304,7 @@ public async Task> InsertBatchAsync( /// Entity to update /// Cancellation token /// Success when updated; failure with error otherwise - public async Task> UpdateAsync( + public Task> UpdateAsync( T entity, CancellationToken cancellationToken = default) where T : class @@ -216,15 +312,11 @@ public async Task> UpdateAsync( ArgumentNullException.ThrowIfNull(entity); cancellationToken.ThrowIfCancellationRequested(); - try + return RunAsync(async () => { await _connection.UpdateAsync(entity, token: cancellationToken).ConfigureAwait(false); - return FinSucc(unit); - } - catch (Exception ex) - { - return FinFail(Error.New(ex)); - } + return unit; + }); } /// @@ -234,7 +326,7 @@ public async Task> UpdateAsync( /// Entity to delete /// Cancellation token /// Success when deleted; failure with error otherwise - public async Task> DeleteAsync( + public Task> DeleteAsync( T entity, CancellationToken cancellationToken = default) where T : class @@ -242,15 +334,11 @@ public async Task> DeleteAsync( ArgumentNullException.ThrowIfNull(entity); cancellationToken.ThrowIfCancellationRequested(); - try + return RunAsync(async () => { await _connection.DeleteAsync(entity, token: cancellationToken).ConfigureAwait(false); - return FinSucc(unit); - } - catch (Exception ex) - { - return FinFail(Error.New(ex)); - } + return unit; + }); } /// @@ -260,7 +348,7 @@ public async Task> DeleteAsync( /// Filter predicate /// Cancellation token /// Success with deleted count; failure with error otherwise - public async Task> DeleteWhereAsync( + public Task> DeleteWhereAsync( Expression> predicate, CancellationToken cancellationToken = default) where T : class @@ -268,19 +356,7 @@ public async Task> DeleteWhereAsync( ArgumentNullException.ThrowIfNull(predicate); cancellationToken.ThrowIfCancellationRequested(); - try - { - var count = await _connection.GetTable() - .Where(predicate) - .DeleteAsync(cancellationToken) - .ConfigureAwait(false); - - return FinSucc(count); - } - catch (Exception ex) - { - return FinFail(Error.New(ex)); - } + return RunAsync(() => _connection.GetTable().Where(predicate).DeleteAsync(cancellationToken)); } /// @@ -290,26 +366,20 @@ public async Task> DeleteWhereAsync( /// Optional filter predicate /// Cancellation token /// Success with count; failure with error otherwise - public async Task> CountAsync( + public Task> CountAsync( Expression>? predicate = null, CancellationToken cancellationToken = default) where T : class { cancellationToken.ThrowIfCancellationRequested(); - try - { - IQueryable q = _connection.GetTable(); - long count = predicate == null - ? await q.LongCountAsync(cancellationToken).ConfigureAwait(false) - : await q.Where(predicate).LongCountAsync(cancellationToken).ConfigureAwait(false); - - return FinSucc(count); - } - catch (Exception ex) + IQueryable q = _connection.GetTable(); + if (predicate is not null) { - return FinFail(Error.New(ex)); + q = q.Where(predicate); } + + return RunAsync(() => q.LongCountAsync(cancellationToken)); } /// @@ -319,7 +389,7 @@ public async Task> CountAsync( /// Filter predicate /// Cancellation token /// Success with true/false; failure with error otherwise - public async Task> ExistsAsync( + public Task> ExistsAsync( Expression> predicate, CancellationToken cancellationToken = default) where T : class @@ -327,55 +397,98 @@ public async Task> ExistsAsync( ArgumentNullException.ThrowIfNull(predicate); cancellationToken.ThrowIfCancellationRequested(); - try - { - IQueryable q = _connection.GetTable().Where(predicate); - var exists = await q.AnyAsync(cancellationToken).ConfigureAwait(false); - return FinSucc(exists); - } - catch (Exception ex) - { - return FinFail(Error.New(ex)); - } + return RunAsync(() => _connection.GetTable().Where(predicate).AnyAsync(cancellationToken)); } /// - /// Executes a transaction with functional error handling. + /// Executes in a transaction: committed when it completes, rolled back when it throws. + /// Prefer the overload that hands the action this context and the token. /// /// Result type /// Transaction action /// Cancellation token - /// Success with result; failure with error if transaction rolls back - public async Task> TransactionAsync( + /// Success with result; failure with the action's error if the transaction rolls back + public Task> TransactionAsync( Func> action, CancellationToken cancellationToken = default) { ArgumentNullException.ThrowIfNull(action); + + return TransactionAsync((_, _) => action(), cancellationToken); + } + + /// + /// Executes in a transaction, passing it this context and the token so the work inside + /// is visibly bound to the transaction's connection: committed when it completes, rolled back when it throws. + /// + /// The rollback runs with . Up to 2.1.4 it used the caller's token: once that + /// token was cancelled the rollback itself threw, and the action's own error was replaced by "The operation was + /// canceled". Cancellation surfaces as , never as a failure. + /// + /// + /// Result type + /// Transaction action, given this context and . + /// Cancellation token + /// Success with result; failure with the action's error if the transaction rolls back + public async Task> TransactionAsync( + Func> action, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(action); cancellationToken.ThrowIfCancellationRequested(); - try + return await RunAsync(async () => { await using var transaction = await _connection.BeginTransactionAsync(cancellationToken).ConfigureAwait(false); try { - var result = await action().ConfigureAwait(false); + var result = await action(this, cancellationToken).ConfigureAwait(false); await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); - return FinSucc(result); + return result; } catch { - await transaction.RollbackAsync(cancellationToken).ConfigureAwait(false); + await transaction.RollbackAsync(CancellationToken.None).ConfigureAwait(false); throw; } + }).ConfigureAwait(false); + } + + /// + /// Gets the underlying linq2db DataConnection for advanced scenarios. + /// + public SharpCoreDBDataConnection Connection => _connection; + + private static readonly BulkCopyOptions DefaultBulkCopyOptions = new() + { + MaxBatchSize = 1000, // Tuned for typical SharpCoreDB workloads + UseParameters = true, + }; + + /// + /// The Fin contract in one place: a failure becomes Fail, and cancellation propagates as + /// . Up to 2.1.4 every method caught everything, so a token cancelled + /// during the call came back as a failure while one cancelled before it threw. + /// + private static async Task> RunAsync(Func> operation) + { + try + { + return FinSucc(await operation().ConfigureAwait(false)); } - catch (Exception ex) + catch (Exception ex) when (ex is not OperationCanceledException) { return FinFail(Error.New(ex)); } } /// - /// Gets the underlying linq2db DataConnection for advanced scenarios. + /// Materialises a query straight into the array a wraps. Up to 2.1.4 the rows were read into + /// a list and then copied again by toSeq. /// - public SharpCoreDBDataConnection Connection => _connection; + private static async Task> ToSeqAsync(IQueryable query, CancellationToken cancellationToken) + { + var rows = await AsyncExtensions.ToArrayAsync(query, cancellationToken).ConfigureAwait(false); + return new Seq(ImmutableCollectionsMarshal.AsImmutableArray(rows)); + } } diff --git a/src/SharpCoreDB.Functional.Linq2DB/README.md b/src/SharpCoreDB.Functional.Linq2DB/README.md index 4503bc34..d1c94824 100644 --- a/src/SharpCoreDB.Functional.Linq2DB/README.md +++ b/src/SharpCoreDB.Functional.Linq2DB/README.md @@ -2,21 +2,20 @@ linq2db adapter for `SharpCoreDB.Functional`. -**Version:** `v1.9.5` (production-ready) -**Package:** `SharpCoreDB.Functional.Linq2DB` +**Package:** `SharpCoreDB.Functional.Linq2DB` (versioned with SharpCoreDB — see NuGet for the current release) --- ## Overview -**SharpCoreDB.Functional.Linq2DB** is the **production-ready** linq2db adapter for SharpCoreDB's functional stack. It delivers compile-time type safety, expressive LINQ, zero change-tracking overhead, and railway-oriented return types (`Option`, `Fin`, `Seq`). +**SharpCoreDB.Functional.Linq2DB** is the linq2db adapter for SharpCoreDB's functional stack. It delivers compile-time type safety, expressive LINQ, zero change-tracking overhead, and railway-oriented return types (`Option`, `Fin`, `Seq`). **Perfect for:** - ✅ High-throughput AI/agentic workflows and GraphRAG - ✅ Compile-time safe queries with full IntelliSense - ✅ Low memory/CPU overhead (no EF-style tracking) - ✅ Bulk operations via `BulkCopyAsync` -- ✅ Native AOT and .NET 11/C# 15 +- ✅ .NET 11 / C# 15 - ✅ Railway-oriented error handling (`Fin`) --- @@ -24,12 +23,12 @@ linq2db adapter for `SharpCoreDB.Functional`. ## Installation ```bash -dotnet add package SharpCoreDB.Functional.Linq2DB --version 2.0.0.0 +dotnet add package SharpCoreDB.Functional.Linq2DB ``` --- -## Quick Start (Production) +## Quick Start ### 1. Basic LINQ (raw linq2db) @@ -76,9 +75,11 @@ The adapter is the SQLite **dialect**, not the SQLite **engine**: - The SQL is executed on a `SharpCoreDBConnection` (from `SharpCoreDB.Data.Provider`), so the rows are stored by SharpCoreDB in its own file format and encrypted with AES-256-GCM when the connection string carries a `Password`. -- No SQLite file is created and the native `e_sqlite3` library is never opened. Microsoft.Data.Sqlite remains a - package dependency because linq2db's SQLite dialect resolves it for its provider plumbing; `BulkCopy` is set to - `MultipleRows` so the provider-specific native bulk-copy path is never taken. +- No SQLite file is created and no native SQLite library ships with the package. Its one SQLite dependency is + `Microsoft.Data.Sqlite.Core` — the managed library only — because linq2db's SQLite dialect reflects over its types + when it builds its provider; nothing ever opens a connection with it. `BulkCopy` uses `MultipleRows`, so the + provider-specific native bulk-copy path is never taken. (Up to 2.1.4 the dependency was the `Microsoft.Data.Sqlite` + metapackage, which also shipped the native `e_sqlite3` binaries for every platform.) This changed in 2.1.3: before that, `SharpCoreDBDataConnection` passed the connection string straight to `ProviderName.SQLite`, so a password was refused (*"the native SQLite library 'e_sqlite3' doesn't support @@ -89,11 +90,69 @@ SharpCoreDB-specific SQL that linq2db does not model. --- -## Key Production Features +## Dependency injection, connections and mapping schemas + +```csharp +// One SharpCoreDBDataConnection per scope, each with its own SharpCoreDB connection, +// and a FunctionalLinq2DbContext over it (a registration of your own is kept). +services.AddSharpCoreDBLinq2Db("Data Source=./app.scdb;Password=…"); + +// Or configure the options yourself. DataOptions is immutable, so the callback returns them. +services.AddSharpCoreDBLinq2Db(options => options + .UseSharpCoreDB("Data Source=./app.scdb;Password=…") + .UseAdditionalMappingSchema(myFluentMappings)); +``` + +- The options built by `CreateOptions`, `UseSharpCoreDB` and `AddSharpCoreDBLinq2Db` hold a connection + **factory**: every `SharpCoreDBDataConnection` opens its own `SharpCoreDBConnection` and disposes it with itself, + so options can be reused safely. (Up to 2.1.4 they held one live connection that every DI scope shared.) +- Your own mapping schema — fluent mappings, attribute overrides, converters — is **kept** and takes priority over + the SharpCoreDB type mappings. Add it with `UseAdditionalMappingSchema` (or `UseMappingSchema` before + `UseSharpCoreDB`). `SharpCoreDBMappingSchema.ConfigureCustomMapping` is obsolete: it changed the process-wide + `Instance` shared by every connection. +- `FunctionalLinq2DbContext.GetByIdAsync(keys)` looks the row up by the entity's mapped primary key — one value + per key column, in `PrimaryKey(order)` order — and throws `ArgumentException` on a key-count mismatch. + +--- + +## Errors, cancellation and transactions + +- Methods returning `Option` or `Seq` (`FindOneAsync`, `GetByIdAsync`, `QueryAsync`, `GetAllAsync`) **throw** + when the query fails. +- Methods returning `Fin` (inserts, updates, deletes, `CountAsync`, `ExistsAsync`, `TransactionAsync`) capture the + failure as `Fail`. +- Cancellation always surfaces as `OperationCanceledException` — whether the token was cancelled before the call or + during it — never as a `Fail`. + +```csharp +// The action gets the context and the token, so the work is visibly bound to the transaction. +// linq2db's own methods on ctx.Connection throw on failure, which is what rolls the transaction back. +var result = await db.TransactionAsync(async (ctx, ct) => +{ + await ctx.Connection.InsertAsync(order, token: ct); + await ctx.Connection.BulkCopyAsync(lines, ct); + return order.Id; +}, cancellationToken); +``` + +A throwing action rolls the transaction back and its own error is returned as `Fail` — the rollback does not use the +caller's token, so a cancelled token cannot replace the action's error. The context's own `Fin`-returning methods do +**not** throw: a `Fail` from `ctx.InsertAsync(…)` inside the action is a value, and the transaction still commits +unless the action throws when it sees one (`if (insert.IsFail) throw new InvalidOperationException(…)`) or uses the +throwing linq2db calls on `ctx.Connection`, as above. + +`InsertBatchAsync(entities, options)` takes linq2db `BulkCopyOptions` (batch size, parameters, notifications); +`BulkCopyType.ProviderSpecific` is mapped to `MultipleRows`. A row count above `int.MaxValue` is a `Fail`, not a +wrapped number. + +--- + +## Key Features - **BulkCopy support** in `InsertBatchAsync` for high-speed ingestion (GraphRAG, analytics) -- **Full type mapping** (ULID as string, GUID as compact string, DateTime ISO, bool→int for SQLite compat) -- **Transaction support** with `TransactionAsync` +- **Type mapping** (ULID as text, GUID as hyphenated `D`-format text, dates read and written culture-invariantly, + bool→int for SQLite compat) +- **Transaction support** with `TransactionAsync` (commit on success, rollback on failure) - **DeleteWhereAsync**, `CountAsync`, `ExistsAsync`, `GetAllAsync` - **Seamless with SharpCoreDB.Functional** (`Prelude`, `Option`, `Fin`, `Seq`, `Error`) - Works alongside `SharpCoreDB.Functional.Dapper` and EF Core functional wrapper diff --git a/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDB.Functional.Linq2DB.csproj b/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDB.Functional.Linq2DB.csproj index e89f88e6..122f6175 100644 --- a/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDB.Functional.Linq2DB.csproj +++ b/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDB.Functional.Linq2DB.csproj @@ -11,10 +11,10 @@ MPCoreDeveloper SharpCoreDB SharpCoreDB.Functional.Linq2DB - Production-ready linq2db adapter for SharpCoreDB.Functional. Zero-dependency Option/Fin/Seq APIs, compile-time LINQ safety, BulkCopy batching, full type mappings (ULID/GUID), and low-overhead queries. Perfect for AI/agentic and GraphRAG workloads on .NET 11 / C# 15. + linq2db adapter for SharpCoreDB: linq2db's SQLite dialect generates the SQL, SharpCoreDB executes it and owns the encrypted file. Option/Fin/Seq functional APIs, compile-time LINQ, multi-row BulkCopy batching, ULID/GUID type mappings and DI registration, on .NET 11 / C# 15. Copyright (c) 2026 MPCoreDeveloper v2.1.4 (2026-11-13): quoted identifiers in CREATE TABLE are catalogued without their quoting, so a schema generated by a query builder (linq2db, EF Core, FluentMigrator) can insert into the table it just created, and ALTER TABLE ADD COLUMN reads a quoted column name the same way; Previous: v2.1.3 (2026-11-12): the linq2db adapter now executes on SharpCoreDB instead of SQLite, so a password is honoured and rows live in an encrypted SharpCoreDB file, and the SQL shapes that blocked it are fixed - pretty-printed INSERTs, SELECT-list projection, bare and negated boolean WHERE, LIKE with ESCAPE, scalar EXISTS, and hex byte[]/Guid literals. Previous: v2.1.2 (2026-10-26): patch release with no API changes - a bound text parameter is no longer re-scanned as SQL text, so a value holding an apostrophe, a doubled quote, a backslash, a comma or a parenthesis now round-trips exactly instead of landing in the neighbouring column or failing with "Mismatched parentheses in VALUES clause"; full detail in docs/CHANGELOG.md. Previous: v2.1.1 (2026-10-25): patch release with no API changes. A float[] parameter now reaches a VECTOR(n) column as a vector instead of the literal text "System.Single[]", and ORDER BY vec_distance_*() LIMIT k now reaches the vector index instead of a full scan. Previous: v2.1.0 (2026-10-10): net11.0-only (C# 15 preview) release line - 2,897 tests green across 16 suites; the fixed-width inline capacity ships at 16 by default on both storage paths (batched multi-row INSERT measured +12 to +19 percent); the overflow arena writes a row's payloads in one append instead of one file open per value; and in-place UPDATE/DELETE with deferred index maintenance put the fixed-width fair-PK layout ahead of SQLite on UPDATE (0.8x) and DELETE (0.5x). Full details: https://github.com/MPCoreDeveloper/SharpCoreDB/blob/master/docs/CHANGELOG.md - sharpcoredb;functional;linq2db;linq;option;fin;net11;csharp15;aot + sharpcoredb;functional;linq2db;linq;option;fin;net11;csharp15 MIT https://github.com/MPCoreDeveloper/SharpCoreDB https://github.com/MPCoreDeveloper/SharpCoreDB @@ -31,9 +31,12 @@ - - + + @@ -43,6 +46,10 @@ + + + + diff --git a/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBDataConnection.cs b/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBDataConnection.cs index f21bb72e..46a54b7d 100644 --- a/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBDataConnection.cs +++ b/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBDataConnection.cs @@ -1,6 +1,7 @@ using LinqToDB; using LinqToDB.Data; using LinqToDB.DataProvider.SQLite; +using LinqToDB.Mapping; using SharpCoreDB.Data.Provider; namespace SharpCoreDB.Functional.Linq2DB; @@ -36,21 +37,27 @@ public SharpCoreDBDataConnection(string connectionString) /// /// Initializes using pre-built (recommended for DI). The options must carry a - /// SharpCoreDB connection — build them with or - /// . + /// SharpCoreDB connection — build them with , + /// or . + /// A mapping schema already on the options is kept and takes priority over the SharpCoreDB type mappings. /// /// linq2db options, already bound to a SharpCoreDB connection. public SharpCoreDBDataConnection(DataOptions options) - : base(options.UseMappingSchema(SharpCoreDBMappingSchema.Instance)) + : base(WithSharpCoreDBMappingSchema(options)) { EnsureConnectionOpen(); } /// - /// Builds that pair linq2db's SQLite SQL dialect with a SharpCoreDB ADO.NET - /// connection. This is the one place the pairing is defined; the string constructor, + /// Builds that pair linq2db's SQLite SQL dialect with SharpCoreDB ADO.NET + /// connections. This is the one place the pairing is defined; the string constructor, /// and /// all route through it. + /// + /// The options hold a connection factory, not a connection: every + /// built from them opens its own and disposes it with itself, so the options + /// can be reused — by a DI registration, for example — without sharing one ADO.NET connection between callers. + /// /// /// SharpCoreDB connection string. /// Options ready for . @@ -58,46 +65,58 @@ public static DataOptions CreateOptions(string connectionString) { ArgumentException.ThrowIfNullOrWhiteSpace(connectionString); - var (provider, connection) = CreateProviderAndConnection(connectionString); - return Bind(new DataOptions(), provider, connection); + return Bind(new DataOptions(), connectionString); } /// - /// Creates the SharpCoreDB connection and the linq2db provider that generates SQL for it — the one place the - /// pairing is defined. + /// Binds the SQLite-dialect provider and a SharpCoreDB connection factory to , + /// together with the settings the adapter requires (multi-row SQL bulk copy, SharpCoreDB type mappings). /// + /// The options to extend. /// SharpCoreDB connection string. - /// The SQLite-dialect provider and the SharpCoreDB connection it will execute on. - internal static (LinqToDB.DataProvider.IDataProvider Provider, SharpCoreDBConnection Connection) CreateProviderAndConnection(string connectionString) + /// The extended options. + internal static DataOptions Bind(DataOptions options, string connectionString) { + ArgumentNullException.ThrowIfNull(options); ArgumentException.ThrowIfNullOrWhiteSpace(connectionString); - var connection = new SharpCoreDBConnection(connectionString); + // The provider selects the SQL *dialect* only. Because linq2db takes its connections from the factory, it + // executes every command on a SharpCoreDBConnection and never opens Microsoft.Data.Sqlite; a connection it + // created through the factory is one it owns and disposes. Up to 2.1.4 the options captured one live + // connection instead, which every DataConnection built from them — every DI scope — then shared. + var provider = SQLiteTools.GetDataProvider(SQLiteProvider.Microsoft); - // The provider name selects the SQL *dialect*; because a live connection is supplied, linq2db executes - // every command on that connection instead of creating one of its own. - var provider = SQLiteTools.GetDataProvider(SQLiteProvider.Microsoft, connectionString, connection); - return (provider, connection); + return WithSharpCoreDBMappingSchema(options + .UseConnectionFactory(provider, _ => new SharpCoreDBConnection(connectionString)) + // Provider-specific SQLite bulk copy needs a native Microsoft.Data.Sqlite connection; generate the + // inserts as multi-row SQL instead so bulk copy works through SharpCoreDB. + .UseBulkCopyType(BulkCopyType.MultipleRows)); } /// - /// Binds a provider and connection to together with the settings the adapter - /// requires (multi-row SQL bulk copy, SharpCoreDB type mappings). + /// Adds to without discarding a mapping schema + /// the caller registered. UseMappingSchema alone "replaces all previous registrations", which up to 2.1.4 + /// silently dropped fluent mappings. The caller's schema goes first because linq2db gives the first base schema + /// the higher priority. /// /// The options to extend. - /// The SQLite-dialect provider. - /// The SharpCoreDB connection. - /// The extended options. - internal static DataOptions Bind(DataOptions options, LinqToDB.DataProvider.IDataProvider provider, SharpCoreDBConnection connection) + /// Options whose mapping schema includes the SharpCoreDB type mappings exactly once. + internal static DataOptions WithSharpCoreDBMappingSchema(DataOptions options) { ArgumentNullException.ThrowIfNull(options); - return options - .UseConnection(provider, connection) - // Provider-specific SQLite bulk copy needs a native Microsoft.Data.Sqlite connection; generate the - // inserts as multi-row SQL instead so bulk copy works through SharpCoreDB. - .UseBulkCopyType(BulkCopyType.MultipleRows) - .UseMappingSchema(SharpCoreDBMappingSchema.Instance); + var current = options.ConnectionOptions.MappingSchema; + if (current is null) + { + return options.UseMappingSchema(SharpCoreDBMappingSchema.Instance); + } + + if (current.ConfigurationList.Contains(SharpCoreDBMappingSchema.ConfigurationName)) + { + return options; + } + + return options.UseMappingSchema(new MappingSchema(current, SharpCoreDBMappingSchema.Instance)); } /// diff --git a/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBMappingSchema.cs b/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBMappingSchema.cs index 51d58fdf..9641624e 100644 --- a/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBMappingSchema.cs +++ b/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBMappingSchema.cs @@ -1,4 +1,5 @@ using System; +using System.Globalization; using LinqToDB; using LinqToDB.Data; using LinqToDB.Mapping; @@ -14,12 +15,18 @@ namespace SharpCoreDB.Functional.Linq2DB; /// public sealed class SharpCoreDBMappingSchema : MappingSchema { + /// + /// The configuration name of this schema; it identifies the schema inside a combined one, so the adapter adds + /// it to a caller's options exactly once. + /// + internal const string ConfigurationName = "SharpCoreDB"; + /// /// Gets the singleton instance of the SharpCoreDB mapping schema. /// public static readonly SharpCoreDBMappingSchema Instance = new(); - private SharpCoreDBMappingSchema() : base("SharpCoreDB") + private SharpCoreDBMappingSchema() : base(ConfigurationName) { // === ULID Value Converter (critical for SQLite/linq2db parameter binding) === // Simple SetConverter is insufficient for INSERT/parameter binding. @@ -42,22 +49,26 @@ private SharpCoreDBMappingSchema() : base("SharpCoreDB") : new DataParameter(null, DBNull.Value, DataType.NVarChar)); SetConverter(str => string.IsNullOrWhiteSpace(str) ? null : Ulid.Parse(str)); - // GUIDs as compact string. A DataParameter (like the ULID one below) is what makes the SQLite provider - // render the value as text: with only the CLR converter registered the parameter kept DataType.Guid and - // the provider rendered it as a 16-byte blob literal (X'…'), which a TEXT column then stored as the - // literal text and could not read back into a Guid. + // GUIDs as text in the hyphenated "D" form. A DataParameter (like the ULID one above) is what makes the SQLite + // provider render the value as text: with only the CLR converter registered the parameter kept + // DataType.Guid and the provider rendered it as a 16-byte blob literal (X'…'), which a TEXT column then + // stored as the literal text and could not read back into a Guid. Guid → string must use the same form as + // the parameter — up to 2.1.4 it used "N", so a value converted that way never matched a stored one. SetConverter(guid => new DataParameter(null, guid.ToString("D"), DataType.NVarChar)); - SetConverter(guid => guid.ToString("N")); + SetConverter(guid => guid.ToString("D")); SetConverter(str => Guid.Parse(str)); - // DateTime as ISO string - SetConverter(dt => dt.ToString("O")); - SetConverter(str => DateTime.Parse(str)); + // Dates are read and written with the invariant culture: up to 2.1.4 the current culture parsed them, so + // under a non-Gregorian calendar (th-TH) the stored year 2026 read back as 1483. RoundtripKind keeps the + // DateTimeKind that an "O" string carries ("Z" stays UTC) instead of converting it to local time. + SetConverter(dt => dt.ToString("O", CultureInfo.InvariantCulture)); + SetConverter(str => + DateTime.Parse(str, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind)); // DateTimeOffset - SetConverter(dto => dto.ToString("O")); - SetConverter(str => DateTimeOffset.Parse(str)); + SetConverter(dto => dto.ToString("O", CultureInfo.InvariantCulture)); + SetConverter(str => DateTimeOffset.Parse(str, CultureInfo.InvariantCulture)); // Boolean <-> integer for SQLite compatibility SetConverter(b => b ? 1 : 0); @@ -77,6 +88,8 @@ private SharpCoreDBMappingSchema() : base("SharpCoreDB") /// Target type /// Conversion function to target /// Conversion function to source + [Obsolete("Changes the process-wide Instance shared by every connection. Register the converters on your own " + + "MappingSchema and add it with DataOptions.UseAdditionalMappingSchema; it takes priority over these defaults.")] public void ConfigureCustomMapping( Func toTarget, Func toSource) diff --git a/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBDataProvider.cs b/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBProviderExtensions.cs similarity index 96% rename from src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBDataProvider.cs rename to src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBProviderExtensions.cs index e02697e5..8da63afa 100644 --- a/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBDataProvider.cs +++ b/src/SharpCoreDB.Functional.Linq2DB/SharpCoreDBProviderExtensions.cs @@ -1,6 +1,4 @@ using LinqToDB; -using LinqToDB.Data; -using LinqToDB.Mapping; namespace SharpCoreDB.Functional.Linq2DB; diff --git a/src/SharpCoreDB/DataStructures/HashIndex.cs b/src/SharpCoreDB/DataStructures/HashIndex.cs index 7b50999e..c8263674 100644 --- a/src/SharpCoreDB/DataStructures/HashIndex.cs +++ b/src/SharpCoreDB/DataStructures/HashIndex.cs @@ -106,7 +106,7 @@ internal void Add(object key, long position) { // Unique path: atomic check + add under outer lock var keyBytes = BuildUnsafeKey(normalizedKey); - if (HasUnsafeRowsForKey(keyBytes)) + if (normalizedKey is not DBNull && HasUnsafeRowsForKey(keyBytes)) { throw new InvalidOperationException( $"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'"); @@ -122,7 +122,7 @@ internal void Add(object key, long position) list = []; _index[normalizedKey] = list; } - else if (_isUnique && list.Count > 0) + else if (_isUnique && list.Count > 0 && normalizedKey is not DBNull) { throw new InvalidOperationException( $"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'"); @@ -463,7 +463,7 @@ private void AddBatchKeysLockedCore(object?[] keys, long[] positions) { // Unique path: atomic check + add under outer lock. var keyBytes = BuildUnsafeKey(normalizedKey); - if (HasUnsafeRowsForKey(keyBytes)) + if (normalizedKey is not DBNull && HasUnsafeRowsForKey(keyBytes)) { throw new InvalidOperationException( $"Duplicate key value '{keys[i]}' violates unique constraint on index '{_columnName}'"); @@ -479,7 +479,7 @@ private void AddBatchKeysLockedCore(object?[] keys, long[] positions) list = []; _index[normalizedKey] = list; } - else if (_isUnique && list.Count > 0) + else if (_isUnique && list.Count > 0 && normalizedKey is not DBNull) { throw new InvalidOperationException( $"Duplicate key value '{keys[i]}' violates unique constraint on index '{_columnName}'"); @@ -552,7 +552,7 @@ public void AddBatch(List> rows, long[] positions) { // Unique path: atomic check + add under outer lock var keyBytes = BuildUnsafeKey(normalizedKey); - if (HasUnsafeRowsForKey(keyBytes)) + if (normalizedKey is not DBNull && HasUnsafeRowsForKey(keyBytes)) { throw new InvalidOperationException( $"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'"); @@ -567,7 +567,7 @@ public void AddBatch(List> rows, long[] positions) list = []; _index[normalizedKey] = list; } - else if (_isUnique && list.Count > 0) + else if (_isUnique && list.Count > 0 && normalizedKey is not DBNull) { throw new InvalidOperationException( $"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'"); diff --git a/src/SharpCoreDB/DataStructures/Table.BatchUpdate.cs b/src/SharpCoreDB/DataStructures/Table.BatchUpdate.cs index 7d0e2933..703c3b4b 100644 --- a/src/SharpCoreDB/DataStructures/Table.BatchUpdate.cs +++ b/src/SharpCoreDB/DataStructures/Table.BatchUpdate.cs @@ -637,15 +637,18 @@ private byte[] SerializeRowOptimized(Dictionary row) /// Helper method to format a value for SQL WHERE clause. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static string FormatValue(object? value) + internal static string FormatValue(object? value) { return value switch { null => "NULL", string s => $"'{s.Replace("'", "''")}'", bool b => b ? "1" : "0", - DateTime dt => $"'{dt:yyyy-MM-dd HH:mm:ss}'", - _ => value.ToString() ?? "NULL", + DateTime dt => SqlLiteralFormatter.Literal(dt), + DateTimeOffset dto => SqlLiteralFormatter.Literal(dto), + DateOnly date => SqlLiteralFormatter.Literal(date), + TimeOnly time => SqlLiteralFormatter.Literal(time), + _ => SqlLiteralFormatter.Invariant(value), }; } diff --git a/src/SharpCoreDB/DataStructures/Table.CRUD.cs b/src/SharpCoreDB/DataStructures/Table.CRUD.cs index 3c9a40ee..4f49ba31 100644 --- a/src/SharpCoreDB/DataStructures/Table.CRUD.cs +++ b/src/SharpCoreDB/DataStructures/Table.CRUD.cs @@ -98,21 +98,6 @@ public void Insert(Dictionary row) } } - // ✅ UNIQUE validation (outside lock) - foreach (var uniqueConstraint in this.UniqueConstraints) - { - if (uniqueConstraint.Count == 1) // Single column unique - { - var colName = uniqueConstraint[0]; - var colIndex = this.Columns.IndexOf(colName); - if (colIndex >= 0 && row.TryGetValue(colName, out var value) && value != null && value != DBNull.Value) - { - // Check if value already exists (simplified - would need index lookup in real impl) - // For now, just validate non-null for single column unique - } - } - } - // ✅ CHECK constraint validation (outside lock) for (int i = 0; i < this.Columns.Count; i++) { @@ -189,7 +174,7 @@ public void Insert(Dictionary row) continue; } - if (!row.TryGetValue(registeredCol, out var value) || value is null) + if (!row.TryGetValue(registeredCol, out var value) || value is null or DBNull) { continue; } @@ -202,6 +187,10 @@ public void Insert(Dictionary row) } } + // UNIQUE columns and unique indexes, on every storage mode (the check above runs for columnar tables + // only and skips an index it has not loaded). Returns at once for a table without unique indexes. + ValidateUniqueValues(row); + // ✅ NEW: Route through storage engine var engine = GetOrCreateStorageEngine(); @@ -278,8 +267,8 @@ public void Insert(Dictionary row) /// this table — i.e. when nothing in the table's schema relies on the per-row-only semantics that the /// batch core does not implement. /// - /// What the batch core does NOT do: CHECK-constraint evaluation (per-column and table-level) and non-PK - /// unique-index enforcement. PK and NOT NULL validation are implemented on both paths, and the + /// What the batch core does NOT do: CHECK-constraint evaluation (per-column and table-level). PK, NOT NULL and + /// UNIQUE (column constraints and unique indexes) validation are implemented on both paths, and the /// batch core joins an open engine transaction rather than nesting one /// (InsertBatchCriticalSection: needsTransaction = !engine.IsInTransaction). /// @@ -307,14 +296,6 @@ internal bool CanUseBatchedInsert } } - foreach (var metadata in this.registeredIndexes.Values) - { - if (metadata.IsUnique) - { - return false; - } - } - return _database?.IsBatchUpdateActive != true; } } @@ -802,6 +783,9 @@ private long[] InsertBatchCriticalSection(object[][] validatedRows, List // Validate primary keys (requires lock for index access) ValidateExistingPrimaryKeys(validatedRows); + // UNIQUE columns and unique indexes: up to 2.1.4 the batched path checked neither. + ValidateUniqueValues(validatedRows); + var engine = GetOrCreateStorageEngine(); bool needsTransaction = !engine.IsInTransaction; @@ -882,6 +866,9 @@ private long[] InsertBatchCriticalSection( List> validatedRows, List serializedRows) { + // UNIQUE columns and unique indexes: up to 2.1.4 the batched path checked neither. + ValidateUniqueValues(validatedRows); + // Validate primary keys (requires lock for index access) if (this.PrimaryKeyIndex >= 0) { @@ -1778,6 +1765,13 @@ public int UpdateAffectedCount(string? where, Dictionary updates var rows = ResolveUpdateRows(where); int affected = 0; + // A new PRIMARY KEY or UNIQUE value must not be another row's, and is checked before any row is written. + // Up to 2.1.4 a primary key moved onto an existing key re-pointed the index and lost the original row. + if (rows.Count > 0 && UpdateTouchesKeyColumns(updates)) + { + ValidateKeyChanges([(rows.ConvertAll(r => r.Row), updates)]); + } + foreach (var (rowPos, row) in rows) { affected++; @@ -2510,6 +2504,9 @@ private int UpdateMultipleCore( // put an atomic per operation into the loop it measures. long structuredOps = 0; + // Key changes are validated for the whole batch before any operation is written (known issues §8/§9). + ValidateBatchKeyChanges(operations); + foreach (var (where, structuredKeyColumn, structuredKeyValue, structuredKeyLiteral, updates) in operations) { // §5.3 session 2 (2026-09-26): the per-operation head of this loop — key resolution, the @@ -5326,6 +5323,11 @@ public bool UpdateByPrimaryKey(object key, Dictionary updates) if (row == null) return false; + if (UpdateTouchesKeyColumns(updates)) + { + ValidateKeyChanges([([row], updates)]); + } + // WP13: capture only what index maintenance needs instead of copying the whole row. Dictionary? oldHashKeys = null; foreach (var kvp in this.hashIndexes) // NOSONAR:S3267 - deliberate: LINQ Select/Where would allocate per point-update on the hot path diff --git a/src/SharpCoreDB/DataStructures/Table.Scanning.cs b/src/SharpCoreDB/DataStructures/Table.Scanning.cs index 9f0f0288..6cf66e3e 100644 --- a/src/SharpCoreDB/DataStructures/Table.Scanning.cs +++ b/src/SharpCoreDB/DataStructures/Table.Scanning.cs @@ -263,6 +263,12 @@ private static bool TryEvaluateTruthTest(Dictionary row, List + /// Evaluates against a row with this table's WHERE semantics — the evaluator + /// ExecuteQuery uses. Database.ExecuteQueryFast calls it for its StructRow scan. + /// + internal bool MatchesWhere(Dictionary row, string? where) => EvaluateWhere(row, where); + private bool EvaluateWhere(Dictionary row, string? where) { if (string.IsNullOrEmpty(where)) return true; @@ -311,6 +317,14 @@ private bool EvaluateWhere(Dictionary row, string? where) } } + // A computed operand — strftime('%Y', at) = '2026', length(name) > 3, CAST(… AS INTEGER) = 5 — is evaluated + // by the shared scalar evaluator. Below, the left token is read as a column name, so such a condition used to + // match no row: every date filter linq2db's SQLite dialect writes takes this shape. + if (SqlScalarExpression.TryEvaluateCondition(row, where, out var computedMatch)) + { + return computedMatch; + } + // ✅ Issue #339/#340: support IN / NOT IN lists for all column types, including // SQLite VALUES forms and tuple predicates. Previously the list was split on spaces // (losing everything after the first comma when the SQL contains spaces, e.g. diff --git a/src/SharpCoreDB/DataStructures/Table.UniqueKeys.cs b/src/SharpCoreDB/DataStructures/Table.UniqueKeys.cs new file mode 100644 index 00000000..a72f9cc5 --- /dev/null +++ b/src/SharpCoreDB/DataStructures/Table.UniqueKeys.cs @@ -0,0 +1,342 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.DataStructures; + +using System; +using System.Collections.Generic; +using System.Globalization; + +/// +/// Table partial — uniqueness of the PRIMARY KEY and of single-column UNIQUE constraints on every write path. +/// +/// Up to 2.1.4 two guarantees were missing (known issues §8 and §9). An UPDATE that changed the primary key never +/// checked the new key: UPDATE t SET id = 1 WHERE id = 3 with a row 1 present was accepted, the primary-key +/// index was re-pointed at the moved row and the original row 1 became unreachable. And a UNIQUE column constraint was +/// parsed and stored but never enforced — the INSERT-time check was a stub, a UNIQUE index was checked only on +/// a single-row INSERT into a columnar table, and multi-row INSERTs and UPDATEs were never checked. +/// +/// +/// A UNIQUE column now gets an automatic unique hash index, as SQLite gives it a sqlite_autoindex_…, so the +/// check is a hash lookup. Every check starts with a test that allocates nothing and returns at once for a table or a +/// statement without keys to protect, which keeps the measured write paths unchanged. +/// +/// +public partial class Table +{ + /// + /// Registers an automatic unique hash index for every single-column UNIQUE constraint that has none — at CREATE TABLE + /// and on every open, so tables created before this existed are covered too. Registration does not build anything: + /// the index is built from the stored rows on first use, and rows stored before (duplicates included) never make + /// registration fail; only a new duplicate is refused. + /// + internal void EnsureUniqueConstraintIndexes() + { + for (int i = 0; i < this.UniqueConstraints.Count; i++) + { + var constraint = this.UniqueConstraints[i]; + if (constraint.Count != 1) + { + continue; + } + + var column = this.Columns.Find(c => c.Equals(constraint[0], StringComparison.OrdinalIgnoreCase)); + if (column is null || (this.PrimaryKeyIndex >= 0 && column == this.Columns[this.PrimaryKeyIndex])) + { + continue; + } + + var indexName = $"sqlite_autoindex_{this.Name}_{i + 1}"; + if (this.indexNameToColumn.ContainsKey(indexName)) + { + continue; + } + + CreateHashIndex(indexName, column, isUnique: true); + } + } + + /// Returns true when any unique index (other than the primary key's) is registered. Allocates nothing. + private bool HasUniqueIndex() + { + foreach (var (column, metadata) in + this.registeredIndexes) // NOSONAR:S3267 - write hot path: LINQ would allocate + { + if (metadata.IsUnique && !IsPrimaryKeyColumn(column)) + { + return true; + } + } + + return false; + } + + /// + /// Returns true when sets the primary key or a uniquely indexed column. Allocates + /// nothing, so every UPDATE path can ask it first. + /// + private bool UpdateTouchesKeyColumns(Dictionary updates) + { + if (this.PrimaryKeyIndex >= 0 && TryGetUpdateValue(updates, this.Columns[this.PrimaryKeyIndex], out _)) + { + return true; + } + + foreach (var (column, metadata) in + this.registeredIndexes) // NOSONAR:S3267 - write hot path: LINQ would allocate + { + if (metadata.IsUnique && !IsPrimaryKeyColumn(column) && TryGetUpdateValue(updates, column, out _)) + { + return true; + } + } + + return false; + } + + // The three entry points below take the no-unique-index exit before anything is captured: a lambda that captures a + // parameter makes the compiler allocate its closure at method entry, so the capturing calls live in the *Core + // helpers and an INSERT into a table without unique indexes allocates nothing here. + + /// Validates one inserted row against the unique indexes. Returns at once without any. + private void ValidateUniqueValues(Dictionary row) + { + if (HasUniqueIndex()) + { + ValidateUniqueValuesCore(row); + } + } + + /// Validates inserted rows (one statement) against the unique indexes and against each other. + private void ValidateUniqueValues(IReadOnlyList> rows) + { + if (HasUniqueIndex()) + { + ValidateUniqueValuesCore(rows); + } + } + + /// Validates inserted rows in table column order (the batch parser's shape). + private void ValidateUniqueValues(object[][] rows) + { + if (HasUniqueIndex()) + { + ValidateUniqueValuesCore(rows); + } + } + + private void ValidateUniqueValuesCore(Dictionary row) => + ValidateUniqueValues(1, (_, column) => row.TryGetValue(column, out var value) ? value : null); + + private void ValidateUniqueValuesCore(IReadOnlyList> rows) => + ValidateUniqueValues(rows.Count, (r, column) => rows[r].TryGetValue(column, out var value) ? value : null); + + private void ValidateUniqueValuesCore(object[][] rows) => + ValidateUniqueValues(rows.Length, (r, column) => + { + int index = this.Columns.IndexOf(column); + return index >= 0 && index < rows[r].Length ? rows[r][index] : null; + }); + + private void ValidateUniqueValues(int rowCount, Func valueAt) + { + foreach (var column in UniqueIndexedColumns()) + { + EnsureIndexLoaded(column); + this.hashIndexes.TryGetValue(column, out var hashIndex); + var seen = new HashSet(UniqueKeyComparer(column)); + for (int r = 0; r < rowCount; r++) + { + var value = valueAt(r, column); + if (value is null or DBNull) + { + continue; // NULLs never conflict, as in SQLite + } + + if (!seen.Add(UniqueKey(value)) || (hashIndex is not null && hashIndex.ContainsKey(value))) + { + throw DuplicateUniqueValue(column, value); + } + } + } + } + + /// + /// Validates a statement's (or a batch's) key changes before any row is written: a primary key or a uniquely indexed + /// column may only take a value no other row holds — counting the values earlier changes in the same batch claim and + /// release — and a SET of such a column may match at most one row. An UPDATE that keeps a row's own value passes. + /// + /// Each change's matched rows and SET values, in execution order. + private void ValidateKeyChanges( + IReadOnlyList<(IReadOnlyList> Rows, Dictionary Updates)> changes) + { + if (this.PrimaryKeyIndex >= 0) + { + var pkColumn = this.Columns[this.PrimaryKeyIndex]; + ValidateKeyColumnChanges(changes, pkColumn, StringComparer.Ordinal, (key, _) => IsPrimaryKeyTaken(key), + isPrimaryKey: true); + } + + foreach (var column in UniqueIndexedColumns()) + { + EnsureIndexLoaded(column); + this.hashIndexes.TryGetValue(column, out var hashIndex); + ValidateKeyColumnChanges( + changes, + column, + UniqueKeyComparer(column), + (_, value) => hashIndex is not null && hashIndex.ContainsKey(value), + isPrimaryKey: false); + } + } + + private void ValidateKeyColumnChanges( + IReadOnlyList<(IReadOnlyList> Rows, Dictionary Updates)> changes, + string column, + StringComparer comparer, + Func isTaken, + bool isPrimaryKey) + { + var claimed = new HashSet(comparer); + var released = new HashSet(comparer); + foreach (var (rows, updates) in changes) + { + if (rows.Count == 0 || !TryGetUpdateValue(updates, column, out var newValue) || newValue is null or DBNull) + { + continue; + } + + var newKey = UniqueKey(newValue); + if (rows.Count > 1) + { + throw isPrimaryKey + ? new InvalidOperationException( + $"Primary key violation: the UPDATE would give {rows.Count} rows the key {newKey}") + : DuplicateUniqueValue(column, newValue); + } + + var oldValue = rows[0].TryGetValue(column, out var current) ? current : null; + var oldKey = oldValue is null or DBNull ? null : UniqueKey(oldValue); + if (oldKey is not null && comparer.Equals(oldKey, newKey)) + { + continue; // the row keeps its own value + } + + if (!claimed.Add(newKey) || (isTaken(newKey, newValue) && !released.Contains(newKey))) + { + throw isPrimaryKey + ? new InvalidOperationException($"Primary key violation: {newKey}") + : DuplicateUniqueValue(column, newValue); + } + + if (oldKey is not null) + { + released.Add(oldKey); + } + } + } + + /// + /// Validates the key changes of a batched UPDATE (UpdateMultipleCore, the path ExecuteBatchSQL and the + /// structured batch API take) before any operation is written. The scan for an operation that touches a key column + /// allocates nothing; only those operations have their rows resolved. + /// + private void ValidateBatchKeyChanges( + IReadOnlyList<(string Where, string? KeyColumn, object? KeyValue, string? KeyLiteral, Dictionary + Updates)> operations) + { + for (int i = 0; i < operations.Count; i++) + { + if (UpdateTouchesKeyColumns(operations[i].Updates)) + { + ValidateBatchKeyChangesCore(operations); + return; + } + } + } + + private void ValidateBatchKeyChangesCore( + IReadOnlyList<(string Where, string? KeyColumn, object? KeyValue, string? KeyLiteral, Dictionary + Updates)> operations) + { + var changes = new List<(IReadOnlyList> Rows, Dictionary Updates)>(); + foreach (var (where, keyColumn, keyValue, keyLiteral, updates) in operations) + { + if (!UpdateTouchesKeyColumns(updates)) + { + continue; + } + + // The same WHERE text UpdateMultipleCore falls back to for a structured operation. + var whereText = where.Length > 0 || keyColumn is null + ? where + : keyLiteral is not null + ? keyColumn + " = " + keyLiteral + : BuildStructuredWhereText(keyColumn, keyValue); + + changes.Add((ResolveUpdateRows(whereText).ConvertAll(r => r.Row), updates)); + } + + ValidateKeyChanges(changes); + } + + /// The uniquely indexed columns other than the primary key (which has its own index and check). + private List UniqueIndexedColumns() + { + var columns = new List(); + foreach (var (column, metadata) in this.registeredIndexes) + { + if (metadata.IsUnique && !IsPrimaryKeyColumn(column)) + { + columns.Add(column); + } + } + + return columns; + } + + private bool IsPrimaryKeyColumn(string column) => + this.PrimaryKeyIndex >= 0 && + column.Equals(this.Columns[this.PrimaryKeyIndex], StringComparison.OrdinalIgnoreCase); + + /// Compares unique keys with the column's collation: case-insensitively for NOCASE columns. + private StringComparer UniqueKeyComparer(string column) + { + int index = this.Columns.IndexOf(column); + var collation = index >= 0 && index < this.ColumnCollations.Count + ? this.ColumnCollations[index] + : CollationType.Binary; + return collation is CollationType.NoCase or CollationType.UnicodeCaseInsensitive + ? StringComparer.OrdinalIgnoreCase + : StringComparer.Ordinal; + } + + private static string UniqueKey(object value) => + value as string ?? Convert.ToString(value, CultureInfo.InvariantCulture) ?? string.Empty; + + private static bool TryGetUpdateValue(Dictionary updates, string column, out object? value) + { + if (updates.TryGetValue(column, out var exact)) + { + value = exact; + return true; + } + + foreach (var (key, candidate) in updates) // NOSONAR:S3267 - SET lists are tiny; LINQ would allocate per call + { + if (key.Equals(column, StringComparison.OrdinalIgnoreCase)) + { + value = candidate; + return true; + } + } + + value = null; + return false; + } + + private static InvalidOperationException DuplicateUniqueValue(string column, object value) => + new($"Duplicate key value '{value}' violates unique constraint on column '{column}'"); +} diff --git a/src/SharpCoreDB/Database/Core/Database.Core.cs b/src/SharpCoreDB/Database/Core/Database.Core.cs index 52012fed..34f03cd5 100644 --- a/src/SharpCoreDB/Database/Core/Database.Core.cs +++ b/src/SharpCoreDB/Database/Core/Database.Core.cs @@ -468,6 +468,10 @@ config is { } fixedWidthConfig && { table.DefaultExpressions.Add(null); } + + // UNIQUE constraints are enforced through automatic unique indexes; tables created before they existed + // get theirs here. Registration builds nothing, so rows stored earlier never block an open. + table.EnsureUniqueConstraintIndexes(); // ✅ Phase 2: Backward compatible — missing column check expressions default to null // REQUIRED: Table.Insert indexes ColumnCheckExpressions[i] without bounds checks, // so this list MUST be padded to the column count after deserialization. diff --git a/src/SharpCoreDB/Database/Execution/Database.PerformanceOptimizations.cs b/src/SharpCoreDB/Database/Execution/Database.PerformanceOptimizations.cs index 9f01e57e..5be0dfb8 100644 --- a/src/SharpCoreDB/Database/Execution/Database.PerformanceOptimizations.cs +++ b/src/SharpCoreDB/Database/Execution/Database.PerformanceOptimizations.cs @@ -41,16 +41,6 @@ public partial class Database RegexOptions.IgnoreCase | RegexOptions.Compiled, TimeSpan.FromSeconds(1)); - /// - /// Cache for compiled WHERE clause expressions. - /// Eliminates re-parsing overhead for repeated queries. - /// - /// Performance: 50-100x faster for repeated WHERE clauses. - /// Typical improvement: 0.5ms → 0.01ms per query. - /// - private static readonly LruCache, bool>> - WhereClauseExpressionCache = new LruCache, bool>>(1000); - /// /// Fast path for SELECT * queries using StructRow. /// Avoids Dictionary materialization, 25x less memory. @@ -120,59 +110,33 @@ public List ExecuteQueryFast(string sql) } else { - // With WHERE clause: use WHERE cache but don't materialize to Dictionary - // This is still more efficient than ExecuteQuery() because: - // 1. StructRow avoids Dictionary allocation per row - // 2. WHERE predicate is cached - // 3. Only materializes values that match WHERE conditions - - var dictionaryPredicate = GetOrCompileWhereClause(whereClause); - + // With a WHERE clause the result is still StructRows; only the WHERE evaluation materialises a row's + // values — by column name, through the table's own evaluator, so the rows match ExecuteQuery's. return actualTable.SelectStructWhere( - row => EvaluateWhereOnStructRow(row, dictionaryPredicate, actualTable), + row => EvaluateWhereOnStructRow(row, whereClause, actualTable), enableCaching: false) .ToList(); } } - + /// - /// Helper to evaluate a WHERE predicate on StructRow by converting to temporary Dictionary. - /// This is still more efficient than the full Dictionary path because we only materialize - /// the rows that match the WHERE clause. + /// Evaluates the WHERE clause against a StructRow with the table's own WHERE evaluator — the one ExecuteQuery uses + /// (IN, LIKE, collations, functions over columns). + /// + /// Up to 2.1.4 the values were keyed col0, col1, … and handed to a separately compiled predicate: + /// a condition naming a column matched no row, and one the compiler could not parse accepted every row. + /// /// - private static bool EvaluateWhereOnStructRow( - StructRow row, - Func, bool> wherePredicate, - Table table) + private static bool EvaluateWhereOnStructRow(StructRow row, string whereClause, Table table) { - // Create a minimal temporary Dictionary just for WHERE evaluation - var tempRow = new Dictionary(); - - // We need column names - get them from the table - // For now, use a simple approach: enumerate columns and add their values - try - { - // Try to get columns 0-N until we hit an error - for (int i = 0; i < 256; i++) // Reasonable upper limit - { - try - { - var value = row.GetValue(i); - tempRow[$"col{i}"] = value ?? DBNull.Value; - } - catch - { - break; // No more columns - } - } - } - catch + var names = row.GetColumnNames(); + var values = new Dictionary(names.Length, StringComparer.OrdinalIgnoreCase); + for (int i = 0; i < names.Length; i++) { - // If something goes wrong, just accept the row - return true; + values[names[i]] = row.IsNull(i) ? DBNull.Value : row.GetValueBoxed(i); } - - return wherePredicate(tempRow); + + return table.MatchesWhere(values, whereClause); } /// @@ -206,120 +170,4 @@ public ValueTask InsertAsyncOptimized(string tableName, Dictionary - /// Gets WHERE clause compiled expression from cache. - /// Returns cached compiled predicate or compiles new one. - /// - /// Performance: Cache hit rate > 80% typical for OLTP. - /// First query: 0.5ms (parsing + compilation). - /// Subsequent: 0.01ms (cache lookup only). - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - internal Func, bool> GetOrCompileWhereClause(string whereClause) - { - ArgumentNullException.ThrowIfNull(whereClause); - - whereClause = whereClause.Trim(); - - // Empty WHERE clause = no filtering - if (string.IsNullOrEmpty(whereClause)) - { - return row => true; - } - - // Try cache first - if (WhereClauseExpressionCache.TryGetValue(whereClause, out var cached)) - { - return cached; - } - - // Cache miss: Compile new predicate - var compiled = SqlParserPerformanceOptimizations.CompileWhereClause(whereClause); - - // Store in cache for future use - WhereClauseExpressionCache.GetOrAdd(whereClause, _ => compiled); - - return compiled; - } - - /// - /// Clear WHERE clause cache (on schema changes). - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - internal void ClearWhereClauseCache() - { - WhereClauseExpressionCache.Clear(); - } -} - -/// -/// LRU Cache for compiled expressions. -/// Simple thread-safe cache with capacity limit. -/// -internal sealed class LruCache where TKey : notnull -{ - private readonly int _capacity; - private readonly Dictionary _cache; - private long _currentTimestamp; - private readonly Lock _lock = new(); - - public LruCache(int capacity) - { - _capacity = capacity; - _cache = new Dictionary(capacity); - } - - public bool TryGetValue(TKey key, out TValue value) - { - lock (_lock) - { - if (_cache.TryGetValue(key, out var entry)) - { - // Update timestamp for LRU - _cache[key] = (entry.value, ++_currentTimestamp); - value = entry.value; - return true; - } - - value = default; - return false; - } - } - - public TValue GetOrAdd(TKey key, Func factory) - { - lock (_lock) - { - if (_cache.TryGetValue(key, out var entry)) - { - _cache[key] = (entry.value, ++_currentTimestamp); - return entry.value; - } - - var newValue = factory(key); - - // Evict oldest if at capacity - if (_cache.Count >= _capacity) - { - var oldestKey = _cache - .OrderBy(x => x.Value.timestamp) - .First() - .Key; - _cache.Remove(oldestKey); - } - - _cache[key] = (newValue, ++_currentTimestamp); - return newValue; - } - } - - public void Clear() - { - lock (_lock) - { - _cache.Clear(); - _currentTimestamp = 0; - } - } } diff --git a/src/SharpCoreDB/Services/CompiledQueryExecutor.cs b/src/SharpCoreDB/Services/CompiledQueryExecutor.cs index 105be498..ed672429 100644 --- a/src/SharpCoreDB/Services/CompiledQueryExecutor.cs +++ b/src/SharpCoreDB/Services/CompiledQueryExecutor.cs @@ -340,26 +340,30 @@ private static string BindParametersInWhereClause(string whereClause, Dictionary /// /// Formats a parameter value for SQL substitution. /// - private static string FormatParameterValue(object? value) + internal static string FormatParameterValue(object? value) { if (value == null) return "NULL"; return value switch { string s => $"'{s.Replace("'", "''")}'", - int i => i.ToString(), - long l => l.ToString(), + int i => i.ToString(System.Globalization.CultureInfo.InvariantCulture), + long l => l.ToString(System.Globalization.CultureInfo.InvariantCulture), double d => d.ToString(System.Globalization.CultureInfo.InvariantCulture), + float f => f.ToString(System.Globalization.CultureInfo.InvariantCulture), decimal m => m.ToString(System.Globalization.CultureInfo.InvariantCulture), bool b => b ? "1" : "0", - DateTime dt => $"'{dt:yyyy-MM-dd HH:mm:ss}'", + DateTime dt => SqlLiteralFormatter.Literal(dt), + DateTimeOffset dto => SqlLiteralFormatter.Literal(dto), + DateOnly date => SqlLiteralFormatter.Literal(date), + TimeOnly time => SqlLiteralFormatter.Literal(time), // Same vector-literal rendering as ParameterBinder.FormatParameter, so both binders agree. float[] vector => VectorBinaryCodec.ToJson(vector), ReadOnlyMemory vectorMemory => VectorBinaryCodec.ToJson(vectorMemory.Span), Memory vectorBuffer => VectorBinaryCodec.ToJson(vectorBuffer.Span), double[] vectorDoubles => VectorBinaryCodec.ToJson(Array.ConvertAll(vectorDoubles, static d => (float)d)), - _ => $"'{value.ToString()?.Replace("'", "''") ?? string.Empty}'" + _ => $"'{SqlLiteralFormatter.Invariant(value).Replace("'", "''")}'" }; } diff --git a/src/SharpCoreDB/Services/PreparedStatements.cs b/src/SharpCoreDB/Services/PreparedStatements.cs index 26877302..f89a5033 100644 --- a/src/SharpCoreDB/Services/PreparedStatements.cs +++ b/src/SharpCoreDB/Services/PreparedStatements.cs @@ -446,12 +446,15 @@ internal static string FormatParameter(object? value) { string str => $"'{EscapeSqlString(str)}'", bool b => b ? "1" : "0", - DateTime dt => $"'{dt:yyyy-MM-dd HH:mm:ss.fff}'", + DateTime dt => SqlLiteralFormatter.Literal(dt), + DateTimeOffset dto => SqlLiteralFormatter.Literal(dto), + DateOnly date => SqlLiteralFormatter.Literal(date), + TimeOnly time => SqlLiteralFormatter.Literal(time), decimal d => d.ToString(System.Globalization.CultureInfo.InvariantCulture), double d => d.ToString(System.Globalization.CultureInfo.InvariantCulture), float f => f.ToString(System.Globalization.CultureInfo.InvariantCulture), - int i => i.ToString(), - long l => l.ToString(), + int i => i.ToString(System.Globalization.CultureInfo.InvariantCulture), + long l => l.ToString(System.Globalization.CultureInfo.InvariantCulture), // Vector parameters are rendered as the bracketed literal the parser reads back into a float[] // (VectorBinaryCodec), and the VALUES/SET splitters keep that literal in one piece. Without this a @@ -461,7 +464,7 @@ internal static string FormatParameter(object? value) Memory vectorBuffer => VectorBinaryCodec.ToJson(vectorBuffer.Span), double[] vectorDoubles => VectorBinaryCodec.ToJson(Array.ConvertAll(vectorDoubles, static d => (float)d)), byte[] bytes => $"0x{System.Convert.ToHexString(bytes)}", - _ => $"'{EscapeSqlString(value.ToString())}'", + _ => $"'{EscapeSqlString(SqlLiteralFormatter.Invariant(value))}'", }; } diff --git a/src/SharpCoreDB/Services/SqlDateTimeFunctions.cs b/src/SharpCoreDB/Services/SqlDateTimeFunctions.cs new file mode 100644 index 00000000..a14aa108 --- /dev/null +++ b/src/SharpCoreDB/Services/SqlDateTimeFunctions.cs @@ -0,0 +1,369 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Services; + +using System; +using System.Globalization; +using System.Text; +using System.Text.RegularExpressions; + +/// +/// SQLite's date and time functions — date, time, datetime, julianday, +/// unixepoch and strftime — over evaluated argument values, so they work on a column as well +/// as on a literal. +/// +/// Up to 2.1.4 only a SELECT without FROM evaluated julianday, unixepoch and strftime, from +/// the raw argument text; date, time and datetime did not exist, a ±HH:MM suffix was not +/// applied, and an unreadable date threw instead of giving NULL. Semantics follow +/// https://sqlite.org/lang_datefunc.html: a time value is a time string, now, or a number read as a Julian +/// day (or as Unix seconds after the unixepoch modifier); a time string without a zone is UTC; an invalid +/// value or modifier yields NULL; and adding months or years keeps the day number and lets it overflow +/// (2026-01-31 +1 month is 2026-03-03). +/// +/// +internal static class SqlDateTimeFunctions +{ + private const double UnixEpochJulianDay = 2440587.5; + + private static readonly DateTime UnixEpoch = new(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc); + + private static readonly Regex TimeStringPattern = new( + @"^(?:(?\d{4})-(?\d{2})-(?\d{2}))?(?:(?:^|[ T])(?\d{2}):(?\d{2})(?::(?\d{2})(?:\.(?\d+))?)?)?\s*(?[Zz]|[+-]\d{2}:\d{2})?$", + RegexOptions.CultureInvariant | RegexOptions.Compiled, + TimeSpan.FromSeconds(1)); + + private static readonly Regex ArithmeticModifierPattern = new( + @"^(?[+-]?\d+(?:\.\d+)?)\s+(?day|hour|minute|second|month|year)s?$", + RegexOptions.CultureInvariant | RegexOptions.IgnoreCase | RegexOptions.Compiled, + TimeSpan.FromSeconds(1)); + + /// Returns true for the names this class evaluates. + internal static bool IsDateTimeFunction(string upperName) => upperName is + "DATE" or "TIME" or "DATETIME" or "JULIANDAY" or "UNIXEPOCH" or "STRFTIME"; + + /// + /// Evaluates one of the date/time functions. Returns NULL (null) when SQLite does: a missing or unreadable + /// time value, an unknown modifier, or an unknown strftime conversion. + /// + /// The upper-case function name. + /// The evaluated arguments. + internal static object? Evaluate(string upperName, ReadOnlySpan args) + { + string? format = null; + if (upperName == "STRFTIME") + { + if (args.Length < 2 || args[0] is null) + { + return null; + } + + format = SqlScalarExpression.ToText(args[0]); + args = args[1..]; + } + + if (!TryResolve(args, out var value)) + { + return null; + } + + return upperName switch + { + "DATE" => value.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture), + "TIME" => value.ToString("HH:mm:ss", CultureInfo.InvariantCulture), + "DATETIME" => value.ToString("yyyy-MM-dd HH:mm:ss", CultureInfo.InvariantCulture), + "JULIANDAY" => ToJulianDay(value), + "UNIXEPOCH" => (long)Math.Floor((value - UnixEpoch).TotalSeconds), + "STRFTIME" => Format(format!, value), + _ => null, + }; + } + + /// Reads the time value (with no argument: now) and applies the modifiers that follow it. + private static bool TryResolve(ReadOnlySpan args, out DateTime value) + { + value = default; + int firstModifier = 1; + if (args.Length == 0) + { + value = DateTime.UtcNow; + firstModifier = 0; + } + else if (args.Length > 1 && args[0] is not null && IsNumber(args[0]) && + SqlScalarExpression.ToText(args[1]).Trim().Equals("unixepoch", StringComparison.OrdinalIgnoreCase)) + { + // 'unixepoch' is only meaningful directly after a numeric time value. + if (!TryUnixSeconds(ToDouble(args[0]), out value)) + { + return false; + } + + firstModifier = 2; + } + else if (!TryParseTimeValue(args[0], out value)) + { + return false; + } + + for (int i = firstModifier; i < args.Length; i++) + { + if (args[i] is null || !TryApplyModifier(ref value, SqlScalarExpression.ToText(args[i]))) + { + return false; + } + } + + return true; + } + + /// Reads a time value: a time string, now, a number (Julian day), or a CLR date. + internal static bool TryParseTimeValue(object? raw, out DateTime value) + { + value = default; + switch (raw) + { + case null or DBNull: + return false; + case DateTime dt: + value = DateTime.SpecifyKind(dt.Kind == DateTimeKind.Local ? dt.ToUniversalTime() : dt, DateTimeKind.Utc); + return true; + case DateTimeOffset dto: + value = dto.UtcDateTime; + return true; + case DateOnly date: + value = date.ToDateTime(TimeOnly.MinValue, DateTimeKind.Utc); + return true; + case TimeOnly time: + value = new DateTime(2000, 1, 1, 0, 0, 0, DateTimeKind.Utc).Add(time.ToTimeSpan()); + return true; + } + + if (IsNumber(raw)) + { + return TryFromJulianDay(ToDouble(raw), out value); + } + + var text = SqlScalarExpression.ToText(raw).Trim(); + if (text.Equals("now", StringComparison.OrdinalIgnoreCase)) + { + value = DateTime.UtcNow; + return true; + } + + if (double.TryParse(text, NumberStyles.Float, CultureInfo.InvariantCulture, out var julian)) + { + return TryFromJulianDay(julian, out value); + } + + var m = TimeStringPattern.Match(text); + if (!m.Success || (!m.Groups["y"].Success && !m.Groups["h"].Success)) + { + return false; + } + + int year = m.Groups["y"].Success ? Int(m, "y") : 2000; + int month = m.Groups["y"].Success ? Int(m, "mo") : 1; + int day = m.Groups["y"].Success ? Int(m, "d") : 1; + int hour = m.Groups["h"].Success ? Int(m, "h") : 0; + int minute = m.Groups["h"].Success ? Int(m, "mi") : 0; + int second = m.Groups["s"].Success ? Int(m, "s") : 0; + + if (year < 1 || month is < 1 or > 12 || day is < 1 or > 31 || hour > 24 || minute > 59 || second > 59) + { + return false; + } + + // SQLite accepts day 31 in any month and normalises the overflow ('2026-02-31' is 2026-03-03). + value = new DateTime(year, month, 1, 0, 0, 0, DateTimeKind.Utc) + .AddDays(day - 1) + .AddHours(hour) + .AddMinutes(minute) + .AddSeconds(second); + + if (m.Groups["f"].Success) + { + var digits = m.Groups["f"].Value; + var fraction = double.Parse("0." + digits, CultureInfo.InvariantCulture); + value = value.AddTicks((long)Math.Round(fraction * TimeSpan.TicksPerSecond)); + } + + if (m.Groups["z"].Success && m.Groups["z"].Value is not ("Z" or "z")) + { + // A "+HH:MM" suffix names local time at that offset; SQLite converts it to UTC. + var zone = m.Groups["z"].Value; + var offset = new TimeSpan( + int.Parse(zone.AsSpan(1, 2), CultureInfo.InvariantCulture), + int.Parse(zone.AsSpan(4, 2), CultureInfo.InvariantCulture), + 0); + value = zone[0] == '+' ? value - offset : value + offset; + } + + return true; + } + + private static bool TryApplyModifier(ref DateTime value, string modifier) + { + var mod = modifier.Trim(); + var lower = mod.ToLowerInvariant(); + + switch (lower) + { + case "start of day": + value = value.Date; + return true; + case "start of month": + value = new DateTime(value.Year, value.Month, 1, 0, 0, 0, DateTimeKind.Utc); + return true; + case "start of year": + value = new DateTime(value.Year, 1, 1, 0, 0, 0, DateTimeKind.Utc); + return true; + case "utc": + return true; // values are already held as UTC + case "localtime": + value = DateTime.SpecifyKind(TimeZoneInfo.ConvertTimeFromUtc(value, TimeZoneInfo.Local), DateTimeKind.Utc); + return true; + } + + if (lower.StartsWith("weekday ", StringComparison.Ordinal) && + int.TryParse(lower.AsSpan("weekday ".Length), NumberStyles.Integer, CultureInfo.InvariantCulture, out var weekday) && + weekday is >= 0 and <= 6) + { + int ahead = (weekday - (int)value.DayOfWeek + 7) % 7; + value = value.AddDays(ahead); + return true; + } + + var m = ArithmeticModifierPattern.Match(mod); + if (!m.Success) + { + return false; + } + + var amount = double.Parse(m.Groups["n"].Value, CultureInfo.InvariantCulture); + try + { + value = m.Groups["unit"].Value.ToLowerInvariant() switch + { + "day" => value.AddDays(amount), + "hour" => value.AddHours(amount), + "minute" => value.AddMinutes(amount), + "second" => value.AddSeconds(amount), + "month" => AddMonthsKeepingDay(value, (int)Math.Truncate(amount)), + "year" => AddMonthsKeepingDay(value, (int)Math.Truncate(amount) * 12), + _ => value, + }; + } + catch (ArgumentOutOfRangeException) + { + return false; + } + + return true; + } + + /// + /// SQLite's month arithmetic: change the month, keep the day number, and let an overflowing day roll into the + /// next month — unlike , which clamps to the last day. + /// + private static DateTime AddMonthsKeepingDay(DateTime value, int months) + { + int totalMonths = (value.Year * 12) + (value.Month - 1) + months; + int year = totalMonths / 12; + int month = (totalMonths % 12) + 1; + return new DateTime(year, month, 1, 0, 0, 0, DateTimeKind.Utc) + .AddDays(value.Day - 1) + .Add(value.TimeOfDay); + } + + /// + /// strftime: the SQLite conversions, read left to right so %% is one percent sign. An unknown + /// conversion yields NULL, as in SQLite. + /// + private static string? Format(string format, DateTime value) + { + var sb = new StringBuilder(format.Length + 8); + var inv = CultureInfo.InvariantCulture; + for (int i = 0; i < format.Length; i++) + { + char c = format[i]; + if (c != '%') + { + sb.Append(c); + continue; + } + + if (++i >= format.Length) + { + return null; + } + + int hour12 = value.Hour % 12 == 0 ? 12 : value.Hour % 12; + switch (format[i]) + { + case 'd': sb.Append(value.Day.ToString("D2", inv)); break; + case 'e': sb.Append(value.Day.ToString(inv).PadLeft(2)); break; + case 'f': sb.Append(value.Second.ToString("D2", inv)).Append('.').Append(value.Millisecond.ToString("D3", inv)); break; + case 'F': sb.Append(value.ToString("yyyy-MM-dd", inv)); break; + case 'H': sb.Append(value.Hour.ToString("D2", inv)); break; + case 'k': sb.Append(value.Hour.ToString(inv).PadLeft(2)); break; + case 'I': sb.Append(hour12.ToString("D2", inv)); break; + case 'l': sb.Append(hour12.ToString(inv).PadLeft(2)); break; + case 'j': sb.Append(value.DayOfYear.ToString("D3", inv)); break; + case 'J': sb.Append(ToJulianDay(value).ToString("0.0########", inv)); break; + case 'm': sb.Append(value.Month.ToString("D2", inv)); break; + case 'M': sb.Append(value.Minute.ToString("D2", inv)); break; + case 'p': sb.Append(value.Hour < 12 ? "AM" : "PM"); break; + case 'P': sb.Append(value.Hour < 12 ? "am" : "pm"); break; + case 'R': sb.Append(value.ToString("HH:mm", inv)); break; + case 's': sb.Append(((long)Math.Floor((value - UnixEpoch).TotalSeconds)).ToString(inv)); break; + case 'S': sb.Append(value.Second.ToString("D2", inv)); break; + case 'T': sb.Append(value.ToString("HH:mm:ss", inv)); break; + case 'u': sb.Append(value.DayOfWeek == DayOfWeek.Sunday ? '7' : (char)('0' + (int)value.DayOfWeek)); break; + case 'w': sb.Append((char)('0' + (int)value.DayOfWeek)); break; + case 'Y': sb.Append(value.Year.ToString("D4", inv)); break; + case '%': sb.Append('%'); break; + default: return null; + } + } + + return sb.ToString(); + } + + private static double ToJulianDay(DateTime value) => + UnixEpochJulianDay + ((value - UnixEpoch).TotalSeconds / 86400.0); + + private static bool TryFromJulianDay(double julian, out DateTime value) + { + value = default; + if (double.IsNaN(julian) || julian < 1721425.5 || julian > 5373484.5) // 0001-01-01 .. 9999-12-31 + { + return false; + } + + value = UnixEpoch.AddSeconds((julian - UnixEpochJulianDay) * 86400.0); + return true; + } + + private static bool TryUnixSeconds(double seconds, out DateTime value) + { + value = default; + try + { + value = UnixEpoch.AddSeconds(seconds); + return true; + } + catch (ArgumentOutOfRangeException) + { + return false; + } + } + + private static bool IsNumber(object? value) => + value is int or long or short or byte or sbyte or uint or ulong or ushort or double or float or decimal; + + private static double ToDouble(object? value) => Convert.ToDouble(value, CultureInfo.InvariantCulture); + + private static int Int(Match m, string group) => int.Parse(m.Groups[group].Value, CultureInfo.InvariantCulture); +} diff --git a/src/SharpCoreDB/Services/SqlFunctions.cs b/src/SharpCoreDB/Services/SqlFunctions.cs index da594c11..eaf29247 100644 --- a/src/SharpCoreDB/Services/SqlFunctions.cs +++ b/src/SharpCoreDB/Services/SqlFunctions.cs @@ -30,7 +30,8 @@ public static class SqlFunctions /// Formatted date string. public static string StrFTime(DateTime dateTime, string format) { - return dateTime.ToString(format); + // Invariant culture: with the current one, th-TH printed the year 2026 as 2569 (its Buddhist calendar). + return dateTime.ToString(format, System.Globalization.CultureInfo.InvariantCulture); } /// @@ -210,8 +211,10 @@ public static string Quote(object? value) string s => $"'{s.Replace("'", "''", StringComparison.Ordinal)}'", bool b => b ? "1" : "0", byte[] bytes => $"X'{Convert.ToHexString(bytes)}'", - DateTime dt => $"'{dt:yyyy-MM-dd HH:mm:ss}'", - DateTimeOffset dto => $"'{dto:yyyy-MM-dd HH:mm:ss zzz}'", + DateTime dt => SqlLiteralFormatter.Literal(dt), + DateTimeOffset dto => SqlLiteralFormatter.Literal(dto), + DateOnly date => SqlLiteralFormatter.Literal(date), + TimeOnly time => SqlLiteralFormatter.Literal(time), _ => Convert.ToString(value, System.Globalization.CultureInfo.InvariantCulture) ?? "NULL", }; @@ -281,9 +284,15 @@ public static string Char(params int[] codePoints) "CURRENT_DATE" => CurrentDate(), "CURRENT_TIME" => CurrentTime(), "SYNC_TIMESTAMP" => SyncTimestamp(), - "DATE" => arguments.Count > 0 && arguments[0] is DateTime dt ? Date(dt) : null, + // SQLite's date family through the engine's one implementation (SqlDateTimeFunctions). The legacy + // (DateTime[, .NET format]) shapes this method answered before keep their behaviour. + "DATE" => arguments.Count > 0 && arguments[0] is DateTime dt + ? Date(dt) + : SqlDateTimeFunctions.Evaluate("DATE", [.. arguments]), "STRFTIME" => arguments.Count >= 2 && arguments[0] is DateTime dt2 && arguments[1] is string fmt - ? StrFTime(dt2, fmt) : null, + ? StrFTime(dt2, fmt) + : SqlDateTimeFunctions.Evaluate("STRFTIME", [.. arguments]), + "TIME" or "DATETIME" or "JULIANDAY" or "UNIXEPOCH" => SqlDateTimeFunctions.Evaluate(upperName, [.. arguments]), "DATEADD" => arguments.Count >= 3 && arguments[0] is DateTime dt3 && arguments[1] is int val && arguments[2] is string unit ? DateAdd(dt3, val, unit) : null, "UNHEX" => arguments.Count >= 1 && arguments[0] is not null ? Unhex(arguments[0].ToString() ?? string.Empty) : null, diff --git a/src/SharpCoreDB/Services/SqlLiteralFormatter.cs b/src/SharpCoreDB/Services/SqlLiteralFormatter.cs new file mode 100644 index 00000000..2ad41196 --- /dev/null +++ b/src/SharpCoreDB/Services/SqlLiteralFormatter.cs @@ -0,0 +1,49 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Services; + +using System; +using System.Globalization; + +/// +/// Culture-invariant SQL text for values that are rendered into a statement: the one definition shared by +/// ParameterBinder.FormatParameter, CompiledQueryExecutor.FormatParameterValue, +/// SqlFunctions.Quote and Table.FormatValue. +/// +/// Up to 2.1.4 each copy formatted on its own with the current culture: th-TH wrote the year 2026 as 2569 (its +/// Buddhist calendar), a DateTimeOffset became '1/1/2026 12:30:15 PM +10:00', three copies dropped a +/// DateTime's milliseconds, sv-SE wrote a negative number with U+2212 instead of '-', and a comma-decimal +/// culture wrote 1,5. Dates use SQLite's canonical time-string form, which its date functions read. +/// +/// +internal static class SqlLiteralFormatter +{ + /// SQLite's canonical date-time text, with milliseconds. + internal const string DateTimeFormat = "yyyy-MM-dd HH:mm:ss.fff"; + + /// The canonical date-time text followed by the offset, which SQLite time strings accept as [+-]HH:MM. + internal const string DateTimeOffsetFormat = "yyyy-MM-dd HH:mm:ss.fffzzz"; + + /// Quoted literal of a . + internal static string Literal(DateTime value) => + $"'{value.ToString(DateTimeFormat, CultureInfo.InvariantCulture)}'"; + + /// Quoted literal of a . + internal static string Literal(DateTimeOffset value) => + $"'{value.ToString(DateTimeOffsetFormat, CultureInfo.InvariantCulture)}'"; + + /// Quoted literal of a . + internal static string Literal(DateOnly value) => + $"'{value.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)}'"; + + /// Quoted literal of a . + internal static string Literal(TimeOnly value) => + $"'{value.ToString("HH:mm:ss.fff", CultureInfo.InvariantCulture)}'"; + + /// Unquoted, culture-invariant text of any other value (numbers, Guids, enums, …). + internal static string Invariant(object value) => + Convert.ToString(value, CultureInfo.InvariantCulture) ?? string.Empty; +} diff --git a/src/SharpCoreDB/Services/SqlOrderBy.cs b/src/SharpCoreDB/Services/SqlOrderBy.cs new file mode 100644 index 00000000..b2473b05 --- /dev/null +++ b/src/SharpCoreDB/Services/SqlOrderBy.cs @@ -0,0 +1,259 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Services; + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using SharpCoreDB.Interfaces; + +/// +/// The ORDER BY list of the single-table SELECT path, read and applied as SQLite does: every key with its own +/// direction, a key may be any scalar expression, a result-column alias or a 1-based result-column ordinal, NULLs sort +/// first when ascending (last when descending) unless NULLS FIRST|LAST says otherwise, and text compares with +/// the key's COLLATE, else the column's declared collation, else BINARY. The sort is stable. +/// +/// Up to 2.1.4 the path read only the token after ORDER BY: a second key was ignored (ORDER BY grp, name came +/// back unsorted — linq2db's shape for OrderBy(…).ThenBy(…)), an expression, alias or qualified key +/// ([x].[name]) matched no row value, and text was compared with the machine's culture. +/// +/// +internal static class SqlOrderBy +{ + /// One ORDER BY key. + internal sealed class Term(SqlScalarExpression.Node key, bool descending, bool nullsFirst, string? collation, string? columnName) + { + public SqlScalarExpression.Node Key { get; } = key; + public bool Descending { get; } = descending; + public bool NullsFirst { get; } = nullsFirst; + public string? Collation { get; } = collation; + public string? ColumnName { get; } = columnName; + } + + /// + /// Reads an ORDER BY list. Returns null when a key is not something this evaluator can compute — for + /// example vec_distance_cosine(…), which keeps its vector-index route — so the caller keeps its own handling. + /// + /// The text after ORDER BY, up to LIMIT. + /// The SELECT list, for aliases and ordinals. + internal static IReadOnlyList? TryParse(string orderClause, string selectClause) + { + if (string.IsNullOrWhiteSpace(orderClause)) + { + return null; + } + + var selectItems = ReadSelectItems(selectClause); + var terms = new List(); + foreach (var item in SqlInPredicate.SplitTopLevelComma(orderClause)) + { + var tokens = SqlLiteralScanner.SplitTopLevelWhitespace(item.Trim()); + int end = tokens.Count; + bool? nullsFirst = null; + bool descending = false; + string? collation = null; + + if (end >= 2 && Is(tokens[end - 2], "NULLS") && (Is(tokens[end - 1], "FIRST") || Is(tokens[end - 1], "LAST"))) + { + nullsFirst = Is(tokens[end - 1], "FIRST"); + end -= 2; + } + + if (end >= 1 && (Is(tokens[end - 1], "ASC") || Is(tokens[end - 1], "DESC"))) + { + descending = Is(tokens[end - 1], "DESC"); + end--; + } + + if (end >= 2 && Is(tokens[end - 2], "COLLATE")) + { + collation = SqlIdentifierScanner.IdentifierText(tokens[end - 1]).ToUpperInvariant(); + end -= 2; + } + + if (end == 0) + { + return null; + } + + var key = ResolveKey(string.Join(" ", tokens.Take(end)), selectItems); + if (key is null) + { + return null; + } + + // SQLite: NULL is the smallest value, so it leads an ascending sort and trails a descending one. + terms.Add(new Term(key.Value.Node, descending, nullsFirst ?? !descending, collation, key.Value.ColumnName)); + } + + return terms; + } + + /// Sorts rows by the terms (stable), before any OFFSET/LIMIT is applied. + /// The rows to sort. + /// The ORDER BY terms. + /// The table, for declared column collations. + internal static List> Sort( + List> rows, + IReadOnlyList terms, + ITable table) + { + if (rows.Count < 2) + { + return rows; + } + + var textComparers = terms.Select(term => TextComparerFor(term, table)).ToArray(); + var keyed = new (object?[] Keys, Dictionary Row)[rows.Count]; + for (int r = 0; r < rows.Count; r++) + { + var keys = new object?[terms.Count]; + for (int t = 0; t < terms.Count; t++) + { + keys[t] = SqlScalarExpression.Evaluate(terms[t].Key, rows[r]); + } + + keyed[r] = (keys, rows[r]); + } + + var comparer = Comparer.Create((a, b) => + { + for (int t = 0; t < terms.Count; t++) + { + int c = CompareKey(a[t], b[t], terms[t], textComparers[t]); + if (c != 0) + { + return c; + } + } + + return 0; + }); + + // Enumerable.OrderBy is a stable sort: rows with equal keys keep their scan order. + return [.. keyed.OrderBy(k => k.Keys, comparer).Select(k => k.Row)]; + } + + private static int CompareKey(object? x, object? y, Term term, IComparer? textComparer) + { + bool xNull = x is null or DBNull; + bool yNull = y is null or DBNull; + if (xNull || yNull) + { + if (xNull && yNull) + { + return 0; + } + + // NULL placement is fixed by NULLS FIRST/LAST, not flipped by DESC. + return xNull == term.NullsFirst ? -1 : 1; + } + + int c; + if (x is string xs && y is string ys) + { + c = textComparer?.Compare(xs, ys) ?? string.CompareOrdinal(xs, ys); + } + else if (!SqlScalarExpression.TryCompare(x, y, out c)) + { + c = 0; + } + + c = Math.Sign(c); + return term.Descending ? -c : c; + } + + /// The text comparer for a term: its COLLATE, else the column's declared collation, else BINARY (null). + private static IComparer? TextComparerFor(Term term, ITable table) + { + switch (term.Collation) + { + case "NOCASE": + return StringComparer.OrdinalIgnoreCase; + case "RTRIM": + return RTrimComparer; + case "BINARY": + return null; + } + + if (term.ColumnName is null) + { + return null; + } + + int index = table.Columns.FindIndex(c => c.Equals(term.ColumnName, StringComparison.OrdinalIgnoreCase)); + if (index < 0 || index >= table.ColumnCollations.Count) + { + return null; + } + + return table.ColumnCollations[index] switch + { + CollationType.NoCase => StringComparer.OrdinalIgnoreCase, + CollationType.RTrim => RTrimComparer, + CollationType.UnicodeCaseInsensitive => StringComparer.InvariantCultureIgnoreCase, + CollationType.Locale when index < table.ColumnLocaleNames.Count && + !string.IsNullOrWhiteSpace(table.ColumnLocaleNames[index]) => + StringComparer.Create(CultureInfo.GetCultureInfo(table.ColumnLocaleNames[index]!), ignoreCase: false), + _ => null, + }; + } + + private static readonly IComparer RTrimComparer = + Comparer.Create(static (a, b) => string.CompareOrdinal(a.TrimEnd(' '), b.TrimEnd(' '))); + + /// + /// Resolves an ORDER BY key: a 1-based ordinal names a result column; an unqualified name that is a result-column + /// alias names that column's expression (SQLite resolves aliases first); anything else is an expression. + /// + private static (SqlScalarExpression.Node Node, string? ColumnName)? ResolveKey( + string text, + List<(string Expression, string? Alias)> selectItems) + { + if (long.TryParse(text, NumberStyles.None, CultureInfo.InvariantCulture, out var ordinal)) + { + if (ordinal < 1 || ordinal > selectItems.Count) + { + return null; + } + + var expression = selectItems[(int)ordinal - 1].Expression; + return expression == "*" || expression.EndsWith(".*", StringComparison.Ordinal) ? null : ParseKey(expression); + } + + if (SqlScalarExpression.TryParse(text, out var node) && + SqlScalarExpression.ColumnName(node) is { } name && + !SqlScalarExpression.IsQualified(node)) + { + foreach (var (expression, alias) in selectItems) + { + if (alias is not null && alias.Equals(name, StringComparison.OrdinalIgnoreCase)) + { + return ParseKey(expression); + } + } + } + + return ParseKey(text); + } + + private static (SqlScalarExpression.Node Node, string? ColumnName)? ParseKey(string text) => + SqlScalarExpression.TryParse(text, out var node) ? (node, SqlScalarExpression.ColumnName(node)) : null; + + private static List<(string Expression, string? Alias)> ReadSelectItems(string selectClause) + { + var clause = selectClause.Trim(); + if (clause.StartsWith("DISTINCT", StringComparison.OrdinalIgnoreCase)) + { + clause = clause["DISTINCT".Length..].Trim(); + } + + return [.. SqlInPredicate.SplitTopLevelComma(clause).Select(item => SqlParser.SplitSelectItemAlias(item.Trim()))]; + } + + private static bool Is(string token, string keyword) => token.Equals(keyword, StringComparison.OrdinalIgnoreCase); +} diff --git a/src/SharpCoreDB/Services/SqlParser.DDL.cs b/src/SharpCoreDB/Services/SqlParser.DDL.cs index cf5c0ca4..65572531 100644 --- a/src/SharpCoreDB/Services/SqlParser.DDL.cs +++ b/src/SharpCoreDB/Services/SqlParser.DDL.cs @@ -374,6 +374,13 @@ private void ExecuteCreateTable(string sql, string[] parts, IWAL? wal) ColumnLocaleNames = columnLocaleNames, }); + // A UNIQUE column gets an automatic unique index, as SQLite's sqlite_autoindex_…: it is what enforces the + // constraint. Up to 2.1.4 the constraint was stored and never checked. + if (table is SharpCoreDB.DataStructures.Table createdTable) + { + createdTable.EnsureUniqueConstraintIndexes(); + } + table.Name = tableName; // B7+: new columnar tables with an explicitly declared PRIMARY KEY default to the fixed-width diff --git a/src/SharpCoreDB/Services/SqlParser.DML.ScalarFunctions.cs b/src/SharpCoreDB/Services/SqlParser.DML.ScalarFunctions.cs index 657a1a38..6706094b 100644 --- a/src/SharpCoreDB/Services/SqlParser.DML.ScalarFunctions.cs +++ b/src/SharpCoreDB/Services/SqlParser.DML.ScalarFunctions.cs @@ -11,7 +11,8 @@ namespace SharpCoreDB.Services; /// /// SqlParser partial — scalar SELECT support: literal expressions, aliases, -/// and change-counter functions (CHANGES, TOTAL_CHANGES, LAST_INSERT_ROWID). +/// and change-counter functions (CHANGES, TOTAL_CHANGES, LAST_INSERT_ROWID). Scalar and date/time functions are +/// evaluated by , the evaluator rows use too. /// public partial class SqlParser { @@ -44,6 +45,14 @@ private List> ExecuteSelectLiteralQuery(string select /// private object? EvaluateLiteralSelectExpression(string expression) { + // Scalar functions, CAST and operators go through the shared evaluator, the same one a row uses, so + // date('…'), time('…'), datetime('…') and upper('abc') answer as SQLite does. Plain literals keep the typed + // parse below. + if (SqlScalarExpression.TryParse(expression, out var computed) && SqlScalarExpression.IsComputed(computed)) + { + return SqlScalarExpression.Evaluate(computed, row: null); + } + var functionMatch = Regex.Match( expression, @"^(?[A-Za-z_][A-Za-z0-9_]*)\((?.*)\)$", @@ -59,9 +68,6 @@ private List> ExecuteSelectLiteralQuery(string select "CHANGES" => _lastChanges, "TOTAL_CHANGES" => _totalChanges, "LAST_INSERT_ROWID" => _lastInsertRowId, - "JULIANDAY" => EvaluateJulianDay(functionMatch.Groups["args"].Value), - "UNIXEPOCH" => EvaluateUnixEpoch(functionMatch.Groups["args"].Value), - "STRFTIME" => EvaluateStrftime(functionMatch.Groups["args"].Value), _ => ParseLiteralScalarValue(expression), }; } @@ -192,99 +198,4 @@ private static List SplitTopLevelComma(string text) return literal; } - - // ─── Date / time scalar functions ──────────────────────────────────────── - - /// Parses the first quoted date argument from a function arg list. - private static DateTime ParseDateArg(string args) - { - var parts = SplitTopLevelComma(args); - if (parts.Count == 0) return DateTime.UtcNow; - var raw = parts[0].Trim().Trim('\'', '"'); - if (raw.Equals("now", StringComparison.OrdinalIgnoreCase)) return DateTime.UtcNow; - var dt = DateTime.Parse(raw, CultureInfo.InvariantCulture, System.Globalization.DateTimeStyles.RoundtripKind); - // SQLite treats dates without timezone as UTC - if (dt.Kind == DateTimeKind.Unspecified) - dt = DateTime.SpecifyKind(dt, DateTimeKind.Utc); - return dt; - } - - /// Applies SQLite date modifiers ('start of month', '+N days', etc.) to a DateTime. - private static DateTime ApplyDateModifiers(DateTime dt, IReadOnlyList modifiers) - { - foreach (var rawMod in modifiers) - { - var mod = rawMod.Trim().Trim('\'', '"').Trim(); - if (mod.Equals("start of month", StringComparison.OrdinalIgnoreCase)) - dt = new DateTime(dt.Year, dt.Month, 1, dt.Hour, dt.Minute, dt.Second, dt.Kind); - else if (mod.Equals("start of year", StringComparison.OrdinalIgnoreCase)) - dt = new DateTime(dt.Year, 1, 1, dt.Hour, dt.Minute, dt.Second, dt.Kind); - else if (mod.Equals("start of day", StringComparison.OrdinalIgnoreCase)) - dt = new DateTime(dt.Year, dt.Month, dt.Day, 0, 0, 0, dt.Kind); - else - { - var m = Regex.Match(mod, @"^([+-]?\d+)\s+(day|month|year|hour|minute|second)s?$", RegexOptions.IgnoreCase, TimeSpan.FromSeconds(1)); - if (m.Success) - { - int n = int.Parse(m.Groups[1].Value, CultureInfo.InvariantCulture); - dt = m.Groups[2].Value.ToUpperInvariant() switch - { - "DAY" => dt.AddDays(n), - "MONTH" => dt.AddMonths(n), - "YEAR" => dt.AddYears(n), - "HOUR" => dt.AddHours(n), - "MINUTE" => dt.AddMinutes(n), - "SECOND" => dt.AddSeconds(n), - _ => dt - }; - } - } - } - return dt; - } - - /// Evaluates JULIANDAY(date [, modifier ...]) → double. - private static object EvaluateJulianDay(string args) - { - var parts = SplitTopLevelComma(args); - var dt = ParseDateArg(args); - if (parts.Count > 1) dt = ApplyDateModifiers(dt, parts.Skip(1).ToList()); - // Julian Day Number: days since noon January 1, 4713 BC - var epoch = new DateTime(2000, 1, 1, 12, 0, 0, DateTimeKind.Utc); // J2000.0 = 2451545.0 - const double j2000 = 2451545.0; - return j2000 + (dt.ToUniversalTime() - epoch).TotalDays; - } - - /// Evaluates UNIXEPOCH(date [, modifier ...]) → long (seconds since 1970-01-01). - private static object EvaluateUnixEpoch(string args) - { - var parts = SplitTopLevelComma(args); - var dt = ParseDateArg(args); - if (parts.Count > 1) dt = ApplyDateModifiers(dt, parts.Skip(1).ToList()); - var unixEpoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc); - return (long)(dt.ToUniversalTime() - unixEpoch).TotalSeconds; - } - - /// Evaluates strftime(format, date [, modifier ...]) → string. - private static object? EvaluateStrftime(string args) - { - var parts = SplitTopLevelComma(args); - if (parts.Count < 2) return null; - var fmt = parts[0].Trim().Trim('\'', '"'); - var dt = ParseDateArg(parts[1]); - if (parts.Count > 2) dt = ApplyDateModifiers(dt, parts.Skip(2).ToList()); - // Map SQLite strftime format specifiers to .NET format - var result = fmt - .Replace("%Y", dt.Year.ToString("D4", CultureInfo.InvariantCulture)) - .Replace("%m", dt.Month.ToString("D2", CultureInfo.InvariantCulture)) - .Replace("%d", dt.Day.ToString("D2", CultureInfo.InvariantCulture)) - .Replace("%H", dt.Hour.ToString("D2", CultureInfo.InvariantCulture)) - .Replace("%M", dt.Minute.ToString("D2", CultureInfo.InvariantCulture)) - .Replace("%S", dt.Second.ToString("D2", CultureInfo.InvariantCulture)) - .Replace("%j", dt.DayOfYear.ToString("D3", CultureInfo.InvariantCulture)) - .Replace("%w", ((int)dt.DayOfWeek).ToString(CultureInfo.InvariantCulture)) - .Replace("%f", $"{dt.Second:D2}.{dt.Millisecond:D3}") - .Replace("%%", "%"); - return result; - } } diff --git a/src/SharpCoreDB/Services/SqlParser.DML.cs b/src/SharpCoreDB/Services/SqlParser.DML.cs index f6bb66ab..868dd7ee 100644 --- a/src/SharpCoreDB/Services/SqlParser.DML.cs +++ b/src/SharpCoreDB/Services/SqlParser.DML.cs @@ -488,8 +488,11 @@ Dictionary BuildRowFromValues(List rowValues) // because the batch core validates every row before writing any. The per-row loop leaves the rows it // had already written in place. That matches SQLite, where one multi-row INSERT is one atomic unit. const int MinRowsForBatchedInsert = 2; + // A single-file table takes the batch path too: its InsertBatch validates every row (NOT NULL) before caching + // any, which makes the statement atomic. Up to 2.1.4 it took the per-row loop, so a failing row + // left the rows before it in place. bool useBatchedInsert = allRowValues.Count >= MinRowsForBatchedInsert - && tableAsTable is { CanUseBatchedInsert: true } + && (tableAsTable is { CanUseBatchedInsert: true } || table is SingleFileTable) && !HasTriggersFor(tableName, TriggerEvent.Insert); List>? batchedRows = useBatchedInsert ? new List>(allRowValues.Count) @@ -527,7 +530,14 @@ Dictionary BuildRowFromValues(List rowValues) // residual (dispatch minus its children) are the last unattributed third of a multi-row pass // (plan §9 priority 2), and they need different fixes, so they are separated here rather than argued. long tableBatchStart = Diagnostics.WritePathProfiler.Stamp(); - tableAsTable!.InsertBatch(batchedRows); + if (tableAsTable is not null) + { + tableAsTable.InsertBatch(batchedRows); + } + else + { + table.InsertBatch(batchedRows); + } Diagnostics.WritePathProfiler.Add(Diagnostics.WritePathProfiler.Stage.TableBatch, tableBatchStart); insertedCount = batchedRows.Count; lastInsertedRow = batchedRows[^1]; @@ -537,7 +547,7 @@ Dictionary BuildRowFromValues(List rowValues) // the same value the per-row loop would have left, so IDatabase.GetLastInsertRowId() cannot // quietly change meaning for this one statement shape. The no-PK case needs nothing — the // position is exactly what Insert records there too. - int pkColIdx = tableAsTable.PrimaryKeyIndex; + int pkColIdx = table.PrimaryKeyIndex; if (pkColIdx >= 0 && lastInsertedRow.TryGetValue(table.Columns[pkColIdx], out var pkRowIdVal) && pkRowIdVal is not null and not DBNull) @@ -1055,6 +1065,17 @@ private List> ExecuteSelectQuery(string sql, string[] (int? limit, int? offset) = ParseLimitClause(parts, limitIdx); + // The whole ORDER BY list as SQLite reads it — every key with its direction, expressions, aliases, ordinals, + // COLLATE and NULLS FIRST/LAST. The token-based `orderBy` above sees only the first key, so `ORDER BY grp, name` + // came back unsorted. A list SqlOrderBy cannot read (vec_distance_* keeps its vector-index route below) leaves + // the previous handling in place. + var orderTerms = orderIdx > 0 && parts.Length > orderIdx + 2 && + parts[orderIdx + 1].Equals(SqlConstants.BY, StringComparison.OrdinalIgnoreCase) + ? SqlOrderBy.TryParse( + string.Join(" ", parts.Skip(orderIdx + 2).Take((limitIdx > orderIdx ? limitIdx : parts.Length) - orderIdx - 2)), + selectClause) + : null; + if (!this.tables.ContainsKey(tableName)) throw new InvalidOperationException($"Table {tableName} does not exist"); @@ -1080,15 +1101,25 @@ private List> ExecuteSelectQuery(string sql, string[] Constants.PersistenceConstants.InternalRowIdColumnName, StringComparison.OrdinalIgnoreCase); + var scanOrderBy = orderTerms is null ? orderBy : null; var results = requestsRowId && tableObj is Table tableWithRowId && tableWithRowId.HasInternalRowId - ? tableWithRowId.SelectIncludingRowId(whereStr, orderBy, asc, noEncrypt) - : tableObj.Select(whereStr, orderBy, asc, noEncrypt); + ? tableWithRowId.SelectIncludingRowId(whereStr, scanOrderBy, asc, noEncrypt) + : tableObj.Select(whereStr, scanOrderBy, asc, noEncrypt); + + // Sorted here, before OFFSET/LIMIT, with SQLite's comparison: NULLs first ascending, BINARY text unless a + // COLLATE or the column's declared collation says otherwise. The scan's own sort compared text with the + // machine's culture. + if (orderTerms is not null) + { + results = SqlOrderBy.Sort(results, orderTerms, tableObj); + } // Apply limit and offset if (offset.HasValue && offset.Value > 0) results = [.. results.Skip(offset.Value)]; // ✅ C# 15: Collection expression with spread - if (limit.HasValue && limit.Value > 0) + // LIMIT 0 returns no rows, as in SQLite; up to 2.1.4 it returned every row. + if (limit.HasValue && limit.Value >= 0) results = [.. results.Take(limit.Value)]; // Deduplicate by primary key @@ -1226,7 +1257,9 @@ private static List> ApplySelectListToRows( clause = clause["DISTINCT".Length..].Trim(); } - foreach (var item in SqlInPredicate.SplitTopLevelComma(clause)) + var items = SqlInPredicate.SplitTopLevelComma(clause); + bool hasExpression = false; + foreach (var item in items) { var text = item.Trim(); if (text.Length == 0 || text == "*" || text.EndsWith(".*", StringComparison.Ordinal)) @@ -1238,13 +1271,82 @@ private static List> ApplySelectListToRows( { if (c is '(' or ')' or '\'' or '+' or '-' or '/' or '%' or '|' or '=' or '<' or '>' or '~' or '*') { - // An expression, not a column reference: leave the row as the scan produced it. - return rows; + hasExpression = true; + break; } } } - return ProjectColumns(rows, clause); + return hasExpression ? ProjectSelectListWithExpressions(rows, items) : ProjectColumns(rows, clause); + } + + /// + /// Projects a SELECT list that contains computed items — upper(name), strftime('%Y', at) AS y, + /// CAST(… AS INTEGER) — evaluating each one per row with . Plain column + /// items are resolved exactly as resolves them. A computed item is named by its alias, + /// or by its text as written, as in SQLite. + /// + /// Up to 2.1.4 a list with any expression in it returned the scanned row unchanged, so SELECT upper(name) + /// FROM t handed an ordinal-based reader the row's first column (its id) instead of the function value. + /// An item this evaluator cannot parse still leaves the rows as scanned. + /// + /// + /// The scanned rows. + /// The SELECT-list items, split at top-level commas. + /// The projected rows. + private static List> ProjectSelectListWithExpressions( + List> rows, + List items) + { + var projections = new List<(string OutputName, string SourceText, SqlScalarExpression.Node? Computed)>(items.Count); + foreach (var item in items) + { + var text = item.Trim(); + var (expressionText, alias) = SplitSelectItemAlias(text); + if (!SqlScalarExpression.TryParse(expressionText, out var node)) + { + return rows; + } + + projections.Add(SqlScalarExpression.IsColumn(node) + ? (alias ?? ExtractBareColumn(expressionText), expressionText, null) + : (alias ?? expressionText, expressionText, node)); + } + + var result = new List>(rows.Count); + foreach (var row in rows) + { + var projected = new Dictionary(projections.Count, StringComparer.OrdinalIgnoreCase); + foreach (var (outputName, sourceText, computed) in projections) + { + var value = computed is null ? FindValue(row, sourceText) : SqlScalarExpression.Evaluate(computed, row); + projected[outputName] = value ?? DBNull.Value; + } + + result.Add(projected); + } + + return result; + } + + /// + /// Splits expr AS alias / expr alias into the expression and the unquoted alias. The alias is only + /// taken when what precedes it parses as an expression, so a || b is not read as a || aliased b. + /// + internal static (string Expression, string? Alias) SplitSelectItemAlias(string item) + { + var match = System.Text.RegularExpressions.Regex.Match( + item, + @"^(?.+?)\s+(?:AS\s+)?(?[A-Za-z_]\w*|\[[^\]]+\]|""[^""]+""|`[^`]+`)$", + System.Text.RegularExpressions.RegexOptions.IgnoreCase | System.Text.RegularExpressions.RegexOptions.Singleline, + TimeSpan.FromSeconds(1)); + + if (match.Success && SqlScalarExpression.TryParse(match.Groups["expr"].Value.Trim(), out _)) + { + return (match.Groups["expr"].Value.Trim(), SqlIdentifierScanner.IdentifierText(match.Groups["alias"].Value)); + } + + return (item, null); } @@ -1667,6 +1769,28 @@ private static (int? limit, int? offset) ParseLimitClause(string[] parts, int li if (limitIdx <= 0) return (null, null); + // SQLite's forms: LIMIT n, LIMIT n OFFSET m and LIMIT m, n; a negative count means no limit (linq2db writes + // LIMIT -1 OFFSET m for Skip without Take). Up to 2.1.4 the two OFFSET-bearing forms threw "Invalid LIMIT + // clause" — so every linq2db Skip()/Take() failed on this path. + var limitText = string.Join(" ", parts.Skip(limitIdx + 1)).Trim().TrimEnd(';').Trim(); + var limitMatch = System.Text.RegularExpressions.Regex.Match( + limitText, + @"^(?:(?[+-]?\d+)(?:\s+OFFSET\s+(?[+-]?\d+))?|(?\d+)\s*,\s*(?[+-]?\d+))$", + System.Text.RegularExpressions.RegexOptions.IgnoreCase, + TimeSpan.FromSeconds(1)); + if (limitMatch.Success && + long.TryParse(limitMatch.Groups["count"].Value, System.Globalization.NumberStyles.AllowLeadingSign, System.Globalization.CultureInfo.InvariantCulture, out var count)) + { + int? offsetValue = null; + if (limitMatch.Groups["offset"].Success && + long.TryParse(limitMatch.Groups["offset"].Value, System.Globalization.NumberStyles.AllowLeadingSign, System.Globalization.CultureInfo.InvariantCulture, out var offsetNumber)) + { + offsetValue = (int)Math.Clamp(offsetNumber, 0, int.MaxValue); + } + + return (count < 0 ? null : (int)Math.Min(count, int.MaxValue), offsetValue); + } + var limitParts = parts.Skip(limitIdx + 1).ToArray(); if (limitParts.Length == 1) { @@ -1693,114 +1817,6 @@ private static (int? limit, int? offset) ParseLimitClause(string[] parts, int li throw new ArgumentException("Invalid LIMIT clause"); } - /// - /// Evaluates a scalar function call with the given arguments. - /// Includes COALESCE, IFNULL, NULLIF, IIF, TYPEOF, and more. - /// - private static object? EvaluateScalarFunction(FunctionCallNode functionCall, object?[] args) - { - // Normalize DBNull to null for consistency - for (int i = 0; i < args.Length; i++) - { - if (args[i] is DBNull) - args[i] = null; - } - - // Phase 5.2: TIMESTAMPDIFF function with modern pattern matching - if (functionCall.FunctionName.Equals("TIMESTAMPDIFF", StringComparison.OrdinalIgnoreCase) && args.Length == 3) - { - return (args[0] is DateTime start && args[1] is DateTime end && args[2] is string unit) - ? unit.ToUpperInvariant() switch - { - "MICROSECOND" => (end - start).TotalMilliseconds * 1000, - "MILLISECOND" => (end - start).TotalMilliseconds, - "SECOND" => (end - start).TotalSeconds, - "MINUTE" => (end - start).TotalMinutes, - "HOUR" => (end - start).TotalHours, - "DAY" => (end - start).TotalDays, - _ => null - } - : null; - } - - // General scalar function evaluation - return functionCall.FunctionName.ToUpperInvariant() switch - { - "COALESCE" => args.FirstOrDefault(arg => arg is not null), - "IFNULL" => args.Length == 2 ? (args[0] ?? args[1]) : throw new ArgumentException("IFNULL requires 2 arguments"), - "NULLIF" => args.Length == 2 ? (SqlParser.AreValuesEqual(args[0], args[1]) ? null : args[0]) : throw new ArgumentException("NULLIF requires 2 arguments"), - "IIF" => args.Length == 3 ? (args[0] is bool condition ? (condition ? args[1] : args[2]) : null) : throw new ArgumentException("IIF requires 3 arguments"), - "TYPEOF" => args.Length == 1 ? (object?)(args[0] switch - { - null => "null", - int or long => "integer", - double or float => "real", - string => "text", - bool => "integer", - _ => "text" - }) : throw new ArgumentException("TYPEOF requires 1 argument"), - "ABS" => args.Length == 1 ? (object?)(args[0] switch - { - double d => Math.Abs(d), - int i => Math.Abs(i), - long l => Math.Abs(l), - _ => null - }) : throw new ArgumentException("ABS requires 1 argument"), - "ROUND" => args.Length >= 1 && args.Length <= 2 && args[0] is double rd ? (object?)Math.Round(rd, args.Length == 2 && args[1] is int ri ? ri : 0) : null, - "CEIL" or "CEILING" => args.Length == 1 && args[0] is double cd ? (object?)Math.Ceiling(cd) : null, - "FLOOR" => args.Length == 1 && args[0] is double fd ? (object?)Math.Floor(fd) : null, - "UPPER" => args.Length == 1 ? (object?)args[0]?.ToString()?.ToUpperInvariant() : throw new ArgumentException("UPPER requires 1 argument"), - "LOWER" => args.Length == 1 ? (object?)args[0]?.ToString()?.ToLowerInvariant() : throw new ArgumentException("LOWER requires 1 argument"), - "LENGTH" => args.Length == 1 ? (object?)(args[0]?.ToString()?.Length ?? 0) : throw new ArgumentException("LENGTH requires 1 argument"), - "TRIM" => args.Length == 1 ? (object?)args[0]?.ToString()?.Trim() : throw new ArgumentException("TRIM requires 1 argument"), - "LTRIM" => args.Length == 1 ? (object?)args[0]?.ToString()?.TrimStart() : throw new ArgumentException("LTRIM requires 1 argument"), - "RTRIM" => args.Length == 1 ? (object?)args[0]?.ToString()?.TrimEnd() : throw new ArgumentException("RTRIM requires 1 argument"), - "SUBSTR" or "SUBSTRING" => args.Length >= 2 && args.Length <= 3 ? (object?)(args[0]?.ToString() is string sstr && args[1] is int sstart ? sstr.Length >= sstart && sstart > 0 ? sstr.Substring(sstart - 1, Math.Min(args.Length == 3 && args[2] is int sl ? sl : sstr.Length - sstart + 1, sstr.Length - sstart + 1)) : "" : null) : throw new ArgumentException("SUBSTR requires 2 or 3 arguments"), - "REPLACE" => args.Length == 3 ? (object?)args[0]?.ToString()?.Replace(args[1]?.ToString() ?? "", args[2]?.ToString() ?? "") : throw new ArgumentException("REPLACE requires 3 arguments"), - "INSTR" => args.Length == 2 ? (object?)(args[0]?.ToString() is string istr && args[1]?.ToString() is string isubstr ? (istr.IndexOf(isubstr, StringComparison.Ordinal) is int iidx && iidx >= 0 ? iidx + 1 : 0) : 0) : throw new ArgumentException("INSTR requires 2 arguments"), - "REGEXP" => args.Length == 2 ? (object?)(args[0]?.ToString() is string rpattern && args[1]?.ToString() is string rinput ? Regex.IsMatch(rinput, rpattern, RegexOptions.None, TimeSpan.FromSeconds(1)) : false) : throw new ArgumentException("REGEXP requires 2 arguments"), - "GLOB" => args.Length == 2 ? (object?)(args[0]?.ToString() is string gpattern && args[1]?.ToString() is string gstr ? Regex.IsMatch(gstr, "^" + Regex.Escape(gpattern).Replace("\\*", ".*").Replace("\\?", ".") + "$", RegexOptions.IgnoreCase, TimeSpan.FromSeconds(1)) : false) : throw new ArgumentException("GLOB requires 2 arguments"), - "NOT" when args.Length == 1 && args[0] is bool nb => (object?)!nb, - "MOD" => args.Length == 2 ? (object?)(args[0] is double ma && args[1] is double mb && mb != 0 ? ma % mb : args[0] is int mia && args[1] is int mib && mib != 0 ? mia % mib : null) : throw new ArgumentException("MOD requires 2 arguments"), - "POW" or "POWER" => args.Length == 2 && args[0] is double pa && args[1] is double pb ? (object?)Math.Pow(pa, pb) : null, - "SQRT" => args.Length == 1 && args[0] is double sqd && sqd >= 0 ? (object?)Math.Sqrt(sqd) : null, - "SIGN" => args.Length == 1 ? (object?)(args[0] switch - { - double sd => Math.Sign(sd), - int si => Math.Sign(si), - long sl => Math.Sign(sl), - _ => 0 - }) : throw new ArgumentException("SIGN requires 1 argument"), - "MIN" => args.Length == 2 ? (args[0] is double mina && args[1] is double minb ? Math.Min(mina, minb) : args[0] is int minia && args[1] is int minib ? Math.Min(minia, minib) : args[0]) : throw new ArgumentException("MIN requires 2 arguments"), - "MAX" => args.Length == 2 ? (args[0] is double maxa && args[1] is double maxb ? Math.Max(maxa, maxb) : args[0] is int maxia && args[1] is int maxib ? Math.Max(maxia, maxib) : args[0]) : throw new ArgumentException("MAX requires 2 arguments"), - "RANDOM" => (object?)Random.Shared.NextInt64(), - _ => throw new NotSupportedException("Function '" + functionCall.FunctionName + "' is not supported.") - }; - } - - /// - /// Evaluates a function call node against a row, resolving arguments from the row context. - /// - private object? EvaluateFunctionCall(FunctionCallNode functionCall, Dictionary row) - { - ArgumentNullException.ThrowIfNull(functionCall); - ArgumentNullException.ThrowIfNull(row); - - var args = new object?[functionCall.Arguments.Count]; - for (int i = 0; i < functionCall.Arguments.Count; i++) - { - args[i] = functionCall.Arguments[i] switch - { - LiteralNode lit => lit.Value, - ColumnReferenceNode col => row.TryGetValue(col.ColumnName, out var v) ? v : null, - FunctionCallNode nested => EvaluateFunctionCall(nested, row), - _ => null - }; - } - - return EvaluateScalarFunction(functionCall, args); - } - private List> HandleDerivedTable(string sql, bool noEncrypt, Dictionary? parameters = null) { // Route all queries (including JOINs) through the enhanced parser + AstExecutor @@ -3173,7 +3189,7 @@ private List> ApplyProjection(List args.Length == 2 && ToDouble(args[0]) is double pa && ToDouble(args[1]) is double pb ? (object?)Math.Pow(pa, pb) : null, "SQRT" => args.Length == 1 && ToDouble(args[0]) is double sd && sd >= 0 ? (object?)Math.Sqrt(sd) : null, "RANDOM" => (object?)Random.Shared.NextInt64(), - _ => null + _ => SqlScalarExpression.TryInvokeScalarFunction(name, args, out var shared) ? shared : null }; } @@ -3402,7 +3418,9 @@ private bool EvaluateInExpression(InExpressionNode inExpr, Dictionary literal.Value, ColumnReferenceNode column => GetValue(row, column), ParameterNode parameter => ResolveParameter(parameter.ParameterName), - FunctionCallNode => null, // Function calls in WHERE are handled by EvaluateBinaryExpression + // A function operand is evaluated like a SELECT-list function; it used to be NULL, so + // strftime('%Y', at) = '2026' matched no row on this path either. + FunctionCallNode func => EvaluateFunctionCallInRow(func, row), _ => throw new NotSupportedException($"Expression type '{expression.GetType().Name}' is not supported for value extraction.") }; } diff --git a/src/SharpCoreDB/Services/SqlParser.Helpers.cs b/src/SharpCoreDB/Services/SqlParser.Helpers.cs index 6fd76580..18a3343b 100644 --- a/src/SharpCoreDB/Services/SqlParser.Helpers.cs +++ b/src/SharpCoreDB/Services/SqlParser.Helpers.cs @@ -341,6 +341,14 @@ public static bool EvaluateJoinWhere(Dictionary row, string wher // like "a = 1 OR b = 2" (a leading "(" would otherwise stick to the column name). where = SqlInPredicate.StripOuterParentheses(where); + // A computed operand (strftime('%Y', at) = '2025', length(name) = 4) is evaluated by the shared scalar + // evaluator. The whitespace split below cut a function call apart, so UPDATE and DELETE with such a WHERE + // changed no row. + if (SqlScalarExpression.TryEvaluateCondition(row, where, out var computedMatch)) + { + return computedMatch; + } + // SQL whitespace may be any kind: splitting on the space character alone left a tab- or // newline-separated condition as a single unusable token. var parts = where.Split([' ', '\t', '\r', '\n'], StringSplitOptions.RemoveEmptyEntries); diff --git a/src/SharpCoreDB/Services/SqlParser.PerformanceOptimizations.cs b/src/SharpCoreDB/Services/SqlParser.PerformanceOptimizations.cs index 2078b544..78991d8a 100644 --- a/src/SharpCoreDB/Services/SqlParser.PerformanceOptimizations.cs +++ b/src/SharpCoreDB/Services/SqlParser.PerformanceOptimizations.cs @@ -137,6 +137,13 @@ public static Func, bool> CompileWhereClause(string w // Pattern: columnName [operator] value // Operators: =, !=, >, <, >=, <= + // The compiled form understands `column op value` chains only. A parenthesis means a function or other computed + // operand (length(name) = 3), which it would read as a column name; the shared evaluator handles it instead. + if (whereClause.Contains('(')) + { + return row => SqlParser.EvaluateJoinWhere(row, whereClause); + } + try { // Split by AND/OR (simple implementation) @@ -147,9 +154,9 @@ public static Func, bool> CompileWhereClause(string w } catch (Exception) { - // Fallback: Accept all rows if parsing fails - // Better than throwing exception - return row => true; + // A condition this cannot compile is evaluated by the shared evaluator. Up to 2.1.4 it accepted every row, + // so an unsupported WHERE silently returned the whole table. + return row => SqlParser.EvaluateJoinWhere(row, whereClause); } } diff --git a/src/SharpCoreDB/Services/SqlScalarExpression.cs b/src/SharpCoreDB/Services/SqlScalarExpression.cs new file mode 100644 index 00000000..fa474d78 --- /dev/null +++ b/src/SharpCoreDB/Services/SqlScalarExpression.cs @@ -0,0 +1,1387 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Services; + +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Globalization; +using System.Text; +using System.Text.RegularExpressions; + +/// +/// Parses and evaluates a scalar SQL expression against a row: literals, column references, SQLite's scalar +/// functions (string, numeric and date/time), CAST, || and arithmetic — and the WHERE conditions +/// built from them. It is the one evaluator every row path shares: the SELECT list of the single-table path, the +/// directory-mode and single-file WHERE evaluators, the UPDATE/DELETE scan evaluator, the AST executor's function +/// fallback, and a SELECT without FROM. +/// +/// Up to 2.1.4 a function applied to a column was not evaluated on any of those paths: the SELECT list returned the +/// row unchanged, and every WHERE evaluator read the left operand as a column name, so +/// strftime('%Y', at) = '2026' matched no row — the shape linq2db's SQLite dialect emits for every +/// comparison. A condition is only taken here when one of its operands is a computed +/// expression; plain column op literal conditions keep their existing evaluators, collations and fast paths. +/// +/// +internal static class SqlScalarExpression +{ + private const int CacheCapacity = 4096; + + private static readonly ConcurrentDictionary ExpressionCache = new(StringComparer.Ordinal); + private static readonly ConcurrentDictionary ConditionCache = new(StringComparer.Ordinal); + private static readonly Dictionary EmptyRow = new(StringComparer.OrdinalIgnoreCase); + + private static readonly Regex NumericPrefix = new( + @"^\s*[+-]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?", + RegexOptions.CultureInvariant | RegexOptions.Compiled, + TimeSpan.FromSeconds(1)); + + private static readonly HashSet ScalarFunctions = new(StringComparer.Ordinal) + { + "UPPER", "LOWER", "LENGTH", "SUBSTR", "SUBSTRING", "TRIM", "LTRIM", "RTRIM", "REPLACE", "INSTR", + "COALESCE", "IFNULL", "NULLIF", "IIF", "TYPEOF", "ABS", "ROUND", "SIGN", "MIN", "MAX", "QUOTE", "CONCAT", + "DATE", "TIME", "DATETIME", "JULIANDAY", "UNIXEPOCH", "STRFTIME", + }; + + // ── Public surface ────────────────────────────────────────────────────────────────────────── + + /// Parses as one complete scalar expression (cached). + internal static bool TryParse(string text, out Node node) + { + node = GetOrAdd(ExpressionCache, text, static t => new Parser(t).ParseWhole())!; + return node is not null; + } + + /// Returns true when the node is computed — a function, CAST, operator — not a literal or column. + internal static bool IsComputed(Node node) => node is not (Literal or Column); + + /// Returns true when the node is a plain column reference. + internal static bool IsColumn(Node node) => node is Column; + + /// The column name of a plain column reference; null for anything else. + internal static string? ColumnName(Node node) => node is Column column ? column.Name : null; + + /// Returns true for a column reference written with a table qualifier (x.name). + internal static bool IsQualified(Node node) => node is Column { Qualifier: not null }; + + /// Evaluates a parsed expression against a row (null is SQL NULL). + internal static object? Evaluate(Node node, Dictionary? row) => Eval(node, row ?? EmptyRow); + + /// + /// Evaluates a WHERE condition against a row when it involves a computed expression. Returns false — + /// "not mine" — when the condition has no computed operand or uses syntax this evaluator does not cover, so the + /// caller's own evaluator stays in charge of it. + /// + /// The row. + /// The condition text. + /// Whether the row matches (SQL NULL counts as no match). + internal static bool TryEvaluateCondition(Dictionary row, string condition, out bool result) + { + result = false; + if (string.IsNullOrWhiteSpace(condition) || condition.IndexOf('(') < 0) + { + return false; // every computed operand this evaluator handles contains a parenthesis + } + + var compiled = GetOrAdd(ConditionCache, condition, static c => ConditionCompiler.Compile(c)); + if (compiled is null) + { + return false; + } + + result = compiled.Evaluate(row) == true; + return true; + } + + /// SQLite's text form of a value (what upper, length, || and CAST AS TEXT see). + internal static string ToText(object? value) => value switch + { + null or DBNull => string.Empty, + string s => s, + bool b => b ? "1" : "0", + double d => FormatReal(d), + float f => FormatReal(f), + DateTime dt => dt.ToString(SqlLiteralFormatter.DateTimeFormat, CultureInfo.InvariantCulture), + DateTimeOffset dto => dto.ToString(SqlLiteralFormatter.DateTimeOffsetFormat, CultureInfo.InvariantCulture), + byte[] bytes => Encoding.UTF8.GetString(bytes), + Guid g => g.ToString("D"), + _ => Convert.ToString(value, CultureInfo.InvariantCulture) ?? string.Empty, + }; + + // ── Expression nodes ──────────────────────────────────────────────────────────────────────── + + /// A parsed scalar expression. + internal abstract class Node; + + private sealed class Literal(object? value) : Node + { + public object? Value { get; } = value; + } + + private sealed class Column(string? qualifier, string name, bool doubleQuoted) : Node + { + public string? Qualifier { get; } = qualifier; + public string Name { get; } = name; + public bool DoubleQuoted { get; } = doubleQuoted; + } + + private sealed class Call(string name, Node[] args) : Node + { + public string Name { get; } = name; + public Node[] Args { get; } = args; + } + + private sealed class Cast(Node operand, string typeName) : Node + { + public Node Operand { get; } = operand; + public string TypeName { get; } = typeName; + } + + private sealed class Binary(string op, Node left, Node right) : Node + { + public string Op { get; } = op; + public Node Left { get; } = left; + public Node Right { get; } = right; + } + + private sealed class Negate(Node operand) : Node + { + public Node Operand { get; } = operand; + } + + // ── Parser ────────────────────────────────────────────────────────────────────────────────── + + private sealed class ParseFailure : Exception; + + private sealed class Parser(string text) + { + private int _pos; + + public Node? ParseWhole() + { + try + { + var node = ParseExpression(0); + SkipWhitespace(); + return _pos == text.Length ? node : null; + } + catch (ParseFailure) + { + return null; + } + } + + private Node ParseExpression(int minPrecedence) + { + var left = ParseUnary(); + while (true) + { + SkipWhitespace(); + var (op, precedence) = PeekBinaryOperator(); + if (op is null || precedence < minPrecedence) + { + return left; + } + + _pos += op.Length; + var right = ParseExpression(precedence + 1); + left = new Binary(op, left, right); + } + } + + private (string? Op, int Precedence) PeekBinaryOperator() + { + if (_pos >= text.Length) + { + return (null, 0); + } + + // SQLite precedence, tightest first: ||, then * / %, then + -. + if (text.AsSpan(_pos).StartsWith("||", StringComparison.Ordinal)) + { + return ("||", 3); + } + + return text[_pos] switch + { + '*' or '/' or '%' => (text[_pos].ToString(), 2), + '+' or '-' => (text[_pos].ToString(), 1), + _ => (null, 0), + }; + } + + private Node ParseUnary() + { + SkipWhitespace(); + if (_pos < text.Length && text[_pos] == '-') + { + _pos++; + var operand = ParseUnary(); + return operand is Literal { Value: long l } ? new Literal(-l) + : operand is Literal { Value: double d } ? new Literal(-d) + : new Negate(operand); + } + + if (_pos < text.Length && text[_pos] == '+') + { + _pos++; + return ParseUnary(); + } + + return ParsePrimary(); + } + + private Node ParsePrimary() + { + SkipWhitespace(); + if (_pos >= text.Length) + { + throw new ParseFailure(); + } + + char c = text[_pos]; + if (c == '(') + { + _pos++; + var inner = ParseExpression(0); + Expect(')'); + return inner; + } + + if (c == '\'') + { + return new Literal(ReadStringLiteral()); + } + + if (char.IsAsciiDigit(c) || (c == '.' && _pos + 1 < text.Length && char.IsAsciiDigit(text[_pos + 1]))) + { + return ReadNumber(); + } + + if (c is '[' or '"' or '`') + { + return ReadColumn(); + } + + if (char.IsLetter(c) || c == '_') + { + int start = _pos; + var word = ReadWord(); + SkipWhitespace(); + if (_pos < text.Length && text[_pos] == '(') + { + return ReadCall(word); + } + + switch (word.ToUpperInvariant()) + { + case "NULL": + return new Literal(null); + case "TRUE": + return new Literal(1L); + case "FALSE": + return new Literal(0L); + case "CURRENT_TIMESTAMP": + return new Call("DATETIME", [new Literal("now")]); + case "CURRENT_DATE": + return new Call("DATE", [new Literal("now")]); + case "CURRENT_TIME": + return new Call("TIME", [new Literal("now")]); + } + + _pos = start; + return ReadColumn(); + } + + throw new ParseFailure(); + } + + private Node ReadCall(string word) + { + var name = word.ToUpperInvariant(); + Expect('('); + + if (name == "CAST") + { + var operand = ParseExpression(0); + SkipWhitespace(); + if (!ReadKeyword("AS")) + { + throw new ParseFailure(); + } + + int typeStart = _pos; + int depth = 0; + while (_pos < text.Length && (text[_pos] != ')' || depth > 0)) + { + depth += text[_pos] == '(' ? 1 : text[_pos] == ')' ? -1 : 0; + _pos++; + } + + var typeName = text[typeStart.._pos].Trim(); + Expect(')'); + return typeName.Length == 0 ? throw new ParseFailure() : new Cast(operand, typeName); + } + + if (!ScalarFunctions.Contains(name)) + { + throw new ParseFailure(); // aggregates, unknown and engine-state functions stay on their own paths + } + + var args = new List(); + SkipWhitespace(); + if (_pos < text.Length && text[_pos] == ')') + { + _pos++; + } + else + { + while (true) + { + args.Add(ParseExpression(0)); + SkipWhitespace(); + if (_pos < text.Length && text[_pos] == ',') + { + _pos++; + continue; + } + + Expect(')'); + break; + } + } + + // One-argument MIN/MAX is the aggregate, not the scalar. + return name is "MIN" or "MAX" && args.Count < 2 ? throw new ParseFailure() : new Call(name, [.. args]); + } + + private Column ReadColumn() + { + var (first, firstQuoted) = ReadIdentifier(); + SkipWhitespace(); + if (_pos < text.Length && text[_pos] == '.') + { + _pos++; + SkipWhitespace(); + var (second, secondQuoted) = ReadIdentifier(); + return new Column(first, second, secondQuoted); + } + + return new Column(null, first, firstQuoted); + } + + private (string Name, bool DoubleQuoted) ReadIdentifier() + { + if (_pos >= text.Length) + { + throw new ParseFailure(); + } + + char open = text[_pos]; + char close = open switch { '[' => ']', '"' => '"', '`' => '`', _ => '\0' }; + if (close != '\0') + { + int end = text.IndexOf(close, _pos + 1); + if (end < 0) + { + throw new ParseFailure(); + } + + var name = text[(_pos + 1)..end]; + _pos = end + 1; + return (name, open == '"'); + } + + if (!(char.IsLetter(open) || open == '_')) + { + throw new ParseFailure(); + } + + return (ReadWord(), false); + } + + private string ReadWord() + { + int start = _pos; + while (_pos < text.Length && (char.IsLetterOrDigit(text[_pos]) || text[_pos] is '_' or '$')) + { + _pos++; + } + + return text[start.._pos]; + } + + private bool ReadKeyword(string keyword) + { + if (_pos + keyword.Length <= text.Length && + text.AsSpan(_pos, keyword.Length).Equals(keyword, StringComparison.OrdinalIgnoreCase) && + (_pos + keyword.Length == text.Length || !char.IsLetterOrDigit(text[_pos + keyword.Length]))) + { + _pos += keyword.Length; + SkipWhitespace(); + return true; + } + + return false; + } + + private string ReadStringLiteral() + { + var sb = new StringBuilder(); + _pos++; // opening quote + while (_pos < text.Length) + { + char c = text[_pos++]; + if (c != '\'') + { + sb.Append(c); + continue; + } + + if (_pos < text.Length && text[_pos] == '\'') + { + sb.Append('\''); // '' is one quote + _pos++; + continue; + } + + return sb.ToString(); + } + + throw new ParseFailure(); // unterminated literal + } + + private Literal ReadNumber() + { + int start = _pos; + bool isReal = false; + while (_pos < text.Length && (char.IsAsciiDigit(text[_pos]) || text[_pos] == '.')) + { + isReal |= text[_pos] == '.'; + _pos++; + } + + if (_pos < text.Length && text[_pos] is 'e' or 'E') + { + isReal = true; + _pos++; + if (_pos < text.Length && text[_pos] is '+' or '-') + { + _pos++; + } + + while (_pos < text.Length && char.IsAsciiDigit(text[_pos])) + { + _pos++; + } + } + + var span = text.AsSpan(start, _pos - start); + if (!isReal && long.TryParse(span, NumberStyles.Integer, CultureInfo.InvariantCulture, out var l)) + { + return new Literal(l); + } + + return double.TryParse(span, NumberStyles.Float, CultureInfo.InvariantCulture, out var d) + ? new Literal(d) + : throw new ParseFailure(); + } + + private void Expect(char c) + { + SkipWhitespace(); + if (_pos >= text.Length || text[_pos] != c) + { + throw new ParseFailure(); + } + + _pos++; + } + + private void SkipWhitespace() + { + while (_pos < text.Length && char.IsWhiteSpace(text[_pos])) + { + _pos++; + } + } + } + + // ── Evaluation ────────────────────────────────────────────────────────────────────────────── + + private static object? Eval(Node node, Dictionary row) => node switch + { + Literal literal => literal.Value, + Column column => ResolveColumn(row, column), + Call call => InvokeFunction(call, row), + Cast cast => CastValue(Eval(cast.Operand, row), cast.TypeName), + Binary binary => EvalBinary(binary.Op, Eval(binary.Left, row), Eval(binary.Right, row)), + Negate negate => EvalBinary("-", 0L, Eval(negate.Operand, row)), + _ => null, + }; + + private static object? ResolveColumn(Dictionary row, Column column) + { + if (column.Qualifier is not null && row.TryGetValue($"{column.Qualifier}.{column.Name}", out var qualified)) + { + return Normalize(qualified); + } + + if (row.TryGetValue(column.Name, out var exact)) + { + return Normalize(exact); + } + + foreach (var (key, value) in row) + { + if (key.Equals(column.Name, StringComparison.OrdinalIgnoreCase) || + key.EndsWith("." + column.Name, StringComparison.OrdinalIgnoreCase)) + { + return Normalize(value); + } + } + + // SQLite reads a double-quoted name that is not a column as a string literal. + return column.DoubleQuoted && column.Qualifier is null ? column.Name : null; + } + + private static object? Normalize(object? value) => value is DBNull ? null : value; + + private static object? InvokeFunction(Call call, Dictionary row) + { + var args = new object?[call.Args.Length]; + for (int i = 0; i < args.Length; i++) + { + args[i] = Eval(call.Args[i], row); + } + + return InvokeScalarFunction(call.Name, args); + } + + /// + /// Evaluates one of SQLite's scalar functions over already-evaluated arguments. Returns false for a name + /// this evaluator does not implement. Shared with the AST executor so both paths agree. + /// + /// The upper-case function name. + /// The evaluated arguments. + /// The function result (null is SQL NULL). + internal static bool TryInvokeScalarFunction(string upperName, object?[] args, out object? result) + { + result = null; + if (!ScalarFunctions.Contains(upperName) || (upperName is "MIN" or "MAX" && args.Length < 2)) + { + return false; + } + + for (int i = 0; i < args.Length; i++) + { + args[i] = Normalize(args[i]); + } + + result = InvokeScalarFunction(upperName, args); + return true; + } + + private static object? InvokeScalarFunction(string name, object?[] a) + { + if (SqlDateTimeFunctions.IsDateTimeFunction(name)) + { + return SqlDateTimeFunctions.Evaluate(name, a); + } + + object? Arg(int i) => i < a.Length ? a[i] : null; + + switch (name) + { + case "UPPER": + return Arg(0) is null ? null : ToText(Arg(0)).ToUpperInvariant(); + case "LOWER": + return Arg(0) is null ? null : ToText(Arg(0)).ToLowerInvariant(); + case "LENGTH": + return Arg(0) switch + { + null => null, + byte[] bytes => (long)bytes.Length, + var v => (long)ToText(v).Length, + }; + case "SUBSTR" or "SUBSTRING": + return a.Length is < 2 or > 3 || Arg(0) is null || Arg(1) is null || (a.Length == 3 && Arg(2) is null) + ? null + : Substr(ToText(Arg(0)), ToInteger(Arg(1)), a.Length == 3 ? ToInteger(Arg(2)) : null); + case "TRIM" or "LTRIM" or "RTRIM": + { + if (Arg(0) is null || (a.Length > 1 && Arg(1) is null)) + { + return null; + } + + var chars = a.Length > 1 ? ToText(Arg(1)).ToCharArray() : [' ']; + var s = ToText(Arg(0)); + return name switch + { + "LTRIM" => s.TrimStart(chars), + "RTRIM" => s.TrimEnd(chars), + _ => s.Trim(chars), + }; + } + + case "REPLACE": + if (a.Length != 3 || Arg(0) is null || Arg(1) is null || Arg(2) is null) + { + return null; + } + + var find = ToText(Arg(1)); + return find.Length == 0 ? ToText(Arg(0)) : ToText(Arg(0)).Replace(find, ToText(Arg(2)), StringComparison.Ordinal); + case "INSTR": + return Arg(0) is null || Arg(1) is null + ? null + : (long)(ToText(Arg(0)).IndexOf(ToText(Arg(1)), StringComparison.Ordinal) + 1); + case "COALESCE": + foreach (var v in a) + { + if (v is not null) + { + return v; + } + } + + return null; + case "IFNULL": + return a.Length == 2 ? Arg(0) ?? Arg(1) : null; + case "NULLIF": + return a.Length == 2 && TryCompare(Arg(0), Arg(1), out var cmp) && cmp == 0 ? null : Arg(0); + case "IIF": + return a.Length == 3 ? (IsTrue(Arg(0)) == true ? Arg(1) : Arg(2)) : null; + case "TYPEOF": + return Arg(0) switch + { + null => "null", + sbyte or byte or short or ushort or int or uint or long or ulong or bool => "integer", + float or double or decimal => "real", + byte[] => "blob", + _ => "text", + }; + case "ABS": + return ToNumber(Arg(0)) switch + { + null => null, + long l => Math.Abs(l), + double d => (object)Math.Abs(d), + _ => null, + }; + case "ROUND": + { + if (Arg(0) is null || (a.Length > 1 && Arg(1) is null)) + { + return null; + } + + int digits = a.Length > 1 ? (int)Math.Clamp(ToInteger(Arg(1)), 0, 15) : 0; + return Math.Round(ToReal(Arg(0)), digits, MidpointRounding.AwayFromZero); + } + + case "SIGN": + return ToNumber(Arg(0)) switch + { + long l => (long)Math.Sign(l), + double d => (long)Math.Sign(d), + _ => null, + }; + case "MIN" or "MAX": + { + object? best = null; + foreach (var v in a) + { + if (v is null) + { + return null; + } + + if (best is null || (TryCompare(v, best, out var c) && (name == "MIN" ? c < 0 : c > 0))) + { + best = v; + } + } + + return best; + } + + case "QUOTE": + return SqlFunctions.Quote(Arg(0)); + case "CONCAT": + { + var sb = new StringBuilder(); + foreach (var v in a) + { + sb.Append(ToText(v)); + } + + return sb.ToString(); + } + + default: + return null; + } + } + + /// SQLite's substr(): 1-based start, a negative start counts from the end, a negative length looks back. + private static string Substr(string s, long start, long? length) + { + long len = s.Length; + long p1 = start; + long p2 = length ?? len; + bool negativeLength = p2 < 0; + if (negativeLength) + { + p2 = -p2; + } + + if (p1 < 0) + { + p1 += len; + if (p1 < 0) + { + p2 += p1; + if (p2 < 0) + { + p2 = 0; + } + + p1 = 0; + } + } + else if (p1 > 0) + { + p1--; + } + else if (p2 > 0) + { + p2--; + } + + if (negativeLength) + { + p1 -= p2; + if (p1 < 0) + { + p2 += p1; + p1 = 0; + } + } + + if (p1 > len) + { + return string.Empty; + } + + if (p1 + p2 > len) + { + p2 = len - p1; + } + + return p2 <= 0 ? string.Empty : s.Substring((int)p1, (int)p2); + } + + private static object? CastValue(object? value, string typeName) + { + if (value is null) + { + return null; + } + + // SQLite's affinity rules, by substring of the declared type name. + var type = typeName.ToUpperInvariant(); + if (type.Contains("INT", StringComparison.Ordinal)) + { + return ToInteger(value); + } + + if (type.Contains("CHAR", StringComparison.Ordinal) || type.Contains("CLOB", StringComparison.Ordinal) || + type.Contains("TEXT", StringComparison.Ordinal)) + { + return ToText(value); + } + + if (type.Contains("BLOB", StringComparison.Ordinal)) + { + return value is byte[] ? value : Encoding.UTF8.GetBytes(ToText(value)); + } + + if (type.Contains("REAL", StringComparison.Ordinal) || type.Contains("FLOA", StringComparison.Ordinal) || + type.Contains("DOUB", StringComparison.Ordinal)) + { + return ToReal(value); + } + + // NUMERIC: an integer when the value is integral, otherwise a real. + return ToNumber(value) switch + { + double d when double.IsInteger(d) && Math.Abs(d) < 9.2e18 => (long)d, + var n => n, + }; + } + + private static object? EvalBinary(string op, object? left, object? right) + { + if (left is null || right is null) + { + return null; + } + + if (op == "||") + { + return ToText(left) + ToText(right); + } + + var l = ToNumber(left); + var r = ToNumber(right); + if (l is long li && r is long ri) + { + return op switch + { + "+" => li + ri, + "-" => li - ri, + "*" => li * ri, + "/" => ri == 0 ? null : li / ri, + "%" => ri == 0 ? null : li % ri, + _ => null, + }; + } + + double ld = ToReal(l); + double rd = ToReal(r); + return op switch + { + "+" => ld + rd, + "-" => ld - rd, + "*" => ld * rd, + "/" => rd == 0 ? null : ld / rd, + "%" => rd == 0 ? null : (double)((long)ld % (long)rd), + _ => null, + }; + } + + // ── Values: numbers, truth, comparison ────────────────────────────────────────────────────── + + /// The numeric value SQLite gives a value: a or a ; NULL stays null. + private static object? ToNumber(object? value) + { + switch (value) + { + case null or DBNull: + return null; + case bool b: + return b ? 1L : 0L; + case sbyte or byte or short or ushort or int or uint or long: + return Convert.ToInt64(value, CultureInfo.InvariantCulture); + case ulong u: + return u <= long.MaxValue ? (long)u : (double)u; + case float or double or decimal: + return Convert.ToDouble(value, CultureInfo.InvariantCulture); + } + + // Text converts by its longest numeric prefix ('12abc' is 12, 'abc' is 0), as in SQLite. + var m = NumericPrefix.Match(ToText(value)); + if (!m.Success) + { + return 0L; + } + + var text = m.Value.Trim(); + if (long.TryParse(text, NumberStyles.AllowLeadingSign, CultureInfo.InvariantCulture, out var l)) + { + return l; + } + + return double.TryParse(text, NumberStyles.Float, CultureInfo.InvariantCulture, out var d) ? d : 0L; + } + + private static long ToInteger(object? value) => ToNumber(value) switch + { + long l => l, + double d when !double.IsNaN(d) => d >= 9.2e18 ? long.MaxValue : d <= -9.2e18 ? long.MinValue : (long)Math.Truncate(d), + _ => 0L, + }; + + private static double ToReal(object? value) => ToNumber(value) switch + { + long l => l, + double d => d, + _ => 0d, + }; + + /// SQLite's truth of a value: NULL is unknown, otherwise its numeric value is non-zero. + private static bool? IsTrue(object? value) => ToNumber(value) switch + { + null => null, + long l => l != 0, + double d => d != 0, + _ => false, + }; + + private static bool IsNumeric(object value) => + value is sbyte or byte or short or ushort or int or uint or long or ulong or float or double or decimal or bool; + + /// + /// Compares two non-NULL values as SQLite does: numbers numerically (a numeric-looking text operand is read as a + /// number when the other side is a number, the leniency the engine's other evaluators already apply), two text + /// values by their bytes (BINARY collation) — '01' is not '1' — and + /// across classes numbers before text before blobs. Returns false when either side is NULL. + /// + internal static bool TryCompare(object? left, object? right, out int comparison) + { + comparison = 0; + left = Normalize(left); + right = Normalize(right); + if (left is null || right is null) + { + return false; + } + + if (left is DateTime or DateTimeOffset or Guid) + { + left = ToText(left); + } + + if (right is DateTime or DateTimeOffset or Guid) + { + right = ToText(right); + } + + bool leftNumeric = IsNumeric(left); + bool rightNumeric = IsNumeric(right); + if (leftNumeric && !rightNumeric && right is string rs && IsWholeNumber(rs)) + { + rightNumeric = true; + } + else if (rightNumeric && !leftNumeric && left is string ls && IsWholeNumber(ls)) + { + leftNumeric = true; + } + if (leftNumeric && rightNumeric) + { + var l = ToNumber(left); + var r = ToNumber(right); + comparison = l is long li && r is long ri ? li.CompareTo(ri) : ToReal(l).CompareTo(ToReal(r)); + return true; + } + + int leftClass = ClassOf(left, leftNumeric); + int rightClass = ClassOf(right, rightNumeric); + if (leftClass != rightClass) + { + comparison = leftClass.CompareTo(rightClass); + return true; + } + + if (left is byte[] lb && right is byte[] rb) + { + comparison = lb.AsSpan().SequenceCompareTo(rb); + return true; + } + + comparison = Math.Sign(string.CompareOrdinal(ToText(left), ToText(right))); + return true; + } + + private static int ClassOf(object value, bool numeric) => numeric ? 0 : value is byte[] ? 2 : 1; + + private static bool IsWholeNumber(string text) + { + var trimmed = text.Trim(); + return trimmed.Length > 0 && + double.TryParse(trimmed, NumberStyles.Float, CultureInfo.InvariantCulture, out _) && + NumericPrefix.Match(trimmed) is { Success: true } m && m.Length == trimmed.Length; + } + + private static string FormatReal(double value) + { + // SQLite prints a real with a decimal point: 5.0, not 5. + var text = value.ToString("R", CultureInfo.InvariantCulture); + return double.IsFinite(value) && !text.Contains('.') && !text.Contains('E') ? text + ".0" : text; + } + + private static TValue? GetOrAdd(ConcurrentDictionary cache, string key, Func factory) + where TValue : class + { + if (cache.TryGetValue(key, out var cached)) + { + return cached; + } + + if (cache.Count >= CacheCapacity) + { + cache.Clear(); // the statements a process uses are few; a full clear keeps the bound trivially + } + + var value = factory(key); + cache[key] = value; + return value; + } + + // ── Conditions ────────────────────────────────────────────────────────────────────────────── + + private abstract class Condition + { + /// SQL three-valued result: null is unknown. + public abstract bool? Evaluate(Dictionary row); + } + + private sealed class AndCondition(Condition[] parts) : Condition + { + public override bool? Evaluate(Dictionary row) + { + bool unknown = false; + foreach (var part in parts) + { + var value = part.Evaluate(row); + if (value == false) + { + return false; + } + + unknown |= value is null; + } + + return unknown ? null : true; + } + } + + private sealed class OrCondition(Condition[] parts) : Condition + { + public override bool? Evaluate(Dictionary row) + { + bool unknown = false; + foreach (var part in parts) + { + var value = part.Evaluate(row); + if (value == true) + { + return true; + } + + unknown |= value is null; + } + + return unknown ? null : false; + } + } + + private sealed class NotCondition(Condition inner) : Condition + { + public override bool? Evaluate(Dictionary row) => !inner.Evaluate(row); + } + + private sealed class CompareCondition(string op, Node left, Node right) : Condition + { + public override bool? Evaluate(Dictionary row) + { + if (!TryCompare(Eval(left, row), Eval(right, row), out var c)) + { + return null; + } + + return op switch + { + "=" or "==" => c == 0, + "!=" or "<>" => c != 0, + "<" => c < 0, + "<=" => c <= 0, + ">" => c > 0, + ">=" => c >= 0, + _ => null, + }; + } + } + + private sealed class IsCondition(Node left, Node right, bool negated) : Condition + { + public override bool? Evaluate(Dictionary row) + { + // IS / IS NOT never yield NULL: NULL IS NULL is true. + var l = Eval(left, row); + var r = Eval(right, row); + bool same = l is null || r is null ? l is null && r is null : TryCompare(l, r, out var c) && c == 0; + return negated ? !same : same; + } + } + + private sealed class LikeCondition(Node value, Node pattern, Node? escape, bool negated) : Condition + { + public override bool? Evaluate(Dictionary row) + { + var v = Eval(value, row); + var p = Eval(pattern, row); + if (v is null || p is null) + { + return null; + } + + char? esc = null; + if (escape is not null) + { + var e = ToText(Eval(escape, row)); + esc = e.Length == 1 ? e[0] : null; + } + + bool matched = SqlLikePattern.IsMatch(ToText(v), ToText(p), esc); + return negated ? !matched : matched; + } + } + + private sealed class TruthCondition(Node expression) : Condition + { + public override bool? Evaluate(Dictionary row) => IsTrue(Eval(expression, row)); + } + + private sealed class ConditionCompiler + { + private static readonly string[] SymbolOperators = ["<=", ">=", "<>", "!=", "==", "=", "<", ">"]; + + private bool _hasComputedOperand; + + public static Condition? Compile(string condition) + { + var compiler = new ConditionCompiler(); + var compiled = compiler.CompileCondition(condition); + return compiled is not null && compiler._hasComputedOperand ? compiled : null; + } + + private Condition? CompileCondition(string text) + { + text = SqlInPredicate.StripOuterParentheses(text.Trim()); + if (text.Length == 0 || FindTopLevelKeyword(text, "BETWEEN") >= 0) + { + return null; // BETWEEN's own AND would be split as a conjunction; leave it to the caller + } + + var orParts = SqlInPredicate.SplitTopLevelLogical(text, "OR"); + if (orParts.Count > 1) + { + return CompileAll(orParts, static p => new OrCondition(p)); + } + + var andParts = SqlInPredicate.SplitTopLevelLogical(text, "AND"); + if (andParts.Count > 1) + { + return CompileAll(andParts, static p => new AndCondition(p)); + } + + if (StartsWithKeyword(text, "NOT")) + { + var inner = CompileCondition(text[3..]); + return inner is null ? null : new NotCondition(inner); + } + + return CompileLeaf(text); + } + + private Condition? CompileAll(List parts, Func combine) + { + var compiled = new Condition[parts.Count]; + for (int i = 0; i < parts.Count; i++) + { + var part = CompileCondition(parts[i]); + if (part is null) + { + return null; + } + + compiled[i] = part; + } + + return combine(compiled); + } + + private Condition? CompileLeaf(string text) + { + // Keyword operators first: "a NOT LIKE b" also contains no symbol operator, but "x IS NOT NULL" must not + // be read as a comparison. + foreach (var keyword in (string[])["IN", "GLOB", "REGEXP", "MATCH", "EXISTS"]) + { + if (FindTopLevelKeyword(text, keyword) >= 0) + { + return null; + } + } + + int likeAt = FindTopLevelKeyword(text, "LIKE"); + if (likeAt >= 0) + { + var left = text[..likeAt].TrimEnd(); + bool negated = EndsWithKeyword(left, "NOT"); + if (negated) + { + left = left[..^3].TrimEnd(); + } + + var right = text[(likeAt + 4)..]; + string? escapeText = null; + int escapeAt = FindTopLevelKeyword(right, "ESCAPE"); + if (escapeAt >= 0) + { + escapeText = right[(escapeAt + 6)..]; + right = right[..escapeAt]; + } + + var value = ParseOperand(left); + var pattern = ParseOperand(right); + var escape = escapeText is null ? null : ParseOperand(escapeText); + return value is null || pattern is null || (escapeText is not null && escape is null) + ? null + : new LikeCondition(value, pattern, escape, negated); + } + + int isAt = FindTopLevelKeyword(text, "IS"); + if (isAt >= 0) + { + var rest = text[(isAt + 2)..].TrimStart(); + bool negated = StartsWithKeyword(rest, "NOT"); + if (negated) + { + rest = rest[3..]; + } + + var left = ParseOperand(text[..isAt]); + var right = ParseOperand(rest); + return left is null || right is null ? null : new IsCondition(left, right, negated); + } + + var (op, at) = FindTopLevelSymbolOperator(text); + if (op is null) + { + var expression = ParseOperand(text); + return expression is null ? null : new TruthCondition(expression); + } + + var lhs = ParseOperand(text[..at]); + var rhs = ParseOperand(text[(at + op.Length)..]); + return lhs is null || rhs is null ? null : new CompareCondition(op, lhs, rhs); + } + + private Node? ParseOperand(string text) + { + if (!TryParse(text.Trim(), out var node)) + { + return null; + } + + _hasComputedOperand |= IsComputed(node); + return node; + } + + private static (string? Op, int At) FindTopLevelSymbolOperator(string text) + { + int depth = 0; + for (int i = 0; i < text.Length; i++) + { + char c = text[i]; + if (TrySkipQuoted(text, ref i)) + { + continue; + } + + if (c == '(') + { + depth++; + } + else if (c == ')') + { + depth--; + } + else if (depth == 0 && c is '<' or '>' or '=' or '!') + { + foreach (var op in SymbolOperators) + { + if (text.AsSpan(i).StartsWith(op, StringComparison.Ordinal)) + { + return (op, i); + } + } + } + } + + return (null, -1); + } + + /// Finds a whole-word keyword outside quotes and parentheses; -1 when absent. + private static int FindTopLevelKeyword(string text, string keyword) + { + int depth = 0; + for (int i = 0; i < text.Length; i++) + { + char c = text[i]; + if (TrySkipQuoted(text, ref i)) + { + continue; + } + + if (c == '(') + { + depth++; + } + else if (c == ')') + { + depth--; + } + else if (depth == 0 && + (i == 0 || !IsWordChar(text[i - 1])) && + i + keyword.Length <= text.Length && + text.AsSpan(i, keyword.Length).Equals(keyword, StringComparison.OrdinalIgnoreCase) && + (i + keyword.Length == text.Length || !IsWordChar(text[i + keyword.Length]))) + { + return i; + } + } + + return -1; + } + + /// Advances past a quoted literal or identifier starting at . + private static bool TrySkipQuoted(string text, ref int i) + { + char open = text[i]; + char close = open switch { '\'' => '\'', '"' => '"', '`' => '`', '[' => ']', _ => '\0' }; + if (close == '\0') + { + return false; + } + + int j = i + 1; + while (j < text.Length) + { + if (text[j] == close) + { + if (open == '\'' && j + 1 < text.Length && text[j + 1] == '\'') + { + j += 2; // '' inside a string literal + continue; + } + + break; + } + + j++; + } + + i = Math.Min(j, text.Length - 1); + return true; + } + + private static bool StartsWithKeyword(string text, string keyword) => + text.Length > keyword.Length && + text.StartsWith(keyword, StringComparison.OrdinalIgnoreCase) && + !IsWordChar(text[keyword.Length]); + + private static bool EndsWithKeyword(string text, string keyword) => + text.Length > keyword.Length && + text.EndsWith(keyword, StringComparison.OrdinalIgnoreCase) && + !IsWordChar(text[^(keyword.Length + 1)]); + + private static bool IsWordChar(char c) => char.IsLetterOrDigit(c) || c is '_' or '$'; + } +} diff --git a/src/SharpCoreDB/SingleFileTable.cs b/src/SharpCoreDB/SingleFileTable.cs index bb9929ac..5c0870c4 100644 --- a/src/SharpCoreDB/SingleFileTable.cs +++ b/src/SharpCoreDB/SingleFileTable.cs @@ -173,13 +173,14 @@ public SingleFileTable(string tableName, List columns, List co ColumnTypes = columnTypes; PrimaryKeyIndex = primaryKeyIndex; - // Copy constraint lists + // Defaults first, then the declared constraints. Up to 2.1.4 InitializeColumnMetadata() ran after the copy and + // reset NOT NULL and AUTOINCREMENT to false for the session that created the table. + InitializeColumnMetadata(); + IsNotNull.Clear(); IsNotNull.AddRange(isNotNull); IsAuto.Clear(); IsAuto.AddRange(isAuto); - - InitializeColumnMetadata(); } /// @@ -254,6 +255,7 @@ public void Insert(Dictionary row) lock (_tableLock) { ApplyDefaults(row); + ValidateNotNull(row, rowIndex: null); _rowCache.Add(row); IndexRow(row); _isDirty = true; @@ -277,10 +279,16 @@ public long[] InsertBatch(List> rows) lock (_tableLock) { + // Every row is checked before any is cached, so a NOT NULL violation leaves no part of the batch behind. + for (int i = 0; i < rows.Count; i++) + { + ApplyDefaults(rows[i]); + ValidateNotNull(rows[i], rowIndex: i); + } + for (int i = 0; i < rows.Count; i++) { var row = rows[i]; - ApplyDefaults(row); _rowCache.Add(row); IndexRow(row); positions[i] = _rowCache.Count - 1; @@ -431,6 +439,10 @@ private static IEnumerable> ApplyOrderBy( bool updatesTouchPk = PrimaryKeyIndex >= 0 && updates.Keys.Any(k => string.Equals(k, Columns[PrimaryKeyIndex], StringComparison.OrdinalIgnoreCase)); + // SQLite raises NOT NULL only for a row the UPDATE actually changes, so it is checked at the first match, + // before anything is written. + var notNullViolation = FindNotNullViolation(updates); + int affected = 0; lock (_tableLock) { @@ -438,6 +450,11 @@ private static IEnumerable> ApplyOrderBy( { if (string.IsNullOrWhiteSpace(condition) || EvaluateCondition(row, condition)) { + if (notNullViolation is not null) + { + throw new InvalidOperationException($"Column '{notNullViolation}' cannot be NULL"); + } + string? oldPkKey = updatesTouchPk ? GetPkKey(row) : null; foreach (var update in updates) @@ -1351,6 +1368,43 @@ private bool IsPkIndexLookupSafe() or DataType.Boolean or DataType.Guid or DataType.Ulid; } + /// + /// Enforces NOT NULL as the directory-mode table does. Up to 2.1.4 the single-file table never checked it, so a NULL + /// went into a NOT NULL column. + /// + private void ValidateNotNull(Dictionary row, int? rowIndex) + { + for (int i = 0; i < Columns.Count && i < IsNotNull.Count; i++) + { + if (IsNotNull[i] && (!row.TryGetValue(Columns[i], out var value) || value is null or DBNull)) + { + throw new InvalidOperationException(rowIndex is null + ? $"Column '{Columns[i]}' cannot be NULL" + : $"Column '{Columns[i]}' cannot be NULL in row {rowIndex}"); + } + } + } + + /// Returns the first NOT NULL column an UPDATE would set to NULL, or null. + private string? FindNotNullViolation(Dictionary updates) + { + foreach (var (column, value) in updates) + { + if (value is not (null or DBNull)) + { + continue; + } + + int index = Columns.FindIndex(c => c.Equals(column, StringComparison.OrdinalIgnoreCase)); + if (index >= 0 && index < IsNotNull.Count && IsNotNull[index]) + { + return Columns[index]; + } + } + + return null; + } + private void ApplyDefaults(Dictionary row) { for (int i = 0; i < Columns.Count; i++) @@ -1654,8 +1708,11 @@ private static string NormalizeColumnName(string columnName) trimmed.Contains(" IN ", StringComparison.OrdinalIgnoreCase) || trimmed.Contains("LIKE", StringComparison.OrdinalIgnoreCase) || trimmed.Contains("BETWEEN", StringComparison.OrdinalIgnoreCase) || - trimmed.Contains(" IS ", StringComparison.OrdinalIgnoreCase)) + trimmed.Contains(" IS ", StringComparison.OrdinalIgnoreCase) || + trimmed.Contains('(')) { + // A parenthesis means a function or other computed operand (strftime('%Y', at) = '2026'): compiled here it + // became a lookup of a column literally named "strftime('%Y', at)" and matched no row. return null; } @@ -1842,6 +1899,13 @@ private static bool EvaluateSingleCondition(Dictionary row, stri { var trimmed = condition.Trim(); + // A computed operand (strftime, upper, length, CAST, …) is evaluated by the shared scalar evaluator; the + // patterns below read the left operand as a column name and matched no row. + if (SqlScalarExpression.TryEvaluateCondition(row, trimmed, out var computedMatch)) + { + return computedMatch; + } + // ✅ Parity: support IS NULL / IS NOT NULL (previously ignored → all rows returned) var isNullMatch = System.Text.RegularExpressions.Regex.Match( trimmed, "^(?" + ColumnReferencePattern + @")\s+IS\s+(?NOT\s+)?NULL$", diff --git a/tests/SharpCoreDB.Functional.Linq2DB.Tests/DateAndStringFilterTests.cs b/tests/SharpCoreDB.Functional.Linq2DB.Tests/DateAndStringFilterTests.cs new file mode 100644 index 00000000..21ba7ae2 --- /dev/null +++ b/tests/SharpCoreDB.Functional.Linq2DB.Tests/DateAndStringFilterTests.cs @@ -0,0 +1,133 @@ +using FluentAssertions; +using LinqToDB; +using LinqToDB.Mapping; +using Xunit; + +namespace SharpCoreDB.Functional.Linq2DB.Tests; + +/// +/// LINQ filters that linq2db's SQLite dialect translates into scalar functions over a column: every +/// comparison (strftime('%Y-%m-%d %H:%M:%f', [x].[at]) = strftime(…, @p)), date parts +/// (CAST(strftime('%Y', …) AS int)), .Date, ToUpper() and Length. +/// +/// Up to 2.1.4 SharpCoreDB evaluated none of these over a table: each filter matched no row, so +/// Where(x => x.At == d), x.At > d and a date-filtered DeleteWhereAsync silently did +/// nothing. +/// +/// +public sealed class DateAndStringFilterTests : IDisposable +{ + private static readonly DateTime Jan1Noon = new(2026, 1, 1, 12, 0, 0); + private static readonly DateTime Mid2025 = new(2025, 6, 15, 8, 30, 0, 250); + private static readonly DateTime Mar10Late = new(2026, 3, 10, 23, 59, 59, 999); + + private readonly string _path = Path.Combine(Path.GetTempPath(), $"linq2db_filters_{Guid.NewGuid():N}.scdb"); + private readonly SharpCoreDBDataConnection _connection; + + public DateAndStringFilterTests() + { + _connection = new SharpCoreDBDataConnection($"Data Source={_path}"); + using (var command = _connection.GetUnderlyingConnection().CreateCommand()) + { + command.CommandText = "CREATE TABLE events (id INTEGER PRIMARY KEY, name TEXT, at TEXT)"; + command.ExecuteNonQuery(); + } + + _connection.Insert(new Event { Id = 1, Name = "Alpha", At = Jan1Noon }); + _connection.Insert(new Event { Id = 2, Name = "beta", At = Mid2025 }); + _connection.Insert(new Event { Id = 3, Name = "Gamma", At = Mar10Late }); + } + + [Fact] + public void Where_DateTimeEquals_FindsTheRow() + { + Ids(_connection.GetTable().Where(e => e.At == Jan1Noon)).Should().Equal(1); + } + + [Fact] + public void Where_DateTimeGreaterThan_FindsLaterRows() + { + var cutoff = new DateTime(2026, 1, 1); + + Ids(_connection.GetTable().Where(e => e.At > cutoff)).Should().Equal(1, 3); + } + + [Fact] + public void Where_DateTimeRange_FindsRowsInside() + { + var from = new DateTime(2025, 1, 1); + var to = new DateTime(2026, 2, 1); + + Ids(_connection.GetTable().Where(e => e.At >= from && e.At < to)).Should().Equal(1, 2); + } + + [Fact] + public void Where_DateTimeYear_FindsRowsInThatYear() + { + Ids(_connection.GetTable().Where(e => e.At.Year == 2026)).Should().Equal(1, 3); + } + + [Fact] + public void Where_DateTimeDate_FindsRowsOnThatDay() + { + var day = new DateTime(2026, 3, 10); + + Ids(_connection.GetTable().Where(e => e.At.Date == day)).Should().Equal(3); + } + + [Fact] + public void Where_ToUpper_ComparesCaseInsensitively() + { + Ids(_connection.GetTable().Where(e => e.Name.ToUpper() == "BETA")).Should().Equal(2); + } + + [Fact] + public void Where_StringLength_FindsMatchingRows() + { + Ids(_connection.GetTable().Where(e => e.Name.Length == 5)).Should().Equal(1, 3); + } + + [Fact] + public void Select_DateTimeYear_ProjectsTheYear() + { + _connection.GetTable().Where(e => e.Id == 2).Select(e => e.At.Year).Single().Should().Be(2025); + } + + [Fact] + public async Task DeleteWhereAsync_ByDateTime_DeletesOnlyMatchingRows() + { + var db = new FunctionalLinq2DbContext(_connection); + var cutoff = new DateTime(2026, 1, 1); + + var deleted = await db.DeleteWhereAsync(e => e.At < cutoff); + + deleted.Match(n => n, _ => -1).Should().Be(1); + Ids(_connection.GetTable()).Should().Equal(1, 3); + } + + public void Dispose() + { + _connection.Dispose(); + try + { + if (File.Exists(_path)) + { + File.Delete(_path); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp file. + } + } + + private static int[] Ids(IQueryable query) => [.. query.Select(e => e.Id).AsEnumerable().Order()]; + + [Table("events")] + public sealed class Event + { + [PrimaryKey, Column("id")] public int Id { get; set; } + [Column("name")] public string Name { get; set; } = ""; + [Column("at")] public DateTime At { get; set; } + } +} diff --git a/tests/SharpCoreDB.Functional.Linq2DB.Tests/DependencyInjectionTests.cs b/tests/SharpCoreDB.Functional.Linq2DB.Tests/DependencyInjectionTests.cs new file mode 100644 index 00000000..13366221 --- /dev/null +++ b/tests/SharpCoreDB.Functional.Linq2DB.Tests/DependencyInjectionTests.cs @@ -0,0 +1,125 @@ +using System.Data.Common; +using FluentAssertions; +using LinqToDB; +using LinqToDB.Mapping; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace SharpCoreDB.Functional.Linq2DB.Tests; + +/// +/// Connection ownership of the DI registrations and of reused . +/// +/// Up to 2.1.4 the options captured one live SharpCoreDBConnection, so every DI scope — every web request +/// — shared that single ADO.NET connection (and its transaction state) for the lifetime of the application, and +/// nothing ever disposed it. The Action<DataOptions> overload could not work at all: +/// is immutable, so the callback's result was discarded and resolving the connection threw +/// "Configuration string is not provided". +/// +/// +public sealed class DependencyInjectionTests : IDisposable +{ + private readonly string _path = Path.Combine(Path.GetTempPath(), $"linq2db_di_{Guid.NewGuid():N}.scdb"); + + private string ConnectionString => $"Data Source={_path}"; + + [Fact] + public void AddSharpCoreDBLinq2Db_WithConfigureCallback_ResolvesAWorkingConnection() + { + using var provider = new ServiceCollection() + .AddSharpCoreDBLinq2Db(options => options.UseSharpCoreDB(ConnectionString)) + .BuildServiceProvider(); + using var scope = provider.CreateScope(); + var connection = scope.ServiceProvider.GetRequiredService(); + + CreateTable(connection); + connection.Insert(new DiRow { Id = 1, Name = "configured" }); + + connection.GetTable().Single().Name.Should().Be("configured"); + } + + [Fact] + public void AddSharpCoreDBLinq2Db_WithConnectionString_GivesEachScopeItsOwnConnection() + { + using var provider = new ServiceCollection().AddSharpCoreDBLinq2Db(ConnectionString).BuildServiceProvider(); + using var first = provider.CreateScope(); + using var second = provider.CreateScope(); + + var a = first.ServiceProvider.GetRequiredService().Connection; + var b = second.ServiceProvider.GetRequiredService().Connection; + + a.Should().NotBeSameAs(b); + } + + [Fact] + public void AddSharpCoreDBLinq2Db_WhenScopeEnds_DisposesItsConnection() + { + using var provider = new ServiceCollection().AddSharpCoreDBLinq2Db(ConnectionString).BuildServiceProvider(); + DbConnection connection; + var disposed = false; + + using (var scope = provider.CreateScope()) + { + connection = scope.ServiceProvider.GetRequiredService().Connection; + connection.Disposed += (_, _) => disposed = true; + } + + disposed.Should().BeTrue(); + } + + [Fact] + public void AddSharpCoreDBLinq2Db_WithOverlappingScopes_SeeEachOthersCommittedRows() + { + // Separate connections to one database must still share its data (SharpCoreDBInstancePool). + using var provider = new ServiceCollection().AddSharpCoreDBLinq2Db(ConnectionString).BuildServiceProvider(); + using var writerScope = provider.CreateScope(); + using var readerScope = provider.CreateScope(); + var writer = writerScope.ServiceProvider.GetRequiredService(); + var reader = readerScope.ServiceProvider.GetRequiredService(); + + CreateTable(writer); + writer.Insert(new DiRow { Id = 7, Name = "shared" }); + + reader.GetTable().Single(r => r.Id == 7).Name.Should().Be("shared"); + } + + [Fact] + public void Constructor_WithReusedOptions_OpensItsOwnConnection() + { + var options = SharpCoreDBDataConnection.CreateOptions(ConnectionString); + + using var first = new SharpCoreDBDataConnection(options); + using var second = new SharpCoreDBDataConnection(options); + + first.Connection.Should().NotBeSameAs(second.Connection); + } + + public void Dispose() + { + try + { + if (File.Exists(_path)) + { + File.Delete(_path); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp file. + } + } + + private static void CreateTable(SharpCoreDBDataConnection connection) + { + using var command = connection.GetUnderlyingConnection().CreateCommand(); + command.CommandText = "CREATE TABLE IF NOT EXISTS di_rows (id INTEGER PRIMARY KEY, name TEXT)"; + command.ExecuteNonQuery(); + } + + [Table("di_rows")] + public sealed class DiRow + { + [PrimaryKey, Column("id")] public long Id { get; set; } + [Column("name")] public string Name { get; set; } = ""; + } +} diff --git a/tests/SharpCoreDB.Functional.Linq2DB.Tests/FunctionalContextContractTests.cs b/tests/SharpCoreDB.Functional.Linq2DB.Tests/FunctionalContextContractTests.cs new file mode 100644 index 00000000..495faf08 --- /dev/null +++ b/tests/SharpCoreDB.Functional.Linq2DB.Tests/FunctionalContextContractTests.cs @@ -0,0 +1,191 @@ +using FluentAssertions; +using LinqToDB; +using LinqToDB.Data; +using LinqToDB.Mapping; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace SharpCoreDB.Functional.Linq2DB.Tests; + +/// +/// The error, cancellation and transaction contract of , its batch insert, and +/// its DI registration. +/// +/// The rule: methods returning Option/Seq throw on a failure, methods returning Fin capture it +/// as Fail, and cancellation always surfaces as . Up to 2.1.4 a token +/// cancelled before the call threw while one cancelled during the call came back as Fail, and +/// TransactionAsync rolled back with the caller's (cancelled) token — the rollback then threw and the +/// action's own error was replaced by "The operation was canceled". +/// +/// +public sealed class FunctionalContextContractTests : IDisposable +{ + private readonly string _path = Path.Combine(Path.GetTempPath(), $"linq2db_contract_{Guid.NewGuid():N}.scdb"); + private readonly SharpCoreDBDataConnection _connection; + private readonly FunctionalLinq2DbContext _sut; + + public FunctionalContextContractTests() + { + _connection = new SharpCoreDBDataConnection(ConnectionString); + _sut = new FunctionalLinq2DbContext(_connection); + CreateTable(_connection); + } + + private string ConnectionString => $"Data Source={_path}"; + + // ── #7 TransactionAsync ───────────────────────────────────────────────────────────────────── + + [Fact] + public async Task TransactionAsync_WhenActionThrows_RollsBackItsWrites() + { + var result = await _sut.TransactionAsync(async () => + { + await _connection.InsertAsync(new ContractRow { Id = 1, Name = "inside" }); + throw new InvalidOperationException("boom"); + }); + + result.IsFail.Should().BeTrue(); + _connection.GetTable().Count().Should().Be(0); + } + + [Fact] + public async Task TransactionAsync_WhenActionThrowsAfterTokenIsCancelled_ReportsTheActionsError() + { + using var cts = new CancellationTokenSource(); + + var result = await _sut.TransactionAsync(() => + { + cts.Cancel(); + throw new InvalidOperationException("the action's own error"); + }, cts.Token); + + result.Match(_ => "", e => e.Message).Should().Be("the action's own error"); + } + + [Fact] + public async Task TransactionAsync_WithContextOverload_PassesContextAndTokenAndCommits() + { + using var cts = new CancellationTokenSource(); + CancellationToken seen = default; + + var result = await _sut.TransactionAsync(async (db, ct) => + { + seen = ct; + return await db.InsertAsync(new ContractRow { Id = 2, Name = "committed" }, ct); + }, cts.Token); + + result.IsSucc.Should().BeTrue(); + seen.Should().Be(cts.Token); + _connection.GetTable().Single().Name.Should().Be("committed"); + } + + // ── #8 cancellation is never a Fail ───────────────────────────────────────────────────────── + + [Fact] + public async Task TransactionAsync_WhenCancelledDuringTheAction_ThrowsOperationCanceled() + { + using var cts = new CancellationTokenSource(); + + await Assert.ThrowsAnyAsync(() => _sut.TransactionAsync(() => + { + cts.Cancel(); + cts.Token.ThrowIfCancellationRequested(); + return Task.FromResult(0); + }, cts.Token)); + } + + // ── #9 InsertBatchAsync ───────────────────────────────────────────────────────────────────── + + [Fact] + public async Task InsertBatchAsync_WithCallerOptions_UsesMultiRowSqlAndInsertsEveryRow() + { + // ProviderSpecific bulk copy needs a native Microsoft.Data.Sqlite connection, which the adapter never has. + var options = new BulkCopyOptions { BulkCopyType = BulkCopyType.ProviderSpecific, MaxBatchSize = 2 }; + var rows = Enumerable.Range(10, 5).Select(i => new ContractRow { Id = i, Name = $"r{i}" }).ToList(); + + var result = await _sut.InsertBatchAsync(rows, options); + + result.Match(n => n, _ => -1).Should().Be(5); + _connection.GetTable().Count().Should().Be(5); + } + + [Fact] + public void ToRowCount_AboveInt32_IsAFailNotAWrappedNumber() + { + var result = FunctionalLinq2DbContext.ToRowCount(3_000_000_000L); + + result.IsFail.Should().BeTrue(); + } + + [Fact] + public void ToRowCount_WithinInt32_IsTheCount() + { + FunctionalLinq2DbContext.ToRowCount(42L).Match(n => n, _ => -1).Should().Be(42); + } + + // ── #10 DI registers the functional context ───────────────────────────────────────────────── + + [Fact] + public void AddSharpCoreDBLinq2Db_WithConnectionString_RegistersTheContextOnTheScopesConnection() + { + using var provider = new ServiceCollection().AddSharpCoreDBLinq2Db(ConnectionString).BuildServiceProvider(); + using var scope = provider.CreateScope(); + + var context = scope.ServiceProvider.GetRequiredService(); + + context.Connection.Should().BeSameAs(scope.ServiceProvider.GetRequiredService()); + } + + [Fact] + public void AddSharpCoreDBLinq2Db_WithConfigureCallback_RegistersTheContext() + { + using var provider = new ServiceCollection() + .AddSharpCoreDBLinq2Db(o => o.UseSharpCoreDB(ConnectionString)) + .BuildServiceProvider(); + using var scope = provider.CreateScope(); + + scope.ServiceProvider.GetRequiredService().Should().NotBeNull(); + } + + [Fact] + public void AddSharpCoreDBLinq2Db_KeepsACallersOwnContextRegistration() + { + var mine = new FunctionalLinq2DbContext(_connection); + using var provider = new ServiceCollection() + .AddSingleton(mine) + .AddSharpCoreDBLinq2Db(ConnectionString) + .BuildServiceProvider(); + + provider.GetRequiredService().Should().BeSameAs(mine); + } + + public void Dispose() + { + _connection.Dispose(); + try + { + if (File.Exists(_path)) + { + File.Delete(_path); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp file. + } + } + + private static void CreateTable(SharpCoreDBDataConnection connection) + { + using var command = connection.GetUnderlyingConnection().CreateCommand(); + command.CommandText = "CREATE TABLE IF NOT EXISTS contract_rows (id INTEGER PRIMARY KEY, name TEXT)"; + command.ExecuteNonQuery(); + } + + [Table("contract_rows")] + public sealed class ContractRow + { + [PrimaryKey, Column("id")] public int Id { get; set; } + [Column("name")] public string Name { get; set; } = ""; + } +} diff --git a/tests/SharpCoreDB.Functional.Linq2DB.Tests/FunctionalLinq2DbContextGuardTests.cs b/tests/SharpCoreDB.Functional.Linq2DB.Tests/FunctionalLinq2DbContextGuardTests.cs index 8d53e5fd..a7666dca 100644 --- a/tests/SharpCoreDB.Functional.Linq2DB.Tests/FunctionalLinq2DbContextGuardTests.cs +++ b/tests/SharpCoreDB.Functional.Linq2DB.Tests/FunctionalLinq2DbContextGuardTests.cs @@ -173,7 +173,14 @@ await Assert.ThrowsAnyAsync(() => [Fact] public async Task TransactionAsync_WithNullAction_ThrowsArgumentNullException() { - await Assert.ThrowsAsync(() => _sut.TransactionAsync(null!)); + await Assert.ThrowsAsync(() => _sut.TransactionAsync((Func>)null!)); + } + + [Fact] + public async Task TransactionAsync_WithNullContextAction_ThrowsArgumentNullException() + { + await Assert.ThrowsAsync(() => + _sut.TransactionAsync((Func>)null!)); } [Fact] diff --git a/tests/SharpCoreDB.Functional.Linq2DB.Tests/GetByIdAsyncTests.cs b/tests/SharpCoreDB.Functional.Linq2DB.Tests/GetByIdAsyncTests.cs new file mode 100644 index 00000000..164dfd8a --- /dev/null +++ b/tests/SharpCoreDB.Functional.Linq2DB.Tests/GetByIdAsyncTests.cs @@ -0,0 +1,152 @@ +using FluentAssertions; +using LinqToDB; +using LinqToDB.Mapping; +using Xunit; + +namespace SharpCoreDB.Functional.Linq2DB.Tests; + +/// +/// looks the row up by its mapped primary key. +/// +/// Up to 2.1.4 it ignored keyValues and returned the first row of the table — GetByIdAsync([2]) and +/// GetByIdAsync([999]) both answered with row 1 — and swallowed every exception into None, so a +/// failing query read as "not found". +/// +/// +public sealed class GetByIdAsyncTests : IDisposable +{ + private readonly string _path = Path.Combine(Path.GetTempPath(), $"linq2db_getbyid_{Guid.NewGuid():N}.scdb"); + private readonly SharpCoreDBDataConnection _connection; + private readonly FunctionalLinq2DbContext _sut; + + public GetByIdAsyncTests() + { + _connection = new SharpCoreDBDataConnection($"Data Source={_path}"); + _sut = new FunctionalLinq2DbContext(_connection); + + Execute("CREATE TABLE gbi_rows (id INTEGER PRIMARY KEY, name TEXT)"); + Execute("CREATE TABLE gbi_pairs (tenant INTEGER, code TEXT, name TEXT)"); + + _connection.Insert(new KeyedRow { Id = 1, Name = "one" }); + _connection.Insert(new KeyedRow { Id = 2, Name = "two" }); + _connection.Insert(new KeyedRow { Id = 3, Name = "three" }); + + _connection.Insert(new PairRow { Tenant = 1, Code = "a", Name = "1a" }); + _connection.Insert(new PairRow { Tenant = 1, Code = "b", Name = "1b" }); + _connection.Insert(new PairRow { Tenant = 2, Code = "a", Name = "2a" }); + } + + [Fact] + public async Task GetByIdAsync_WithExistingKey_ReturnsThatRow() + { + var result = await _sut.GetByIdAsync([2L]); + + result.Match(r => r.Name, () => "").Should().Be("two"); + } + + [Fact] + public async Task GetByIdAsync_WithMissingKey_ReturnsNone() + { + var result = await _sut.GetByIdAsync([999L]); + + result.IsNone.Should().BeTrue(); + } + + [Fact] + public async Task GetByIdAsync_WithKeyOfConvertibleType_ConvertsToPropertyType() + { + // An int literal for a long key is the common call shape: GetByIdAsync([3]). + var result = await _sut.GetByIdAsync([3]); + + result.Match(r => r.Name, () => "").Should().Be("three"); + } + + [Fact] + public async Task GetByIdAsync_WithCompositeKey_MatchesEveryKeyColumnInOrder() + { + var result = await _sut.GetByIdAsync([2, "a"]); + + result.Match(r => r.Name, () => "").Should().Be("2a"); + } + + [Fact] + public async Task GetByIdAsync_WithWrongKeyCount_ThrowsArgumentException() + { + await Assert.ThrowsAsync(() => _sut.GetByIdAsync([2])); + } + + [Fact] + public async Task GetByIdAsync_WithEmptyKeyValues_ThrowsArgumentException() + { + await Assert.ThrowsAsync(() => _sut.GetByIdAsync([])); + } + + [Fact] + public async Task GetByIdAsync_WithUnconvertibleKey_ThrowsArgumentException() + { + await Assert.ThrowsAsync(() => _sut.GetByIdAsync(["not a number"])); + } + + [Fact] + public async Task GetByIdAsync_OnEntityWithoutPrimaryKey_ThrowsInvalidOperationException() + { + await Assert.ThrowsAsync(() => _sut.GetByIdAsync([1])); + } + + [Fact] + public async Task GetByIdAsync_WhenQueryFails_PropagatesTheException() + { + // The table does not exist: that is an error, not "not found". + await Assert.ThrowsAnyAsync(() => _sut.GetByIdAsync([1])); + } + + public void Dispose() + { + _connection.Dispose(); + try + { + if (File.Exists(_path)) + { + File.Delete(_path); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp file. + } + } + + private void Execute(string sql) + { + using var command = _connection.GetUnderlyingConnection().CreateCommand(); + command.CommandText = sql; + command.ExecuteNonQuery(); + } + + [Table("gbi_rows")] + public sealed class KeyedRow + { + [PrimaryKey, Column("id")] public long Id { get; set; } + [Column("name")] public string Name { get; set; } = ""; + } + + [Table("gbi_pairs")] + public sealed class PairRow + { + [PrimaryKey(0), Column("tenant")] public int Tenant { get; set; } + [PrimaryKey(1), Column("code")] public string Code { get; set; } = ""; + [Column("name")] public string Name { get; set; } = ""; + } + + [Table("gbi_rows")] + public sealed class NoKeyRow + { + [Column("id")] public long Id { get; set; } + } + + [Table("gbi_table_that_does_not_exist")] + public sealed class MissingTableRow + { + [PrimaryKey, Column("id")] public long Id { get; set; } + } +} diff --git a/tests/SharpCoreDB.Functional.Linq2DB.Tests/MappingSchemaTests.cs b/tests/SharpCoreDB.Functional.Linq2DB.Tests/MappingSchemaTests.cs new file mode 100644 index 00000000..a3ea508e --- /dev/null +++ b/tests/SharpCoreDB.Functional.Linq2DB.Tests/MappingSchemaTests.cs @@ -0,0 +1,209 @@ +using System.Globalization; +using FluentAssertions; +using LinqToDB; +using LinqToDB.Common; +using LinqToDB.Data; +using LinqToDB.Mapping; +using Xunit; + +namespace SharpCoreDB.Functional.Linq2DB.Tests; + +/// +/// How combines with a caller's own mapping schema, and the text formats +/// its converters read and write. +/// +/// Up to 2.1.4 both new SharpCoreDBDataConnection(DataOptions) and UseSharpCoreDB applied the +/// schema with UseMappingSchema, which linq2db documents as "replaces all previous registrations": a fluent +/// mapping of Entity to custom_table silently queried [Entity] instead. The Guid converters +/// disagreed with each other ("D" for parameters, "N" for Guid → string), and stored dates +/// were parsed with the current culture — under a Buddhist-calendar culture (th-TH) the year 2026 read back as 1483. +/// +/// +public sealed class MappingSchemaTests : IDisposable +{ + private readonly string _path = Path.Combine(Path.GetTempPath(), $"linq2db_schema_{Guid.NewGuid():N}.scdb"); + + private string ConnectionString => $"Data Source={_path}"; + + [Fact] + public void Constructor_WithUserMappingSchema_KeepsTheUserMappings() + { + var options = SharpCoreDBDataConnection.CreateOptions(ConnectionString) + .UseAdditionalMappingSchema(UserSchema()); + + using var connection = new SharpCoreDBDataConnection(options); + Execute(connection, "CREATE TABLE custom_table (custom_id INTEGER PRIMARY KEY, custom_code TEXT)"); + connection.Insert(new Unattributed { Id = 5, Code = Ulid.NewUlid() }); + + connection.GetTable().Single().Id.Should().Be(5); + } + + [Fact] + public void UseSharpCoreDB_OnOptionsWithAMappingSchema_KeepsTheUserMappings() + { + var options = new DataOptions() + .UseMappingSchema(UserSchema()) + .UseSharpCoreDB(ConnectionString); + + using var connection = new SharpCoreDBDataConnection(options); + Execute(connection, "CREATE TABLE custom_table (custom_id INTEGER PRIMARY KEY, custom_code TEXT)"); + connection.Insert(new Unattributed { Id = 6, Code = Ulid.NewUlid() }); + + connection.GetTable().Single().Id.Should().Be(6); + } + + [Fact] + public void Constructor_WithUserMappingSchema_StillAppliesTheSharpCoreDbConverters() + { + var options = SharpCoreDBDataConnection.CreateOptions(ConnectionString) + .UseAdditionalMappingSchema(UserSchema()); + var code = Ulid.NewUlid(); + + using var connection = new SharpCoreDBDataConnection(options); + Execute(connection, "CREATE TABLE custom_table (custom_id INTEGER PRIMARY KEY, custom_code TEXT)"); + connection.Insert(new Unattributed { Id = 8, Code = code }); + + connection.GetTable().Single(e => e.Code == code).Id.Should().Be(8); + } + + [Fact] + public void Constructor_WithUserConverter_LetsTheUserConverterWin() + { + // The caller's schema has priority over the SharpCoreDB defaults: here it stores Guids in "N" form. + var user = UserSchema(); + user.SetConverter(g => + new DataParameter(null, g.ToString("N"), LinqToDB.DataType.NVarChar)); + var options = SharpCoreDBDataConnection.CreateOptions(ConnectionString).UseAdditionalMappingSchema(user); + var guid = Guid.NewGuid(); + + using var connection = new SharpCoreDBDataConnection(options); + Execute(connection, "CREATE TABLE guid_rows (id INTEGER PRIMARY KEY, value TEXT)"); + connection.Insert(new GuidRow { Id = 1, Value = guid }); + + Scalar(connection, "SELECT value FROM guid_rows WHERE id = 1").Should().Be(guid.ToString("N")); + } + + [Fact] + public void GuidToString_UsesTheFormatStoredByParameters() + { + var guid = Guid.NewGuid(); + using var connection = new SharpCoreDBDataConnection(ConnectionString); + Execute(connection, "CREATE TABLE guid_rows (id INTEGER PRIMARY KEY, value TEXT)"); + connection.Insert(new GuidRow { Id = 1, Value = guid }); + + var stored = Scalar(connection, "SELECT value FROM guid_rows WHERE id = 1"); + + Converter.ChangeTypeTo(guid, SharpCoreDBMappingSchema.Instance).Should().Be(stored); + } + + [Fact] + public void DateTime_UnderANonGregorianCulture_RoundTrips() + { + var value = new DateTime(2026, 1, 1, 12, 30, 15, 250); + using var connection = new SharpCoreDBDataConnection(ConnectionString); + Execute(connection, "CREATE TABLE date_rows (id INTEGER PRIMARY KEY, at TEXT, at_offset TEXT)"); + + var read = WithCulture("th-TH", () => + { + connection.Insert(new DateRow + { + Id = 1, At = value, AtOffset = new DateTimeOffset(value, TimeSpan.FromHours(10)) + }); + return connection.GetTable().Single(r => r.Id == 1); + }); + + read.At.Should().Be(value); + read.AtOffset.Should().Be(new DateTimeOffset(value, TimeSpan.FromHours(10))); + } + + [Fact] + public void DateTime_StoredWithUtcMarker_ReadsBackAsUtc() + { + // A round-trip ("O") string written by another client — plain ADO.NET, a migration — carries its kind. + // Up to 2.1.4 the reader parsed it without RoundtripKind, converting it to local time (Kind = Local). + using var connection = new SharpCoreDBDataConnection(ConnectionString); + Execute(connection, "CREATE TABLE date_rows (id INTEGER PRIMARY KEY, at TEXT, at_offset TEXT)"); + Execute(connection, + "INSERT INTO date_rows (id, at, at_offset) VALUES (1, '2026-01-01T12:00:00.0000000Z', '2026-01-01 12:00:00.000+00:00')"); + + var read = connection.GetTable().Single(r => r.Id == 1).At; + + read.Kind.Should().Be(DateTimeKind.Utc); + read.Should().Be(new DateTime(2026, 1, 1, 12, 0, 0, DateTimeKind.Utc)); + } + + public void Dispose() + { + try + { + if (File.Exists(_path)) + { + File.Delete(_path); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp file. + } + } + + private static MappingSchema UserSchema() + { + var schema = new MappingSchema(); + new FluentMappingBuilder(schema) + .Entity().HasTableName("custom_table") + .Property(e => e.Id).HasColumnName("custom_id").IsPrimaryKey() + .Property(e => e.Code).HasColumnName("custom_code") + .Build(); + return schema; + } + + private static T WithCulture(string name, Func action) + { + var previous = CultureInfo.CurrentCulture; + CultureInfo.CurrentCulture = CultureInfo.GetCultureInfo(name); + try + { + return action(); + } + finally + { + CultureInfo.CurrentCulture = previous; + } + } + + private static void Execute(SharpCoreDBDataConnection connection, string sql) + { + using var command = connection.GetUnderlyingConnection().CreateCommand(); + command.CommandText = sql; + command.ExecuteNonQuery(); + } + + private static string? Scalar(SharpCoreDBDataConnection connection, string sql) + { + using var command = connection.GetUnderlyingConnection().CreateCommand(); + command.CommandText = sql; + return Convert.ToString(command.ExecuteScalar(), CultureInfo.InvariantCulture); + } + + public sealed class Unattributed + { + public long Id { get; set; } + public Ulid Code { get; set; } = null!; + } + + [Table("guid_rows")] + public sealed class GuidRow + { + [PrimaryKey, Column("id")] public long Id { get; set; } + [Column("value")] public Guid Value { get; set; } + } + + [Table("date_rows")] + public sealed class DateRow + { + [PrimaryKey, Column("id")] public long Id { get; set; } + [Column("at")] public DateTime At { get; set; } + [Column("at_offset")] public DateTimeOffset AtOffset { get; set; } + } +} diff --git a/tests/SharpCoreDB.Functional.Linq2DB.Tests/OrderingTests.cs b/tests/SharpCoreDB.Functional.Linq2DB.Tests/OrderingTests.cs new file mode 100644 index 00000000..e20ba2a8 --- /dev/null +++ b/tests/SharpCoreDB.Functional.Linq2DB.Tests/OrderingTests.cs @@ -0,0 +1,92 @@ +using FluentAssertions; +using LinqToDB; +using LinqToDB.Mapping; +using Xunit; + +namespace SharpCoreDB.Functional.Linq2DB.Tests; + +/// +/// LINQ ordering through the adapter: linq2db emits ORDER BY [x].[a], [x].[b] DESC for +/// OrderBy(…).ThenByDescending(…) and wraps computed keys in SQL functions. +/// +/// Up to 2.1.4 the engine read only the first ORDER BY token — here [x].[grp],, which matched no column — so +/// every multi-key, qualified or computed ordering came back in storage order. +/// +/// +public sealed class OrderingTests : IDisposable +{ + private readonly string _path = Path.Combine(Path.GetTempPath(), $"linq2db_ordering_{Guid.NewGuid():N}.scdb"); + private readonly SharpCoreDBDataConnection _connection; + + public OrderingTests() + { + _connection = new SharpCoreDBDataConnection($"Data Source={_path}"); + using (var command = _connection.GetUnderlyingConnection().CreateCommand()) + { + command.CommandText = "CREATE TABLE ranked (id INTEGER PRIMARY KEY, grp INTEGER, name TEXT, at TEXT)"; + command.ExecuteNonQuery(); + } + + _connection.Insert(new Ranked { Id = 1, Grp = 2, Name = "bb", At = new DateTime(2026, 3, 1) }); + _connection.Insert(new Ranked { Id = 2, Grp = 1, Name = "a", At = new DateTime(2024, 1, 1) }); + _connection.Insert(new Ranked { Id = 3, Grp = 2, Name = "ccc", At = new DateTime(2025, 6, 1) }); + _connection.Insert(new Ranked { Id = 4, Grp = 1, Name = "dddd", At = new DateTime(2023, 1, 1) }); + } + + [Fact] + public void OrderBy_ThenBy_SortsByBothKeys() + { + Ids(_connection.GetTable().OrderBy(r => r.Grp).ThenBy(r => r.Name)).Should().Equal(2, 4, 1, 3); + } + + [Fact] + public void OrderByDescending_ThenBy_SortsEachKeyInItsOwnDirection() + { + Ids(_connection.GetTable().OrderByDescending(r => r.Grp).ThenBy(r => r.Name)).Should().Equal(1, 3, 2, 4); + } + + [Fact] + public void OrderByDescending_DateTime_SortsNewestFirst() + { + Ids(_connection.GetTable().OrderByDescending(r => r.At)).Should().Equal(1, 3, 2, 4); + } + + [Fact] + public void OrderBy_StringLength_SortsByTheComputedKey() + { + Ids(_connection.GetTable().OrderByDescending(r => r.Name.Length)).Should().Equal(4, 3, 1, 2); + } + + [Fact] + public void OrderBy_ThenTake_TakesFromTheSortedRows() + { + Ids(_connection.GetTable().OrderBy(r => r.Grp).ThenByDescending(r => r.Id).Take(2)).Should().Equal(4, 2); + } + + public void Dispose() + { + _connection.Dispose(); + try + { + if (File.Exists(_path)) + { + File.Delete(_path); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp file. + } + } + + private static int[] Ids(IQueryable query) => [.. query.Select(r => r.Id)]; + + [Table("ranked")] + public sealed class Ranked + { + [PrimaryKey, Column("id")] public int Id { get; set; } + [Column("grp")] public int Grp { get; set; } + [Column("name")] public string Name { get; set; } = ""; + [Column("at")] public DateTime At { get; set; } + } +} diff --git a/tests/SharpCoreDB.Functional.Linq2DB.Tests/SharpCoreDB.Functional.Linq2DB.Tests.csproj b/tests/SharpCoreDB.Functional.Linq2DB.Tests/SharpCoreDB.Functional.Linq2DB.Tests.csproj index 5c9e8dfd..7b2df763 100644 --- a/tests/SharpCoreDB.Functional.Linq2DB.Tests/SharpCoreDB.Functional.Linq2DB.Tests.csproj +++ b/tests/SharpCoreDB.Functional.Linq2DB.Tests/SharpCoreDB.Functional.Linq2DB.Tests.csproj @@ -13,8 +13,9 @@ - - + + diff --git a/tests/SharpCoreDB.Functional.Linq2DB.Tests/TypeMappingTests.cs b/tests/SharpCoreDB.Functional.Linq2DB.Tests/TypeMappingTests.cs index ef6fb0bd..52ae9c8a 100644 --- a/tests/SharpCoreDB.Functional.Linq2DB.Tests/TypeMappingTests.cs +++ b/tests/SharpCoreDB.Functional.Linq2DB.Tests/TypeMappingTests.cs @@ -62,44 +62,21 @@ is_active INTEGER public void Dispose() { + // Disposing the connection disposes the SharpCoreDB connection it opened, which releases the file; there are + // no SQLite -wal/-shm sidecars to retry for since the adapter stopped writing SQLite files (2.1.3). Connection?.Dispose(); - // Robust cleanup for Windows SQLite locking and sidecar files - var cleanupPaths = new[] { DbPath, DbPath + "-wal", DbPath + "-shm" }; - - for (int i = 0; i < 20; i++) + try { - try + if (File.Exists(DbPath)) { - foreach (var path in cleanupPaths) - { - if (File.Exists(path)) - { - File.Delete(path); - } - } - - return; - } - catch (IOException) - { - if (i == 19) - { - return; - } - - System.Threading.Thread.Sleep(200); - } - catch (UnauthorizedAccessException) - { - if (i == 19) - { - return; - } - - System.Threading.Thread.Sleep(200); + File.Delete(DbPath); } } + catch (IOException) + { + // Best-effort cleanup of a temp file. + } } } diff --git a/tests/SharpCoreDB.Tests/ColumnNotNullConstraintTests.cs b/tests/SharpCoreDB.Tests/ColumnNotNullConstraintTests.cs new file mode 100644 index 00000000..56e94b03 --- /dev/null +++ b/tests/SharpCoreDB.Tests/ColumnNotNullConstraintTests.cs @@ -0,0 +1,115 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.IO; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// A declared NOT NULL constraint is enforced on single-file storage as it is on directory storage — on INSERT, on +/// a batch INSERT, and on UPDATE — in the creating session and after a reopen. +/// +/// Up to 2.1.4 the single-file table never checked NOT NULL at all: its INSERT paths only applied defaults, so a NULL +/// went into a NOT NULL column in any session. Its constructor also reset the NOT NULL and AUTOINCREMENT flags it had +/// just copied (InitializeColumnMetadata() ran after the copy), so the creating session's table did not even +/// know about the constraint. The flags written to the file were correct. +/// +/// +public sealed class ColumnNotNullConstraintTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _path = Path.Combine(Path.GetTempPath(), $"SCDB_SingleFileConstraints_{Guid.NewGuid():N}.scdb"); + + public ColumnNotNullConstraintTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + } + + public void Dispose() + { + try + { + if (File.Exists(_path)) + { + File.Delete(_path); + } + + var directory = Path.ChangeExtension(_path, null); + if (Directory.Exists(directory)) + { + Directory.Delete(directory, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp file. + } + } + + [Fact] + public void Insert_NullIntoNotNullColumn_InTheCreatingSession_IsRejected() + { + var db = Open(); + db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT NOT NULL)"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, NULL)")); + Assert.Empty(db.ExecuteQuery("SELECT id FROM t")); + } + + [Fact] + public void Insert_NullIntoNotNullColumn_AfterReopen_IsRejected() + { + var db = Open(); + db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT NOT NULL)"); + (db as IDisposable)?.Dispose(); + + var reopened = Open(); + Assert.ThrowsAny(() => reopened.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, NULL)")); + (reopened as IDisposable)?.Dispose(); + } + + [Fact] + public void Insert_ValueIntoNotNullColumn_IsAccepted() + { + var db = Open(); + db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT NOT NULL)"); + + db.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, 'kept')"); + + Assert.Single(db.ExecuteQuery("SELECT id FROM t")); + } + + [Theory] + [InlineData("directory")] + [InlineData("single-file")] + public void Update_SetNotNullColumnToNull_IsRejectedAndChangesNothing(string mode) + { + var db = mode == "single-file" ? Open() : _factory.Create(Path.ChangeExtension(_path, null), "pw"); + db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT NOT NULL)"); + db.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, 'kept')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET name = NULL WHERE id = 1")); + Assert.Equal("kept", Assert.Single(db.ExecuteQuery("SELECT name FROM t"))["name"]); + } + + [Fact] + public void Update_SetNotNullColumnToNull_MatchingNoRow_IsAccepted() + { + var db = Open(); + db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT NOT NULL)"); + + db.ExecuteSQL("UPDATE t SET name = NULL WHERE id = 99"); + + Assert.Empty(db.ExecuteQuery("SELECT id FROM t")); + } + + private IDatabase Open() => _factory.CreateWithOptions(_path, "pw", DatabaseOptions.CreateSingleFileDefault()); +} diff --git a/tests/SharpCoreDB.Tests/ExecuteQueryFastWhereTests.cs b/tests/SharpCoreDB.Tests/ExecuteQueryFastWhereTests.cs new file mode 100644 index 00000000..5ff45a53 --- /dev/null +++ b/tests/SharpCoreDB.Tests/ExecuteQueryFastWhereTests.cs @@ -0,0 +1,86 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Services; +using Xunit; + +/// +/// filters with the same WHERE semantics as ExecuteQuery, and the public +/// never accepts a row it cannot evaluate. +/// +/// Up to 2.1.4 the fast path keyed each row's values col0, col1, … before evaluating the WHERE, so a +/// condition naming a column matched nothing (grp = 2 returned 0 rows), and a condition the compiled predicate +/// could not parse fell back to "accept every row" (length(name) = 3 returned the whole table). +/// +/// +public sealed class ExecuteQueryFastWhereTests : IDisposable +{ + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_QueryFast_{Guid.NewGuid():N}"); + private readonly Database _db; + + public ExecuteQueryFastWhereTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + var factory = services.BuildServiceProvider().GetRequiredService(); + _db = (Database)factory.Create(_root, "pw"); + _db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, grp INTEGER, name TEXT)"); + _db.ExecuteSQL("INSERT INTO t (id, grp, name) VALUES (1, 2, 'bb')"); + _db.ExecuteSQL("INSERT INTO t (id, grp, name) VALUES (2, 1, 'a')"); + _db.ExecuteSQL("INSERT INTO t (id, grp, name) VALUES (3, 2, 'ccc')"); + } + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + [Theory] + [InlineData("SELECT * FROM t WHERE grp = 2", new[] { 1, 3 })] + [InlineData("SELECT * FROM t WHERE name = 'a'", new[] { 2 })] + [InlineData("SELECT * FROM t WHERE grp IN (1)", new[] { 2 })] + [InlineData("SELECT * FROM t WHERE name LIKE 'c%'", new[] { 3 })] + [InlineData("SELECT * FROM t WHERE grp = 2 AND name = 'bb'", new[] { 1 })] + [InlineData("SELECT * FROM t WHERE length(name) = 3", new[] { 3 })] + public void ExecuteQueryFast_WithWhere_ReturnsTheMatchingRows(string sql, int[] expectedIds) + { + var ids = _db.ExecuteQueryFast(sql) + .Select(r => Convert.ToInt32(r.GetValueBoxed(0), CultureInfo.InvariantCulture)) + .Order() + .ToArray(); + + Assert.Equal(expectedIds, ids); + } + + [Fact] + public void CompileWhereClause_WithAConditionItCannotParse_EvaluatesItInsteadOfAcceptingEveryRow() + { + var predicate = SqlParserPerformanceOptimizations.CompileWhereClause("length(name) = 3"); + + Assert.False(predicate(Row("bb"))); + Assert.True(predicate(Row("ccc"))); + } + + private static Dictionary Row(string name) => + new(StringComparer.OrdinalIgnoreCase) { ["id"] = 1L, ["name"] = name }; +} diff --git a/tests/SharpCoreDB.Tests/MultiRowInsertBatchingTests.cs b/tests/SharpCoreDB.Tests/MultiRowInsertBatchingTests.cs index 569ea2d0..23089b61 100644 --- a/tests/SharpCoreDB.Tests/MultiRowInsertBatchingTests.cs +++ b/tests/SharpCoreDB.Tests/MultiRowInsertBatchingTests.cs @@ -137,13 +137,19 @@ public void CanUseBatchedInsert_IsFalse_WithCheckConstraint() } [Fact] - public void CanUseBatchedInsert_IsFalse_WithUniqueSecondaryIndex() + public void CanUseBatchedInsert_IsTrue_WithUniqueSecondaryIndex_BecauseTheBatchCoreEnforcesIt() { + // Up to 2.1.4 a unique secondary index forced the per-row loop, because the batch core did not enforce it. The + // batch core now validates unique indexes (Table.UniqueKeys), so the batched path stays on — and still rejects a + // duplicate, keeping none of the statement's rows. var db = Open(NewDir("gate_unique")); db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT NOT NULL, score INTEGER)"); TableOf(db).CreateHashIndex("idx_t_name_unique", "name", isUnique: true); + db.ExecuteSQL("INSERT INTO t (id, name, score) VALUES (1, 'a', 1)"); - Assert.False(TableOf(db).CanUseBatchedInsert); + Assert.True(TableOf(db).CanUseBatchedInsert); + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, name, score) VALUES (2, 'b', 2), (3, 'a', 3)")); + Assert.Single(db.ExecuteQuery("SELECT id FROM t")); (db as IDisposable)?.Dispose(); } diff --git a/tests/SharpCoreDB.Tests/PrimaryKeyUniquenessTests.cs b/tests/SharpCoreDB.Tests/PrimaryKeyUniquenessTests.cs new file mode 100644 index 00000000..101a3f65 --- /dev/null +++ b/tests/SharpCoreDB.Tests/PrimaryKeyUniquenessTests.cs @@ -0,0 +1,119 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.IO; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// A multi-row INSERT that fails is one atomic unit — nothing of it is kept — on both storage modes, as in SQLite, and +/// directory storage enforces PRIMARY KEY uniqueness. +/// +/// Up to 2.1.4 a single-file multi-row INSERT ran row by row, so a row failing NOT NULL left the rows before it in +/// place; directory storage validates the whole statement first, and single-file storage now takes the same batch +/// path. Single-file PRIMARY KEY uniqueness is not enforced (the skipped cases, known issue §11). +/// +/// +public sealed class PrimaryKeyUniquenessTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_PkUnique_{Guid.NewGuid():N}"); + + public PrimaryKeyUniquenessTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + [Theory] + [InlineData("directory")] + [InlineData("single-file", Skip = "Single-file storage does not enforce PRIMARY KEY uniqueness, and cannot do so safely yet: its format persists the PrimaryKey flag on one column only, so a composite key (EF Core declares one inline PRIMARY KEY per key column) is indistinguishable from a single-column key. See docs/sharpcoredb-known-issues.md, §11.")] + public void Insert_DuplicatePrimaryKey_IsRejected(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, 'x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, 'dup')")); + Assert.Single(db.ExecuteQuery("SELECT id FROM t")); + } + + [Theory] + [InlineData("directory")] + [InlineData("single-file", Skip = "Single-file storage does not enforce PRIMARY KEY uniqueness, and cannot do so safely yet: its format persists the PrimaryKey flag on one column only, so a composite key (EF Core declares one inline PRIMARY KEY per key column) is indistinguishable from a single-column key. See docs/sharpcoredb-known-issues.md, §11.")] + public void MultiRowInsert_RepeatingAnExistingKey_KeepsNoRowOfTheStatement(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, 'x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, name) VALUES (2, 'y'), (1, 'dup')")); + Assert.Single(db.ExecuteQuery("SELECT id FROM t")); + } + + [Theory] + [InlineData("directory")] + [InlineData("single-file", Skip = "Single-file storage does not enforce PRIMARY KEY uniqueness, and cannot do so safely yet: its format persists the PrimaryKey flag on one column only, so a composite key (EF Core declares one inline PRIMARY KEY per key column) is indistinguishable from a single-column key. See docs/sharpcoredb-known-issues.md, §11.")] + public void MultiRowInsert_RepeatingAKeyWithinTheStatement_KeepsNoRow(string mode) + { + var (db, _) = Open(mode); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, name) VALUES (5, 'a'), (5, 'b')")); + Assert.Empty(db.ExecuteQuery("SELECT id FROM t")); + } + + [Theory] + [InlineData("directory")] + [InlineData("single-file")] + public void MultiRowInsert_WithANotNullViolation_KeepsNoRow(string mode) + { + var (db, _) = Open(mode); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, 'a'), (2, NULL), (3, 'c')")); + Assert.Empty(db.ExecuteQuery("SELECT id FROM t")); + } + + [Fact(Skip = "Single-file storage does not enforce PRIMARY KEY uniqueness, and cannot do so safely yet: its format persists the PrimaryKey flag on one column only, so a composite key (EF Core declares one inline PRIMARY KEY per key column) is indistinguishable from a single-column key. See docs/sharpcoredb-known-issues.md, §11.")] + public void Insert_DuplicatePrimaryKey_AfterReopen_IsRejectedOnSingleFile() + { + var (db, path) = Open("single-file"); + db.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, 'x')"); + (db as IDisposable)?.Dispose(); + + var reopened = _factory.CreateWithOptions(path, "pw", DatabaseOptions.CreateSingleFileDefault()); + Assert.ThrowsAny(() => reopened.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, 'dup')")); + Assert.Single(reopened.ExecuteQuery("SELECT id FROM t")); + (reopened as IDisposable)?.Dispose(); + } + + private (IDatabase Db, string Path) Open(string mode) + { + var path = Path.Combine(_root, mode == "single-file" ? $"db_{Guid.NewGuid():N}.scdb" : $"db_{Guid.NewGuid():N}"); + var db = mode == "single-file" + ? _factory.CreateWithOptions(path, "pw", DatabaseOptions.CreateSingleFileDefault()) + : _factory.Create(path, "pw"); + db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT NOT NULL)"); + return (db, path); + } +} diff --git a/tests/SharpCoreDB.Tests/PrimaryKeyUpdateTests.cs b/tests/SharpCoreDB.Tests/PrimaryKeyUpdateTests.cs new file mode 100644 index 00000000..495bfdaa --- /dev/null +++ b/tests/SharpCoreDB.Tests/PrimaryKeyUpdateTests.cs @@ -0,0 +1,126 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// An UPDATE that changes a PRIMARY KEY may not move a row onto a key another row holds, and a rejected UPDATE changes +/// nothing — on columnar and page-based storage, in the session and after a reopen. +/// +/// Up to 2.1.4 the new key was never checked (known issue §8): UPDATE t SET id = 1 WHERE id = 3 with a row 1 +/// present was accepted, the primary-key index was re-pointed at the moved row, and the original row 1 became +/// unreachable — silent data loss. Single-file storage enforces no primary-key uniqueness at all yet (§11). +/// +/// +public sealed class PrimaryKeyUpdateTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_PkUpdate_{Guid.NewGuid():N}"); + + public PrimaryKeyUpdateTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public static TheoryData Modes => new() { "columnar", "page-based" }; + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_PrimaryKeyOntoAnExistingKey_IsRejectedAndLosesNoRow(string mode) + { + var (db, _) = Open(mode); + + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET id = 1 WHERE id = 3")); + Assert.Equal([(1, "one"), (3, "three")], Rows(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_PrimaryKeyOntoAnExistingKey_AfterReopen_LosesNoRow(string mode) + { + var (db, path) = Open(mode); + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET id = 1 WHERE id = 3")); + (db as IDisposable)?.Dispose(); + + var reopened = _factory.Create(path, "pw"); + Assert.Equal([(1, "one"), (3, "three")], Rows(reopened)); + (reopened as IDisposable)?.Dispose(); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_PrimaryKeyToAFreeKey_MovesTheRow(string mode) + { + var (db, _) = Open(mode); + + db.ExecuteSQL("UPDATE t SET id = 7 WHERE id = 3"); + + Assert.Equal([(1, "one"), (7, "three")], Rows(db)); + Assert.Equal("three", db.ExecuteQuery("SELECT name FROM t WHERE id = 7").Single()["name"]); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_PrimaryKeyToItsOwnValue_IsAccepted(string mode) + { + var (db, _) = Open(mode); + + db.ExecuteSQL("UPDATE t SET id = 3, name = 'still three' WHERE id = 3"); + + Assert.Equal([(1, "one"), (3, "still three")], Rows(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_PrimaryKeyOfSeveralRowsToOneKey_IsRejectedAndChangesNothing(string mode) + { + var (db, _) = Open(mode); + + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET id = 9 WHERE id > 0")); + Assert.Equal([(1, "one"), (3, "three")], Rows(db)); + } + + private (IDatabase Db, string Path) Open(string mode) + { + var path = Path.Combine(_root, $"db_{Guid.NewGuid():N}"); + var db = _factory.Create(path, "pw"); + var create = "CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT)"; + db.ExecuteSQL(mode == "page-based" ? create + " STORAGE = PAGE_BASED" : create); + db.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, 'one')"); + db.ExecuteSQL("INSERT INTO t (id, name) VALUES (3, 'three')"); + return (db, path); + } + + private static (int Id, string? Name)[] Rows(IDatabase db) => + [.. db.ExecuteQuery("SELECT id, name FROM t") + .Select(r => (Convert.ToInt32(r["id"], CultureInfo.InvariantCulture), r["name"] as string)) + .OrderBy(r => r.Item1)]; +} diff --git a/tests/SharpCoreDB.Tests/ScalarFunctionOverColumnTests.cs b/tests/SharpCoreDB.Tests/ScalarFunctionOverColumnTests.cs new file mode 100644 index 00000000..e5fcb65f --- /dev/null +++ b/tests/SharpCoreDB.Tests/ScalarFunctionOverColumnTests.cs @@ -0,0 +1,266 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// A scalar function applied to a column — strftime('%Y', at), upper(name), length(name) — is +/// evaluated per row, in the SELECT list and in WHERE, on both storage modes, as SQLite does. +/// +/// Up to 2.1.4 only a SELECT without FROM evaluated functions. Over a table the SELECT list returned the whole row +/// unchanged (so SELECT upper(name) FROM t read back the id column), and every WHERE evaluator +/// treated the left operand as a column name: strftime(…, at) = … matched no row. linq2db's SQLite dialect +/// wraps every comparison in strftime, so every date filter written through the +/// linq2db adapter returned nothing. +/// +/// +public sealed class ScalarFunctionOverColumnTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_ScalarOverColumn_{Guid.NewGuid():N}"); + + public ScalarFunctionOverColumnTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public static TheoryData StorageModes => new() { "directory", "single-file" }; + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + // ── SELECT list ───────────────────────────────────────────────────────────────────────────── + + [Theory] + [MemberData(nameof(StorageModes))] + public void Select_FunctionOfColumn_ReturnsOnlyTheFunctionValue(string mode) + { + var db = Open(mode); + + var row = Assert.Single(db.ExecuteQuery("SELECT upper(name) FROM ev WHERE id = 1")); + + Assert.Equal("ALPHA", Assert.Single(row).Value); + } + + [Theory] + [MemberData(nameof(StorageModes))] + public void Select_ColumnAndAliasedFunction_KeepsOrderAndAlias(string mode) + { + var db = Open(mode); + + var row = Assert.Single(db.ExecuteQuery("SELECT id, strftime('%Y', at) AS y FROM ev WHERE id = 2")); + + Assert.Equal(["id", "y"], row.Keys); + Assert.Equal(2L, Convert.ToInt64(row["id"], CultureInfo.InvariantCulture)); + Assert.Equal("2025", row["y"]); + } + + [Theory] + [MemberData(nameof(StorageModes))] + public void Select_DateTimeFunctionsOfColumn_UseSqliteFormats(string mode) + { + var db = Open(mode); + + var row = Assert.Single(db.ExecuteQuery("SELECT date(at), time(at), datetime(at) FROM ev WHERE id = 3")); + + Assert.Equal(["2026-03-10", "23:59:59", "2026-03-10 23:59:59"], row.Values); + } + + [Theory] + [MemberData(nameof(StorageModes))] + public void Select_FunctionOfNullColumn_IsNull(string mode) + { + var db = Open(mode); + + var row = Assert.Single(db.ExecuteQuery("SELECT length(name) AS n FROM ev WHERE id = 4")); + + Assert.True(row["n"] is null or DBNull); + } + + [Theory] + [MemberData(nameof(StorageModes))] + public void Select_CastOfFunction_ReturnsInteger(string mode) + { + var db = Open(mode); + + var row = Assert.Single(db.ExecuteQuery("SELECT CAST(strftime('%m', at) AS INTEGER) AS m FROM ev WHERE id = 2")); + + Assert.Equal(6L, row["m"]); + } + + // ── WHERE ─────────────────────────────────────────────────────────────────────────────────── + + [Theory] + [InlineData("directory", "strftime('%Y', at) = '2026'", new[] { 1, 3 })] + [InlineData("single-file", "strftime('%Y', at) = '2026'", new[] { 1, 3 })] + [InlineData("directory", "strftime('%Y-%m-%d %H:%M:%f', [x].[at]) = strftime('%Y-%m-%d %H:%M:%f', '2026-01-01 12:00:00.000')", new[] { 1 })] + [InlineData("single-file", "strftime('%Y-%m-%d %H:%M:%f', [x].[at]) = strftime('%Y-%m-%d %H:%M:%f', '2026-01-01 12:00:00.000')", new[] { 1 })] + [InlineData("directory", "strftime('%Y-%m-%d %H:%M:%f', [x].[at]) > strftime('%Y-%m-%d %H:%M:%f', '2026-01-01 00:00:00.000')", new[] { 1, 3 })] + [InlineData("single-file", "strftime('%Y-%m-%d %H:%M:%f', [x].[at]) > strftime('%Y-%m-%d %H:%M:%f', '2026-01-01 00:00:00.000')", new[] { 1, 3 })] + [InlineData("directory", "length(name) = 4", new[] { 2 })] + [InlineData("single-file", "length(name) = 4", new[] { 2 })] + [InlineData("directory", "upper(name) = 'GAMMA'", new[] { 3 })] + [InlineData("single-file", "upper(name) = 'GAMMA'", new[] { 3 })] + [InlineData("directory", "substr(name, 1, 1) = 'A'", new[] { 1 })] + [InlineData("single-file", "substr(name, 1, 1) = 'A'", new[] { 1 })] + [InlineData("directory", "CAST(strftime('%Y', at) AS INTEGER) = 2025", new[] { 2 })] + [InlineData("single-file", "CAST(strftime('%Y', at) AS INTEGER) = 2025", new[] { 2 })] + [InlineData("directory", "lower(name) LIKE 'g%'", new[] { 3 })] + [InlineData("single-file", "lower(name) LIKE 'g%'", new[] { 3 })] + [InlineData("directory", "date(at) = '2026-03-10' AND id > 1", new[] { 3 })] + [InlineData("single-file", "date(at) = '2026-03-10' AND id > 1", new[] { 3 })] + [InlineData("directory", "strftime('%Y', at) = '2026' OR name = 'beta'", new[] { 1, 2, 3 })] + [InlineData("single-file", "strftime('%Y', at) = '2026' OR name = 'beta'", new[] { 1, 2, 3 })] + [InlineData("directory", "length(name) IS NULL", new[] { 4 })] + [InlineData("single-file", "length(name) IS NULL", new[] { 4 })] + // Two text values compare as text, as in SQLite: '01' is not '1'. + [InlineData("directory", "substr(at, 6, 2) = '1'", new int[0])] + [InlineData("single-file", "substr(at, 6, 2) = '1'", new int[0])] + [InlineData("directory", "substr(at, 6, 2) = '01'", new[] { 1 })] + [InlineData("single-file", "substr(at, 6, 2) = '01'", new[] { 1 })] + public void Where_FunctionOfColumn_FiltersRows(string mode, string where, int[] expectedIds) + { + var db = Open(mode); + + var ids = Ids(db.ExecuteQuery($"SELECT id FROM ev x WHERE {where}")); + + Assert.Equal(expectedIds, ids); + } + + [Theory] + [MemberData(nameof(StorageModes))] + public void CountStar_WithFunctionInWhere_CountsMatchingRows(string mode) + { + var db = Open(mode); + + var row = Assert.Single(db.ExecuteQuery("SELECT COUNT(*) FROM ev WHERE strftime('%Y', at) = '2026'")); + + Assert.Equal(2L, Convert.ToInt64(Assert.Single(row).Value, CultureInfo.InvariantCulture)); + } + + [Theory] + [MemberData(nameof(StorageModes))] + public void Update_WithFunctionInWhere_ChangesOnlyMatchingRows(string mode) + { + var db = Open(mode); + + db.ExecuteSQL("UPDATE ev SET name = 'changed' WHERE strftime('%Y', at) = '2025'"); + + var names = db.ExecuteQuery("SELECT id, name FROM ev") + .ToDictionary(r => Convert.ToInt32(r["id"], CultureInfo.InvariantCulture), r => r["name"] as string); + Assert.Equal("Alpha", names[1]); + Assert.Equal("changed", names[2]); + Assert.Equal("Gamma", names[3]); + Assert.Null(names[4]); + } + + [Theory] + [MemberData(nameof(StorageModes))] + public void Delete_WithFunctionInWhere_DeletesOnlyMatchingRows(string mode) + { + var db = Open(mode); + + db.ExecuteSQL("DELETE FROM ev WHERE date(at) = '2026-03-10'"); + + Assert.Equal([1, 2, 4], Ids(db.ExecuteQuery("SELECT id FROM ev"))); + } + + [Theory] + [MemberData(nameof(StorageModes))] + public void Where_FunctionOfColumnWithSubquery_FiltersRowsOnTheAstPath(string mode) + { + // A subquery routes the statement to the AST executor, which mapped a function operand to NULL. + var db = Open(mode); + + var ids = Ids(db.ExecuteQuery("SELECT id FROM ev WHERE strftime('%Y', at) = '2026' AND id IN (SELECT id FROM ev)")); + + Assert.Equal([1, 3], ids); + } + + // ── SELECT without FROM (literal arguments) ───────────────────────────────────────────────── + + [Theory] + [InlineData("SELECT date('2026-03-10 23:59:59')", "2026-03-10")] + [InlineData("SELECT time('2026-03-10 23:59:59.999')", "23:59:59")] + [InlineData("SELECT datetime('2026-03-10T23:59:59')", "2026-03-10 23:59:59")] + [InlineData("SELECT strftime('%Y-%m-%d %H:%M', '2026-01-02T03:04:05+02:00')", "2026-01-02 01:04")] + [InlineData("SELECT date('2026-01-31', '+1 month')", "2026-03-03")] + [InlineData("SELECT date('2026-03-10', 'start of month')", "2026-03-01")] + [InlineData("SELECT strftime('%j %w %%', '2026-03-10')", "069 2 %")] + [InlineData("SELECT upper('abc')", "ABC")] + public void SelectWithoutFrom_ScalarFunctions_MatchSqlite(string sql, string expected) + { + var db = Open("directory"); + + var row = Assert.Single(db.ExecuteQuery(sql)); + + Assert.Equal(expected, Convert.ToString(Assert.Single(row).Value, CultureInfo.InvariantCulture)); + } + + [Theory] + [InlineData("SELECT julianday('2000-01-01 12:00:00')", 2451545.0)] + [InlineData("SELECT unixepoch('1970-01-02 00:00:00')", 86400.0)] + public void SelectWithoutFrom_NumericDateFunctions_MatchSqlite(string sql, double expected) + { + var db = Open("directory"); + + var row = Assert.Single(db.ExecuteQuery(sql)); + + Assert.Equal(expected, Convert.ToDouble(Assert.Single(row).Value, CultureInfo.InvariantCulture), 6); + } + + [Fact] + public void SelectWithoutFrom_InvalidDate_IsNull() + { + var db = Open("directory"); + + var row = Assert.Single(db.ExecuteQuery("SELECT date('not a date') AS d")); + + Assert.True(row["d"] is null or DBNull); + } + + // ── Helpers ───────────────────────────────────────────────────────────────────────────────── + + private IDatabase Open(string mode) + { + var db = mode == "single-file" + ? _factory.CreateWithOptions(Path.Combine(_root, $"db_{Guid.NewGuid():N}.scdb"), "pw", DatabaseOptions.CreateSingleFileDefault()) + : _factory.Create(Path.Combine(_root, $"db_{Guid.NewGuid():N}"), "pw"); + + db.ExecuteSQL("CREATE TABLE ev (id INTEGER PRIMARY KEY, name TEXT, at TEXT)"); + db.ExecuteSQL("INSERT INTO ev (id, name, at) VALUES (1, 'Alpha', '2026-01-01 12:00:00.000')"); + db.ExecuteSQL("INSERT INTO ev (id, name, at) VALUES (2, 'beta', '2025-06-15 08:30:00.250')"); + db.ExecuteSQL("INSERT INTO ev (id, name, at) VALUES (3, 'Gamma', '2026-03-10 23:59:59.999')"); + db.ExecuteSQL("INSERT INTO ev (id, name, at) VALUES (4, NULL, NULL)"); + return db; + } + + private static int[] Ids(List> rows) => + [.. rows.Select(r => Convert.ToInt32(r["id"], CultureInfo.InvariantCulture)).Order()]; +} diff --git a/tests/SharpCoreDB.Tests/SingleTableOrderByTests.cs b/tests/SharpCoreDB.Tests/SingleTableOrderByTests.cs new file mode 100644 index 00000000..8908c4aa --- /dev/null +++ b/tests/SharpCoreDB.Tests/SingleTableOrderByTests.cs @@ -0,0 +1,131 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// ORDER BY on the single-table SELECT path follows SQLite: every key in the list, each with its own direction, +/// expressions and SELECT-list aliases as keys, NULLs first when ascending, BINARY collation unless the column or the +/// key says otherwise — and the sort happens before OFFSET/LIMIT. +/// +/// Up to 2.1.4 the path read only the single token after ORDER BY. A second key was ignored — ORDER BY grp, name +/// came back unsorted, which is what linq2db emits for OrderBy(…).ThenBy(…) — an expression or alias key left +/// the rows unsorted, a qualified key ([x].[name]) matched no row value, and a single text column was ordered +/// with the machine's culture (case-insensitively under en-US) instead of SQLite's BINARY collation. +/// +/// +public sealed class SingleTableOrderByTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_OrderBy_{Guid.NewGuid():N}"); + + public SingleTableOrderByTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + [Theory] + [InlineData("directory", "SELECT id FROM t ORDER BY name", new[] { 5, 4, 2, 1, 3 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY name", new[] { 5, 4, 2, 1, 3 })] + [InlineData("directory", "SELECT id FROM t ORDER BY name DESC", new[] { 3, 1, 2, 4, 5 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY name DESC", new[] { 3, 1, 2, 4, 5 })] + [InlineData("directory", "SELECT id FROM t ORDER BY name NULLS LAST", new[] { 4, 2, 1, 3, 5 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY name NULLS LAST", new[] { 4, 2, 1, 3, 5 })] + [InlineData("directory", "SELECT id FROM t ORDER BY grp, name", new[] { 5, 4, 2, 1, 3 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY grp, name", new[] { 5, 4, 2, 1, 3 })] + [InlineData("directory", "SELECT id FROM t ORDER BY grp DESC, name ASC", new[] { 1, 3, 4, 2, 5 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY grp DESC, name ASC", new[] { 1, 3, 4, 2, 5 })] + [InlineData("directory", "SELECT id FROM t ORDER BY length(name) DESC", new[] { 4, 3, 1, 2, 5 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY length(name) DESC", new[] { 4, 3, 1, 2, 5 })] + [InlineData("directory", "SELECT id FROM t ORDER BY strftime('%Y', at)", new[] { 5, 4, 2, 3, 1 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY strftime('%Y', at)", new[] { 5, 4, 2, 3, 1 })] + [InlineData("directory", "SELECT id FROM t ORDER BY upper(name)", new[] { 5, 2, 1, 3, 4 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY upper(name)", new[] { 5, 2, 1, 3, 4 })] + [InlineData("directory", "SELECT id, upper(name) AS u FROM t ORDER BY u DESC", new[] { 4, 3, 1, 2, 5 })] + [InlineData("single-file", "SELECT id, upper(name) AS u FROM t ORDER BY u DESC", new[] { 4, 3, 1, 2, 5 })] + [InlineData("directory", "SELECT id FROM t ORDER BY name COLLATE NOCASE", new[] { 5, 2, 1, 3, 4 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY name COLLATE NOCASE", new[] { 5, 2, 1, 3, 4 })] + [InlineData("directory", "SELECT id FROM t ORDER BY nc", new[] { 5, 2, 1, 3, 4 })] + [InlineData("directory", "SELECT [x].[id] FROM t x ORDER BY [x].[name]", new[] { 5, 4, 2, 1, 3 })] + [InlineData("single-file", "SELECT [x].[id] FROM t x ORDER BY [x].[name]", new[] { 5, 4, 2, 1, 3 })] + [InlineData("directory", "SELECT id, name FROM t ORDER BY 2", new[] { 5, 4, 2, 1, 3 })] + [InlineData("single-file", "SELECT id, name FROM t ORDER BY 2", new[] { 5, 4, 2, 1, 3 })] + [InlineData("directory", "SELECT id FROM t ORDER BY grp, length(name) DESC LIMIT 3", new[] { 5, 4, 2 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY grp, length(name) DESC LIMIT 3", new[] { 5, 4, 2 })] + [InlineData("directory", "SELECT id FROM t ORDER BY name LIMIT 2 OFFSET 1", new[] { 4, 2 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY name LIMIT 2 OFFSET 1", new[] { 4, 2 })] + [InlineData("directory", "SELECT id FROM t WHERE grp = 2 ORDER BY length(name) DESC", new[] { 3, 1 })] + [InlineData("single-file", "SELECT id FROM t WHERE grp = 2 ORDER BY length(name) DESC", new[] { 3, 1 })] + // SQLite's LIMIT forms: `LIMIT m, n` (offset m) and a negative count meaning no limit — linq2db writes + // LIMIT -1 OFFSET m for Skip without Take. Up to 2.1.4 both, and LIMIT n OFFSET m above, threw "Invalid LIMIT + // clause", and LIMIT 0 returned every row. + [InlineData("directory", "SELECT id FROM t ORDER BY name LIMIT 1, 2", new[] { 4, 2 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY name LIMIT 1, 2", new[] { 4, 2 })] + [InlineData("directory", "SELECT id FROM t ORDER BY name LIMIT -1 OFFSET 3", new[] { 1, 3 })] + [InlineData("single-file", "SELECT id FROM t ORDER BY name LIMIT -1 OFFSET 3", new[] { 1, 3 })] + [InlineData("directory", "SELECT id FROM t ORDER BY name LIMIT 0", new int[0])] + [InlineData("single-file", "SELECT id FROM t ORDER BY name LIMIT 0", new int[0])] + public void OrderBy_OnTheSingleTablePath_SortsAsSqliteDoes(string mode, string sql, int[] expectedIds) + { + var db = Open(mode); + + var ids = db.ExecuteQuery(sql).Select(r => Convert.ToInt32(r["id"], CultureInfo.InvariantCulture)).ToArray(); + + Assert.Equal(expectedIds, ids); + } + + [Fact(Skip = "Single-file storage does not record a column's declared COLLATE (it is neither parsed nor persisted, " + + "and its WHERE evaluator has no collation support): an owner decision on extending the column-entry " + + "format is pending. See docs/sharpcoredb-known-issues.md, 'Single-file storage ignores COLLATE'.")] + public void OrderBy_ColumnDeclaredNoCase_OnSingleFile_SortsCaseInsensitively() + { + var db = Open("single-file"); + + var ids = db.ExecuteQuery("SELECT id FROM t ORDER BY nc").Select(r => Convert.ToInt32(r["id"], CultureInfo.InvariantCulture)).ToArray(); + + Assert.Equal([5, 2, 1, 3, 4], ids); + } + + private IDatabase Open(string mode) + { + var db = mode == "single-file" + ? _factory.CreateWithOptions(Path.Combine(_root, $"db_{Guid.NewGuid():N}.scdb"), "pw", DatabaseOptions.CreateSingleFileDefault()) + : _factory.Create(Path.Combine(_root, $"db_{Guid.NewGuid():N}"), "pw"); + + db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, grp INTEGER, name TEXT, nc TEXT COLLATE NOCASE, at TEXT)"); + db.ExecuteSQL("INSERT INTO t (id, grp, name, nc, at) VALUES (1, 2, 'bb', 'bb', '2026-03-01 00:00:00.000')"); + db.ExecuteSQL("INSERT INTO t (id, grp, name, nc, at) VALUES (2, 1, 'a', 'a', '2024-01-01 00:00:00.000')"); + db.ExecuteSQL("INSERT INTO t (id, grp, name, nc, at) VALUES (3, 2, 'ccc', 'ccc', '2025-06-01 00:00:00.000')"); + db.ExecuteSQL("INSERT INTO t (id, grp, name, nc, at) VALUES (4, 1, 'Dddd', 'Dddd', '2023-01-01 00:00:00.000')"); + db.ExecuteSQL("INSERT INTO t (id, grp, name, nc, at) VALUES (5, NULL, NULL, NULL, NULL)"); + return db; + } +} diff --git a/tests/SharpCoreDB.Tests/SqlFunctionsDateFamilyTests.cs b/tests/SharpCoreDB.Tests/SqlFunctionsDateFamilyTests.cs new file mode 100644 index 00000000..8c9bd8dd --- /dev/null +++ b/tests/SharpCoreDB.Tests/SqlFunctionsDateFamilyTests.cs @@ -0,0 +1,62 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Collections.Generic; +using System.Globalization; +using SharpCoreDB.Services; +using Xunit; + +/// +/// The public answers SQLite's date and time functions with SQLite's +/// argument order and format language, through the engine's one implementation, while keeping its legacy +/// (DateTime, .NET format) shape for STRFTIME and DATE. +/// +/// Up to 2.1.4 STRFTIME and DATE only answered for a argument (anything else was +/// NULL), TIME, DATETIME, JULIANDAY and UNIXEPOCH threw , +/// and formatted with the current culture, so th-TH printed the year 2026 as 2569. +/// +/// +public sealed class SqlFunctionsDateFamilyTests +{ + [Theory] + [InlineData("STRFTIME", new object[] { "%Y-%m", "2026-03-10" }, "2026-03")] + [InlineData("DATE", new object[] { "2026-03-10 10:00:00" }, "2026-03-10")] + [InlineData("TIME", new object[] { "2026-03-10 10:11:12" }, "10:11:12")] + [InlineData("DATETIME", new object[] { "2026-03-10", "+1 day" }, "2026-03-11 00:00:00")] + [InlineData("UNIXEPOCH", new object[] { "1970-01-02" }, "86400")] + [InlineData("JULIANDAY", new object[] { "2000-01-01 12:00:00" }, "2451545")] + public void EvaluateFunction_DateFamilyWithSqliteArguments_AnswersAsSqlite(string name, object[] args, string expected) + { + var value = SqlFunctions.EvaluateFunction(name, new List(args)); + + Assert.Equal(expected, Convert.ToString(value, CultureInfo.InvariantCulture)); + } + + [Fact] + public void EvaluateFunction_StrftimeWithLegacyDateTimeAndDotNetFormat_KeepsItsBehaviour() + { + var value = SqlFunctions.EvaluateFunction("STRFTIME", [new DateTime(2026, 3, 10), "yyyy/MM"]); + + Assert.Equal("2026/03", value); + } + + [Fact] + public void StrFTime_UnderANonGregorianCulture_UsesTheGregorianYear() + { + var previous = CultureInfo.CurrentCulture; + CultureInfo.CurrentCulture = CultureInfo.GetCultureInfo("th-TH"); + try + { + Assert.Equal("2026-03-10", SqlFunctions.StrFTime(new DateTime(2026, 3, 10), "yyyy-MM-dd")); + } + finally + { + CultureInfo.CurrentCulture = previous; + } + } +} diff --git a/tests/SharpCoreDB.Tests/SqlLiteralCultureTests.cs b/tests/SharpCoreDB.Tests/SqlLiteralCultureTests.cs new file mode 100644 index 00000000..47a33144 --- /dev/null +++ b/tests/SharpCoreDB.Tests/SqlLiteralCultureTests.cs @@ -0,0 +1,185 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.DataStructures; +using SharpCoreDB.Interfaces; +using SharpCoreDB.Services; +using Xunit; + +/// +/// Values rendered into SQL text are culture-invariant, in every copy of the "value to SQL literal" switch. +/// +/// Up to 2.1.4 the four copies — ParameterBinder.FormatParameter (the ADO.NET binder), +/// CompiledQueryExecutor.FormatParameterValue, SqlFunctions.Quote and Table.FormatValue — +/// formatted dates with the current culture's calendar (th-TH stored the year 2026 as 2569), rendered a +/// DateTimeOffset, DateOnly or TimeOnly through its culture-dependent ToString() +/// ('1/1/2026 12:30:15 PM +10:00' under en-US, without its milliseconds), three of them dropped the +/// milliseconds of a DateTime, and Table.FormatValue wrote a double as 1,5 on a +/// comma-decimal machine. Found through the linq2db adapter, whose DateTimeOffset column read back as 2569. +/// +/// +public sealed class SqlLiteralCultureTests : IDisposable +{ + private static readonly DateTime SampleDateTime = new(2026, 1, 1, 12, 30, 15, 250); + private static readonly DateTimeOffset SampleOffset = new(SampleDateTime, TimeSpan.FromHours(10)); + + private readonly string _dirPath = Path.Combine(Path.GetTempPath(), $"SCDB_SqlLiteralCulture_{Guid.NewGuid():N}"); + + public static TheoryData Cultures => new() { "en-US", "th-TH", "sv-SE", "de-DE" }; + + /// Every copy of the formatter, by name, so a failure names the copy that regressed. + public static TheoryData CulturesAndFormatters + { + get + { + var data = new TheoryData(); + foreach (var culture in new[] { "en-US", "th-TH", "sv-SE", "de-DE" }) + { + foreach (var formatter in Formatters.Keys) + { + data.Add(culture, formatter); + } + } + + return data; + } + } + + private static readonly Dictionary> Formatters = new() + { + ["ParameterBinder.FormatParameter"] = ParameterBinder.FormatParameter, + ["CompiledQueryExecutor.FormatParameterValue"] = CompiledQueryExecutor.FormatParameterValue, + ["SqlFunctions.Quote"] = SqlFunctions.Quote, + ["Table.FormatValue"] = Table.FormatValue, + }; + + [Theory] + [MemberData(nameof(CulturesAndFormatters))] + public void Format_DateTime_IsInvariantWithMilliseconds(string culture, string formatter) + { + var literal = WithCulture(culture, () => Formatters[formatter](SampleDateTime)); + + Assert.Equal("'2026-01-01 12:30:15.250'", literal); + } + + [Theory] + [MemberData(nameof(CulturesAndFormatters))] + public void Format_DateTimeOffset_IsInvariantWithOffset(string culture, string formatter) + { + var literal = WithCulture(culture, () => Formatters[formatter](SampleOffset)); + + Assert.Equal("'2026-01-01 12:30:15.250+10:00'", literal); + } + + [Theory] + [MemberData(nameof(CulturesAndFormatters))] + public void Format_DateOnlyAndTimeOnly_AreInvariant(string culture, string formatter) + { + var (date, time) = WithCulture(culture, () => ( + Formatters[formatter](DateOnly.FromDateTime(SampleDateTime)), + Formatters[formatter](TimeOnly.FromDateTime(SampleDateTime)))); + + Assert.Equal("'2026-01-01'", date); + Assert.Equal("'12:30:15.250'", time); + } + + [Theory] + [MemberData(nameof(CulturesAndFormatters))] + public void Format_NegativeIntegers_UseAsciiMinus(string culture, string formatter) + { + var (i, l) = WithCulture(culture, () => (Formatters[formatter](-5), Formatters[formatter](-5L))); + + Assert.Equal("-5", i.Trim('\'')); + Assert.Equal("-5", l.Trim('\'')); + } + + [Theory] + [MemberData(nameof(Cultures))] + public void TableFormatValue_Double_UsesInvariantDecimalPoint(string culture) + { + var literal = WithCulture(culture, () => Table.FormatValue(-1.5)); + + Assert.Equal("-1.5", literal); + } + + [Theory] + [MemberData(nameof(Cultures))] + public void ExecuteSQL_WithTemporalParameters_StoresInvariantText(string culture) + { + var db = Open(); + + WithCulture(culture, () => + { + db.ExecuteSQL( + "INSERT INTO t (id, at, at_offset, day, clock, delta) VALUES (@id, @at, @offset, @day, @clock, @delta)", + new Dictionary + { + ["@id"] = 1, + ["@at"] = SampleDateTime, + ["@offset"] = SampleOffset, + ["@day"] = DateOnly.FromDateTime(SampleDateTime), + ["@clock"] = TimeOnly.FromDateTime(SampleDateTime), + ["@delta"] = -5, + }); + return 0; + }); + + var row = Assert.Single(db.ExecuteQuery("SELECT * FROM t WHERE id = 1")); + Assert.Equal("2026-01-01 12:30:15.250", Text(row["at"])); + Assert.Equal("2026-01-01 12:30:15.250+10:00", Text(row["at_offset"])); + Assert.Equal("2026-01-01", Text(row["day"])); + Assert.Equal("12:30:15.250", Text(row["clock"])); + Assert.Equal("-5", Text(row["delta"])); + } + + public void Dispose() + { + try + { + if (Directory.Exists(_dirPath)) + { + Directory.Delete(_dirPath, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + private IDatabase Open() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + var factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_dirPath); + var db = factory.Create(_dirPath, "pw", isReadOnly: false, config: new DatabaseConfig { NoEncryptMode = true }); + db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, at TEXT, at_offset TEXT, day TEXT, clock TEXT, delta TEXT)"); + return db; + } + + private static string? Text(object? value) => Convert.ToString(value, CultureInfo.InvariantCulture); + + private static T WithCulture(string name, Func action) + { + var previous = CultureInfo.CurrentCulture; + CultureInfo.CurrentCulture = CultureInfo.GetCultureInfo(name); + try + { + return action(); + } + finally + { + CultureInfo.CurrentCulture = previous; + } + } +} diff --git a/tests/SharpCoreDB.Tests/UniqueConstraintTests.cs b/tests/SharpCoreDB.Tests/UniqueConstraintTests.cs new file mode 100644 index 00000000..c4e2ff81 --- /dev/null +++ b/tests/SharpCoreDB.Tests/UniqueConstraintTests.cs @@ -0,0 +1,221 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// A single-column UNIQUE constraint — column-level email TEXT UNIQUE or table-level UNIQUE (email) — +/// is enforced on INSERT, multi-row INSERT and UPDATE, on columnar and page-based storage, and after a reopen; NULLs +/// never conflict, as in SQLite. Single-file storage is not covered yet (the skipped test). +/// +/// Up to 2.1.4 the constraint was parsed and stored but never checked (known issue §9): the INSERT-time validation +/// was a stub, a UNIQUE index was checked only on a single-row INSERT into a columnar table, and a multi-row +/// INSERT or an UPDATE was never checked at all. A UNIQUE column now gets an automatic unique index, as in SQLite +/// (sqlite_autoindex_…), and one validator covers every write path. +/// +/// +public sealed class UniqueConstraintTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_Unique_{Guid.NewGuid():N}"); + + public UniqueConstraintTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public static TheoryData Modes => new() { "columnar", "page-based" }; + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_DuplicateIntoUniqueColumn_IsRejected(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'a@x')")); + Assert.Equal([1], Ids(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_NullsIntoUniqueColumn_AreAllowed(string mode) + { + var (db, _) = Open(mode); + + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, NULL)"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, NULL)"); + + Assert.Equal([1, 2], Ids(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void MultiRowInsert_DuplicateWithinTheStatement_KeepsNoRow(string mode) + { + var (db, _) = Open(mode); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x'), (2, 'b@x'), (3, 'a@x')")); + Assert.Empty(Ids(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void MultiRowInsert_DuplicatingAStoredValue_KeepsNoRowOfTheStatement(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'b@x'), (3, 'a@x')")); + Assert.Equal([1], Ids(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_UniqueColumnToAnotherRowsValue_IsRejectedAndChangesNothing(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'b@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET email = 'a@x' WHERE id = 2")); + Assert.Equal("b@x", Email(db, 2)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_UniqueColumnToItsOwnValue_IsAccepted(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + db.ExecuteSQL("UPDATE t SET email = 'a@x' WHERE id = 1"); + + Assert.Equal("a@x", Email(db, 1)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_UniqueColumnToAFreeValue_IsAccepted(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'b@x')"); + + db.ExecuteSQL("UPDATE t SET email = 'c@x' WHERE id = 2"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (3, 'b@x')"); + + Assert.Equal("c@x", Email(db, 2)); + Assert.Equal([1, 2, 3], Ids(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_UniqueColumnOnSeveralRowsToOneValue_IsRejectedAndChangesNothing(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'b@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET email = 'z@x' WHERE id > 0")); + Assert.Equal("a@x", Email(db, 1)); + Assert.Equal("b@x", Email(db, 2)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_DuplicateIntoUniqueColumn_AfterReopen_IsRejected(string mode) + { + var (db, path) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + (db as IDisposable)?.Dispose(); + + var reopened = Reopen(mode, path); + Assert.ThrowsAny(() => reopened.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'a@x')")); + Assert.Equal([1], Ids(reopened)); + (reopened as IDisposable)?.Dispose(); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_DuplicateIntoTableLevelUniqueColumn_IsRejected(string mode) + { + var (db, _) = Open(mode, "CREATE TABLE t (id INTEGER PRIMARY KEY, email TEXT, UNIQUE (email))"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'a@x')")); + Assert.Equal([1], Ids(db)); + } + + [Fact(Skip = "Single-file storage does not enforce UNIQUE yet: its persisted ColumnFlags.Unique is also written for " + + "every inline PRIMARY KEY column (\"PRIMARY KEY implies UNIQUE\"), so a reopened file cannot tell a declared " + + "UNIQUE from a column of an EF Core composite key. Needs the column-entry format decision of known issues " + + "§10/§11.")] + public void Insert_DuplicateIntoUniqueColumn_IsRejectedOnSingleFile() + { + var (db, _) = Open("single-file"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'a@x')")); + Assert.Equal([1], Ids(db)); + } + + [Fact] + public void MultiRowInsert_DuplicatingAUniqueIndex_IsRejected() + { + // A UNIQUE index was checked only on a single-row INSERT; the batched multi-row path skipped it. + var (db, _) = Open("columnar", "CREATE TABLE t (id INTEGER PRIMARY KEY, email TEXT)"); + db.ExecuteSQL("CREATE UNIQUE INDEX ux_email ON t (email)"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'b@x'), (3, 'a@x')")); + Assert.Equal([1], Ids(db)); + } + + private (IDatabase Db, string Path) Open(string mode, string? ddl = null) + { + var path = Path.Combine(_root, mode == "single-file" ? $"db_{Guid.NewGuid():N}.scdb" : $"db_{Guid.NewGuid():N}"); + var db = Reopen(mode, path); + var create = ddl ?? "CREATE TABLE t (id INTEGER PRIMARY KEY, email TEXT UNIQUE)"; + db.ExecuteSQL(mode == "page-based" ? create + " STORAGE = PAGE_BASED" : create); + return (db, path); + } + + private IDatabase Reopen(string mode, string path) => mode == "single-file" + ? _factory.CreateWithOptions(path, "pw", DatabaseOptions.CreateSingleFileDefault()) + : _factory.Create(path, "pw"); + + private static int[] Ids(IDatabase db) => + [.. db.ExecuteQuery("SELECT id FROM t").Select(r => Convert.ToInt32(r["id"], CultureInfo.InvariantCulture)).Order()]; + + private static string? Email(IDatabase db, int id) => + db.ExecuteQuery($"SELECT email FROM t WHERE id = {id}").Single()["email"] as string; +} diff --git a/tools/measure-coverage.ps1 b/tools/measure-coverage.ps1 index 968ce56f..192a1548 100644 --- a/tools/measure-coverage.ps1 +++ b/tools/measure-coverage.ps1 @@ -47,8 +47,10 @@ (Category!=Debug&Category!=Manual&Category!=Performance). .PARAMETER Projects - Test projects to measure, relative to the repository root. Default the three suites codecov.yml counts - (SharpCoreDB.Tests, SharpCoreDB.VectorSearch.Tests, SharpCoreDB.EntityFrameworkCore.Tests). + Test projects to measure, relative to the repository root. Default the four suites CI runs + (SharpCoreDB.Tests, SharpCoreDB.VectorSearch.Tests, SharpCoreDB.EntityFrameworkCore.Tests, + SharpCoreDB.Functional.Linq2DB.Tests). The linq2db suite was left out until 2026-11-17, so the adapter + assembly it covers counted as 0% covered — or, since it was never loaded by the other suites, not at all. .PARAMETER OutputDirectory Where the Cobertura reports are written, relative to the repository root. Default artifacts/coverage. @@ -80,7 +82,8 @@ param( [string[]]$Projects = @( 'tests/SharpCoreDB.Tests/SharpCoreDB.Tests.csproj', 'tests/SharpCoreDB.VectorSearch.Tests/SharpCoreDB.VectorSearch.Tests.csproj', - 'tests/SharpCoreDB.EntityFrameworkCore.Tests/SharpCoreDB.EntityFrameworkCore.Tests.csproj' + 'tests/SharpCoreDB.EntityFrameworkCore.Tests/SharpCoreDB.EntityFrameworkCore.Tests.csproj', + 'tests/SharpCoreDB.Functional.Linq2DB.Tests/SharpCoreDB.Functional.Linq2DB.Tests.csproj' ), [string]$OutputDirectory = 'artifacts/coverage', [string]$Configuration = 'Release',