diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index ecee93e0..86ba22aa 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -52,6 +52,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **`SqlFunctions.EvaluateFunction` answered SQLite's date functions only for a `DateTime` argument** (`STRFTIME`, `DATE`; anything else was NULL) and threw for `TIME`, `DATETIME`, `JULIANDAY`, `UNIXEPOCH`. They now go through `SqlDateTimeFunctions` with SQLite's argument order and format language; the legacy `(DateTime, .NET format)` shape keeps its behaviour. `SqlFunctions.StrFTime` formatted with the current culture (th-TH printed 2026 as 2569) and is now invariant. Pinned by `SqlFunctionsDateFamilyTests`. - **Recorded, not fixed (`docs/sharpcoredb-known-issues.md` §8–§11):** an UPDATE that moves a primary key onto an existing key loses a row on directory storage and duplicates it on single-file storage; a column-level `UNIQUE` is enforced on neither mode; single-file storage ignores a declared `COLLATE`; and single-file storage enforces no PRIMARY KEY uniqueness, which cannot be switched on safely until composite keys are modelled — EF Core declares one inline `PRIMARY KEY` per key column and both modes keep only the last. Turning single-file enforcement on was tried and backed out: `CompleteExampleIntegrationTests.CompositeKey_*` showed existing EF Core databases would reject legitimate inserts. §8 and §9 need the write-path performance gate; §10 and §11 need an owner decision on the column-entry format. Suites: core **2360 / 0 failed (5 skipped: the documented single-file cases)**, VectorSearch 255 / 0, EF Core 116 / 0, linq2DB **96 / 0**, regression canaries 51 / 0. +- **An UPDATE that moved a PRIMARY KEY onto an existing key lost a row (directory storage).** `UPDATE t SET id = 1 WHERE id = 3` with a row 1 present was accepted: the primary-key index was re-pointed at the moved row and the original row 1 became unreachable. Every UPDATE path — `Table.UpdateAffectedCount` (SQL UPDATE), `UpdateMultipleCore` (`ExecuteBatchSQL`, the structured batch API) and `UpdateByPrimaryKey` — now validates new key values before any row is written, counting what earlier operations of a batch claim and release; a SET of the key matching more than one row is rejected too. The check starts with a test that allocates nothing and returns at once for an UPDATE that does not touch a key. Pinned by `PrimaryKeyUpdateTests` (columnar and page-based, in the session and after a reopen). Known issue §8; its single-file half stays with §11. +- **A UNIQUE constraint was never enforced; a UNIQUE index only partly.** A column declared `UNIQUE` (or `UNIQUE (col)`) accepted duplicates on every storage mode: the INSERT-time check was a stub. A UNIQUE *index* was checked only by a single-row INSERT into a columnar table — not on page-based tables, not by a multi-row INSERT, not by an UPDATE — and rejected a second NULL as a duplicate. A single-column UNIQUE now gets an automatic unique hash index on directory storage, as SQLite's `sqlite_autoindex_…` (registered at CREATE TABLE and on every open, so existing tables are covered; rows stored earlier never block an open). One validator covers INSERT, both batch critical sections and every UPDATE path; NULLs never conflict; and a table with a unique index keeps the batched — atomic — multi-row INSERT, which it used to give up for a per-row loop that left earlier rows behind. Pinned by `UniqueConstraintTests` (columnar, page-based) and `MultiRowInsertBatchingTests.CanUseBatchedInsert_IsTrue_WithUniqueSecondaryIndex_BecauseTheBatchCoreEnforcesIt`, which replaces the test that pinned the old per-row fallback. Known issue §9. Still open: single-file storage, whose persisted `ColumnFlags.Unique` is also written for every inline PRIMARY KEY column, so a reopened file cannot tell a declared UNIQUE from an EF Core composite key's column (the format decision of §10/§11; enforcing it was tried and backed out when `CompleteExampleIntegrationTests.CompositeKey_*` failed); and a composite `UNIQUE (a, b)` on INSERT. Suites: core **2392 / 0 failed (6 skipped: documented single-file cases)**, VectorSearch 255 / 0, EF Core 116 / 0, linq2DB 96 / 0, regression canaries 51 / 0; write-path gate: GATE PASSED on a NOISY box (WSearch and Defender need elevation), every metric ok, worst best-of-reps ratio 1.22x on raw UPDATE against a 1.50x limit while its median ratio was 0.60x (faster). + ## [2.1.4] - 2026-11-13 > The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema diff --git a/docs/sharpcoredb-known-issues.md b/docs/sharpcoredb-known-issues.md index 82c0b52f..09abf5ba 100644 --- a/docs/sharpcoredb-known-issues.md +++ b/docs/sharpcoredb-known-issues.md @@ -421,10 +421,14 @@ attribution before merge. --- -## 8. 🔴 OPEN — An UPDATE that moves a PRIMARY KEY onto an existing key loses or duplicates a row +## 8. ✅ FIXED (directory storage) — An UPDATE that moves a PRIMARY KEY onto an existing key lost a row -> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/linq2db-adapter-and-scalar-functions`).** Reproduced on both storage -> modes with the statements below; not fixed in this change. +> ✅ **FIXED (2026-11-17, `fix/pk-update-and-unique-constraints`) on directory storage.** Every UPDATE path — +> `Table.UpdateAffectedCount` (SQL UPDATE), `UpdateMultipleCore` (`ExecuteBatchSQL` and the structured batch API) and +> `UpdateByPrimaryKey` — now validates a new key before any row is written: a key another row holds, or a SET of the key +> that matches more than one row, is rejected and nothing changes. Pinned by `PrimaryKeyUpdateTests` (columnar and +> page-based, in the session and after a reopen). The single-file half — a duplicated key, not a lost row — is the same +> composite-key question as §11 and is tracked there. Found 2026-11-17. Neither storage mode checks the new key of an UPDATE that changes the primary key: @@ -444,10 +448,17 @@ SELECT id FROM t; --- -## 9. 🟠 OPEN — A column-level `UNIQUE` constraint is not enforced +## 9. ✅ FIXED (directory storage) — A column-level `UNIQUE` constraint was not enforced -> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/linq2db-adapter-and-scalar-functions`).** Reproduced on both storage -> modes; not fixed in this change. +> ✅ **FIXED (2026-11-17, `fix/pk-update-and-unique-constraints`) on directory storage.** A single-column UNIQUE +> (column-level or table-level `UNIQUE (col)`) gets an automatic unique hash index, as SQLite's `sqlite_autoindex_…` — +> registered at CREATE TABLE and on every open, so existing tables are covered. INSERT, multi-row INSERT (now atomic on +> tables with a unique index) and every UPDATE path are checked; NULLs never conflict. On the way: a UNIQUE *index* was +> enforced only by a single-row INSERT into a columnar table, and a second NULL was rejected as a duplicate. Pinned by +> `UniqueConstraintTests` (columnar and page-based). **Still open:** single-file storage (its persisted +> `ColumnFlags.Unique` is also written for every inline PRIMARY KEY column, so a reopened file cannot tell a declared +> UNIQUE from an EF Core composite key's column — the format decision of §10/§11; tried and backed out when +> `CompleteExampleIntegrationTests.CompositeKey_*` failed) and a composite `UNIQUE (a, b)` on INSERT. Found 2026-11-17. A column declared `UNIQUE` accepts duplicates on both storage modes: @@ -489,7 +500,8 @@ on-disk format extension. A `Locale(…)` collation would still need a name fiel Found 2026-11-17 while fixing single-file constraint enforcement. - **Single-file storage does not enforce PRIMARY KEY uniqueness.** `INSERT … VALUES (2, 'y'), (1, 'dup')` into a - table already holding id 1 leaves two rows with id 1. + table already holding id 1 leaves two rows with id 1, and an UPDATE that moves a key onto an existing one (§8's + single-file half) duplicates it. - **Neither mode models a composite key.** EF Core's `EnsureCreated` (`SharpCoreDBDatabaseCreator`) declares one inline `PRIMARY KEY` per key column; both storage modes keep only the *last* such column as `PrimaryKeyIndex`. Directory storage then enforces uniqueness on that one column — a false violation as soon as two rows share it @@ -519,7 +531,7 @@ single-file cases in `PrimaryKeyUniquenessTests`. | 5 | SQL validator `@` prefix | 🟡 Low | Keys without `@` prefix | **Fixed** — `NormalizeKey` strips `@` / `:` | | 6 | INTEGER → Int32 | 🟡 Low | `BIGINT` columns | **Still the default** — opt in with `UseSqliteIntegerAffinity` | | 7 | Bound text re-parsed as SQL | 🔴 High | Write through `ITable.Insert` instead of SQL text | **Fixed** — one `SqlLiteralScanner` (SQL/SQLite: a doubled quote is one quote, a backslash is data) behind every scanner; `ParseValue` strips one pair and un-collapses doubling; a tuple with too many values is rejected. `ParameterLiteralRoundTripTests` 74/74 | -| 8 | PK-changing UPDATE loses/duplicates a row | 🔴 High | — | **Open** — directory loses the overwritten row, single-file duplicates the key; write-path gate needed (§8) | -| 9 | Column `UNIQUE` not enforced | 🟠 Medium | — | **Open** — both storage modes accept duplicates (§9) | +| 8 | PK-changing UPDATE loses/duplicates a row | 🔴 High | — | **Fixed on directory storage** — every UPDATE path validates the new key first; single-file part tracked in §11 | +| 9 | Column `UNIQUE` not enforced | 🟠 Medium | — | **Fixed on directory storage** — automatic unique index; single-file (format decision, §10/§11) and composite `UNIQUE (a, b)` still open | | 10 | Single-file ignores `COLLATE` | 🟡 Low | — | **Open, owner decision** — needs a column-entry format extension (§10) | | 11 | PK uniqueness / composite keys | 🔴 High | — | **Open, owner decision** — single-file enforces no PK uniqueness; composite keys are modelled as their last column (§11) | \ No newline at end of file diff --git a/src/SharpCoreDB/DataStructures/HashIndex.cs b/src/SharpCoreDB/DataStructures/HashIndex.cs index 7b50999e..c8263674 100644 --- a/src/SharpCoreDB/DataStructures/HashIndex.cs +++ b/src/SharpCoreDB/DataStructures/HashIndex.cs @@ -106,7 +106,7 @@ internal void Add(object key, long position) { // Unique path: atomic check + add under outer lock var keyBytes = BuildUnsafeKey(normalizedKey); - if (HasUnsafeRowsForKey(keyBytes)) + if (normalizedKey is not DBNull && HasUnsafeRowsForKey(keyBytes)) { throw new InvalidOperationException( $"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'"); @@ -122,7 +122,7 @@ internal void Add(object key, long position) list = []; _index[normalizedKey] = list; } - else if (_isUnique && list.Count > 0) + else if (_isUnique && list.Count > 0 && normalizedKey is not DBNull) { throw new InvalidOperationException( $"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'"); @@ -463,7 +463,7 @@ private void AddBatchKeysLockedCore(object?[] keys, long[] positions) { // Unique path: atomic check + add under outer lock. var keyBytes = BuildUnsafeKey(normalizedKey); - if (HasUnsafeRowsForKey(keyBytes)) + if (normalizedKey is not DBNull && HasUnsafeRowsForKey(keyBytes)) { throw new InvalidOperationException( $"Duplicate key value '{keys[i]}' violates unique constraint on index '{_columnName}'"); @@ -479,7 +479,7 @@ private void AddBatchKeysLockedCore(object?[] keys, long[] positions) list = []; _index[normalizedKey] = list; } - else if (_isUnique && list.Count > 0) + else if (_isUnique && list.Count > 0 && normalizedKey is not DBNull) { throw new InvalidOperationException( $"Duplicate key value '{keys[i]}' violates unique constraint on index '{_columnName}'"); @@ -552,7 +552,7 @@ public void AddBatch(List> rows, long[] positions) { // Unique path: atomic check + add under outer lock var keyBytes = BuildUnsafeKey(normalizedKey); - if (HasUnsafeRowsForKey(keyBytes)) + if (normalizedKey is not DBNull && HasUnsafeRowsForKey(keyBytes)) { throw new InvalidOperationException( $"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'"); @@ -567,7 +567,7 @@ public void AddBatch(List> rows, long[] positions) list = []; _index[normalizedKey] = list; } - else if (_isUnique && list.Count > 0) + else if (_isUnique && list.Count > 0 && normalizedKey is not DBNull) { throw new InvalidOperationException( $"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'"); diff --git a/src/SharpCoreDB/DataStructures/Table.CRUD.cs b/src/SharpCoreDB/DataStructures/Table.CRUD.cs index 3c9a40ee..4f49ba31 100644 --- a/src/SharpCoreDB/DataStructures/Table.CRUD.cs +++ b/src/SharpCoreDB/DataStructures/Table.CRUD.cs @@ -98,21 +98,6 @@ public void Insert(Dictionary row) } } - // ✅ UNIQUE validation (outside lock) - foreach (var uniqueConstraint in this.UniqueConstraints) - { - if (uniqueConstraint.Count == 1) // Single column unique - { - var colName = uniqueConstraint[0]; - var colIndex = this.Columns.IndexOf(colName); - if (colIndex >= 0 && row.TryGetValue(colName, out var value) && value != null && value != DBNull.Value) - { - // Check if value already exists (simplified - would need index lookup in real impl) - // For now, just validate non-null for single column unique - } - } - } - // ✅ CHECK constraint validation (outside lock) for (int i = 0; i < this.Columns.Count; i++) { @@ -189,7 +174,7 @@ public void Insert(Dictionary row) continue; } - if (!row.TryGetValue(registeredCol, out var value) || value is null) + if (!row.TryGetValue(registeredCol, out var value) || value is null or DBNull) { continue; } @@ -202,6 +187,10 @@ public void Insert(Dictionary row) } } + // UNIQUE columns and unique indexes, on every storage mode (the check above runs for columnar tables + // only and skips an index it has not loaded). Returns at once for a table without unique indexes. + ValidateUniqueValues(row); + // ✅ NEW: Route through storage engine var engine = GetOrCreateStorageEngine(); @@ -278,8 +267,8 @@ public void Insert(Dictionary row) /// this table — i.e. when nothing in the table's schema relies on the per-row-only semantics that the /// batch core does not implement. /// - /// What the batch core does NOT do: CHECK-constraint evaluation (per-column and table-level) and non-PK - /// unique-index enforcement. PK and NOT NULL validation are implemented on both paths, and the + /// What the batch core does NOT do: CHECK-constraint evaluation (per-column and table-level). PK, NOT NULL and + /// UNIQUE (column constraints and unique indexes) validation are implemented on both paths, and the /// batch core joins an open engine transaction rather than nesting one /// (InsertBatchCriticalSection: needsTransaction = !engine.IsInTransaction). /// @@ -307,14 +296,6 @@ internal bool CanUseBatchedInsert } } - foreach (var metadata in this.registeredIndexes.Values) - { - if (metadata.IsUnique) - { - return false; - } - } - return _database?.IsBatchUpdateActive != true; } } @@ -802,6 +783,9 @@ private long[] InsertBatchCriticalSection(object[][] validatedRows, List // Validate primary keys (requires lock for index access) ValidateExistingPrimaryKeys(validatedRows); + // UNIQUE columns and unique indexes: up to 2.1.4 the batched path checked neither. + ValidateUniqueValues(validatedRows); + var engine = GetOrCreateStorageEngine(); bool needsTransaction = !engine.IsInTransaction; @@ -882,6 +866,9 @@ private long[] InsertBatchCriticalSection( List> validatedRows, List serializedRows) { + // UNIQUE columns and unique indexes: up to 2.1.4 the batched path checked neither. + ValidateUniqueValues(validatedRows); + // Validate primary keys (requires lock for index access) if (this.PrimaryKeyIndex >= 0) { @@ -1778,6 +1765,13 @@ public int UpdateAffectedCount(string? where, Dictionary updates var rows = ResolveUpdateRows(where); int affected = 0; + // A new PRIMARY KEY or UNIQUE value must not be another row's, and is checked before any row is written. + // Up to 2.1.4 a primary key moved onto an existing key re-pointed the index and lost the original row. + if (rows.Count > 0 && UpdateTouchesKeyColumns(updates)) + { + ValidateKeyChanges([(rows.ConvertAll(r => r.Row), updates)]); + } + foreach (var (rowPos, row) in rows) { affected++; @@ -2510,6 +2504,9 @@ private int UpdateMultipleCore( // put an atomic per operation into the loop it measures. long structuredOps = 0; + // Key changes are validated for the whole batch before any operation is written (known issues §8/§9). + ValidateBatchKeyChanges(operations); + foreach (var (where, structuredKeyColumn, structuredKeyValue, structuredKeyLiteral, updates) in operations) { // §5.3 session 2 (2026-09-26): the per-operation head of this loop — key resolution, the @@ -5326,6 +5323,11 @@ public bool UpdateByPrimaryKey(object key, Dictionary updates) if (row == null) return false; + if (UpdateTouchesKeyColumns(updates)) + { + ValidateKeyChanges([([row], updates)]); + } + // WP13: capture only what index maintenance needs instead of copying the whole row. Dictionary? oldHashKeys = null; foreach (var kvp in this.hashIndexes) // NOSONAR:S3267 - deliberate: LINQ Select/Where would allocate per point-update on the hot path diff --git a/src/SharpCoreDB/DataStructures/Table.UniqueKeys.cs b/src/SharpCoreDB/DataStructures/Table.UniqueKeys.cs new file mode 100644 index 00000000..a72f9cc5 --- /dev/null +++ b/src/SharpCoreDB/DataStructures/Table.UniqueKeys.cs @@ -0,0 +1,342 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.DataStructures; + +using System; +using System.Collections.Generic; +using System.Globalization; + +/// +/// Table partial — uniqueness of the PRIMARY KEY and of single-column UNIQUE constraints on every write path. +/// +/// Up to 2.1.4 two guarantees were missing (known issues §8 and §9). An UPDATE that changed the primary key never +/// checked the new key: UPDATE t SET id = 1 WHERE id = 3 with a row 1 present was accepted, the primary-key +/// index was re-pointed at the moved row and the original row 1 became unreachable. And a UNIQUE column constraint was +/// parsed and stored but never enforced — the INSERT-time check was a stub, a UNIQUE index was checked only on +/// a single-row INSERT into a columnar table, and multi-row INSERTs and UPDATEs were never checked. +/// +/// +/// A UNIQUE column now gets an automatic unique hash index, as SQLite gives it a sqlite_autoindex_…, so the +/// check is a hash lookup. Every check starts with a test that allocates nothing and returns at once for a table or a +/// statement without keys to protect, which keeps the measured write paths unchanged. +/// +/// +public partial class Table +{ + /// + /// Registers an automatic unique hash index for every single-column UNIQUE constraint that has none — at CREATE TABLE + /// and on every open, so tables created before this existed are covered too. Registration does not build anything: + /// the index is built from the stored rows on first use, and rows stored before (duplicates included) never make + /// registration fail; only a new duplicate is refused. + /// + internal void EnsureUniqueConstraintIndexes() + { + for (int i = 0; i < this.UniqueConstraints.Count; i++) + { + var constraint = this.UniqueConstraints[i]; + if (constraint.Count != 1) + { + continue; + } + + var column = this.Columns.Find(c => c.Equals(constraint[0], StringComparison.OrdinalIgnoreCase)); + if (column is null || (this.PrimaryKeyIndex >= 0 && column == this.Columns[this.PrimaryKeyIndex])) + { + continue; + } + + var indexName = $"sqlite_autoindex_{this.Name}_{i + 1}"; + if (this.indexNameToColumn.ContainsKey(indexName)) + { + continue; + } + + CreateHashIndex(indexName, column, isUnique: true); + } + } + + /// Returns true when any unique index (other than the primary key's) is registered. Allocates nothing. + private bool HasUniqueIndex() + { + foreach (var (column, metadata) in + this.registeredIndexes) // NOSONAR:S3267 - write hot path: LINQ would allocate + { + if (metadata.IsUnique && !IsPrimaryKeyColumn(column)) + { + return true; + } + } + + return false; + } + + /// + /// Returns true when sets the primary key or a uniquely indexed column. Allocates + /// nothing, so every UPDATE path can ask it first. + /// + private bool UpdateTouchesKeyColumns(Dictionary updates) + { + if (this.PrimaryKeyIndex >= 0 && TryGetUpdateValue(updates, this.Columns[this.PrimaryKeyIndex], out _)) + { + return true; + } + + foreach (var (column, metadata) in + this.registeredIndexes) // NOSONAR:S3267 - write hot path: LINQ would allocate + { + if (metadata.IsUnique && !IsPrimaryKeyColumn(column) && TryGetUpdateValue(updates, column, out _)) + { + return true; + } + } + + return false; + } + + // The three entry points below take the no-unique-index exit before anything is captured: a lambda that captures a + // parameter makes the compiler allocate its closure at method entry, so the capturing calls live in the *Core + // helpers and an INSERT into a table without unique indexes allocates nothing here. + + /// Validates one inserted row against the unique indexes. Returns at once without any. + private void ValidateUniqueValues(Dictionary row) + { + if (HasUniqueIndex()) + { + ValidateUniqueValuesCore(row); + } + } + + /// Validates inserted rows (one statement) against the unique indexes and against each other. + private void ValidateUniqueValues(IReadOnlyList> rows) + { + if (HasUniqueIndex()) + { + ValidateUniqueValuesCore(rows); + } + } + + /// Validates inserted rows in table column order (the batch parser's shape). + private void ValidateUniqueValues(object[][] rows) + { + if (HasUniqueIndex()) + { + ValidateUniqueValuesCore(rows); + } + } + + private void ValidateUniqueValuesCore(Dictionary row) => + ValidateUniqueValues(1, (_, column) => row.TryGetValue(column, out var value) ? value : null); + + private void ValidateUniqueValuesCore(IReadOnlyList> rows) => + ValidateUniqueValues(rows.Count, (r, column) => rows[r].TryGetValue(column, out var value) ? value : null); + + private void ValidateUniqueValuesCore(object[][] rows) => + ValidateUniqueValues(rows.Length, (r, column) => + { + int index = this.Columns.IndexOf(column); + return index >= 0 && index < rows[r].Length ? rows[r][index] : null; + }); + + private void ValidateUniqueValues(int rowCount, Func valueAt) + { + foreach (var column in UniqueIndexedColumns()) + { + EnsureIndexLoaded(column); + this.hashIndexes.TryGetValue(column, out var hashIndex); + var seen = new HashSet(UniqueKeyComparer(column)); + for (int r = 0; r < rowCount; r++) + { + var value = valueAt(r, column); + if (value is null or DBNull) + { + continue; // NULLs never conflict, as in SQLite + } + + if (!seen.Add(UniqueKey(value)) || (hashIndex is not null && hashIndex.ContainsKey(value))) + { + throw DuplicateUniqueValue(column, value); + } + } + } + } + + /// + /// Validates a statement's (or a batch's) key changes before any row is written: a primary key or a uniquely indexed + /// column may only take a value no other row holds — counting the values earlier changes in the same batch claim and + /// release — and a SET of such a column may match at most one row. An UPDATE that keeps a row's own value passes. + /// + /// Each change's matched rows and SET values, in execution order. + private void ValidateKeyChanges( + IReadOnlyList<(IReadOnlyList> Rows, Dictionary Updates)> changes) + { + if (this.PrimaryKeyIndex >= 0) + { + var pkColumn = this.Columns[this.PrimaryKeyIndex]; + ValidateKeyColumnChanges(changes, pkColumn, StringComparer.Ordinal, (key, _) => IsPrimaryKeyTaken(key), + isPrimaryKey: true); + } + + foreach (var column in UniqueIndexedColumns()) + { + EnsureIndexLoaded(column); + this.hashIndexes.TryGetValue(column, out var hashIndex); + ValidateKeyColumnChanges( + changes, + column, + UniqueKeyComparer(column), + (_, value) => hashIndex is not null && hashIndex.ContainsKey(value), + isPrimaryKey: false); + } + } + + private void ValidateKeyColumnChanges( + IReadOnlyList<(IReadOnlyList> Rows, Dictionary Updates)> changes, + string column, + StringComparer comparer, + Func isTaken, + bool isPrimaryKey) + { + var claimed = new HashSet(comparer); + var released = new HashSet(comparer); + foreach (var (rows, updates) in changes) + { + if (rows.Count == 0 || !TryGetUpdateValue(updates, column, out var newValue) || newValue is null or DBNull) + { + continue; + } + + var newKey = UniqueKey(newValue); + if (rows.Count > 1) + { + throw isPrimaryKey + ? new InvalidOperationException( + $"Primary key violation: the UPDATE would give {rows.Count} rows the key {newKey}") + : DuplicateUniqueValue(column, newValue); + } + + var oldValue = rows[0].TryGetValue(column, out var current) ? current : null; + var oldKey = oldValue is null or DBNull ? null : UniqueKey(oldValue); + if (oldKey is not null && comparer.Equals(oldKey, newKey)) + { + continue; // the row keeps its own value + } + + if (!claimed.Add(newKey) || (isTaken(newKey, newValue) && !released.Contains(newKey))) + { + throw isPrimaryKey + ? new InvalidOperationException($"Primary key violation: {newKey}") + : DuplicateUniqueValue(column, newValue); + } + + if (oldKey is not null) + { + released.Add(oldKey); + } + } + } + + /// + /// Validates the key changes of a batched UPDATE (UpdateMultipleCore, the path ExecuteBatchSQL and the + /// structured batch API take) before any operation is written. The scan for an operation that touches a key column + /// allocates nothing; only those operations have their rows resolved. + /// + private void ValidateBatchKeyChanges( + IReadOnlyList<(string Where, string? KeyColumn, object? KeyValue, string? KeyLiteral, Dictionary + Updates)> operations) + { + for (int i = 0; i < operations.Count; i++) + { + if (UpdateTouchesKeyColumns(operations[i].Updates)) + { + ValidateBatchKeyChangesCore(operations); + return; + } + } + } + + private void ValidateBatchKeyChangesCore( + IReadOnlyList<(string Where, string? KeyColumn, object? KeyValue, string? KeyLiteral, Dictionary + Updates)> operations) + { + var changes = new List<(IReadOnlyList> Rows, Dictionary Updates)>(); + foreach (var (where, keyColumn, keyValue, keyLiteral, updates) in operations) + { + if (!UpdateTouchesKeyColumns(updates)) + { + continue; + } + + // The same WHERE text UpdateMultipleCore falls back to for a structured operation. + var whereText = where.Length > 0 || keyColumn is null + ? where + : keyLiteral is not null + ? keyColumn + " = " + keyLiteral + : BuildStructuredWhereText(keyColumn, keyValue); + + changes.Add((ResolveUpdateRows(whereText).ConvertAll(r => r.Row), updates)); + } + + ValidateKeyChanges(changes); + } + + /// The uniquely indexed columns other than the primary key (which has its own index and check). + private List UniqueIndexedColumns() + { + var columns = new List(); + foreach (var (column, metadata) in this.registeredIndexes) + { + if (metadata.IsUnique && !IsPrimaryKeyColumn(column)) + { + columns.Add(column); + } + } + + return columns; + } + + private bool IsPrimaryKeyColumn(string column) => + this.PrimaryKeyIndex >= 0 && + column.Equals(this.Columns[this.PrimaryKeyIndex], StringComparison.OrdinalIgnoreCase); + + /// Compares unique keys with the column's collation: case-insensitively for NOCASE columns. + private StringComparer UniqueKeyComparer(string column) + { + int index = this.Columns.IndexOf(column); + var collation = index >= 0 && index < this.ColumnCollations.Count + ? this.ColumnCollations[index] + : CollationType.Binary; + return collation is CollationType.NoCase or CollationType.UnicodeCaseInsensitive + ? StringComparer.OrdinalIgnoreCase + : StringComparer.Ordinal; + } + + private static string UniqueKey(object value) => + value as string ?? Convert.ToString(value, CultureInfo.InvariantCulture) ?? string.Empty; + + private static bool TryGetUpdateValue(Dictionary updates, string column, out object? value) + { + if (updates.TryGetValue(column, out var exact)) + { + value = exact; + return true; + } + + foreach (var (key, candidate) in updates) // NOSONAR:S3267 - SET lists are tiny; LINQ would allocate per call + { + if (key.Equals(column, StringComparison.OrdinalIgnoreCase)) + { + value = candidate; + return true; + } + } + + value = null; + return false; + } + + private static InvalidOperationException DuplicateUniqueValue(string column, object value) => + new($"Duplicate key value '{value}' violates unique constraint on column '{column}'"); +} diff --git a/src/SharpCoreDB/Database/Core/Database.Core.cs b/src/SharpCoreDB/Database/Core/Database.Core.cs index 52012fed..34f03cd5 100644 --- a/src/SharpCoreDB/Database/Core/Database.Core.cs +++ b/src/SharpCoreDB/Database/Core/Database.Core.cs @@ -468,6 +468,10 @@ config is { } fixedWidthConfig && { table.DefaultExpressions.Add(null); } + + // UNIQUE constraints are enforced through automatic unique indexes; tables created before they existed + // get theirs here. Registration builds nothing, so rows stored earlier never block an open. + table.EnsureUniqueConstraintIndexes(); // ✅ Phase 2: Backward compatible — missing column check expressions default to null // REQUIRED: Table.Insert indexes ColumnCheckExpressions[i] without bounds checks, // so this list MUST be padded to the column count after deserialization. diff --git a/src/SharpCoreDB/Services/SqlParser.DDL.cs b/src/SharpCoreDB/Services/SqlParser.DDL.cs index cf5c0ca4..65572531 100644 --- a/src/SharpCoreDB/Services/SqlParser.DDL.cs +++ b/src/SharpCoreDB/Services/SqlParser.DDL.cs @@ -374,6 +374,13 @@ private void ExecuteCreateTable(string sql, string[] parts, IWAL? wal) ColumnLocaleNames = columnLocaleNames, }); + // A UNIQUE column gets an automatic unique index, as SQLite's sqlite_autoindex_…: it is what enforces the + // constraint. Up to 2.1.4 the constraint was stored and never checked. + if (table is SharpCoreDB.DataStructures.Table createdTable) + { + createdTable.EnsureUniqueConstraintIndexes(); + } + table.Name = tableName; // B7+: new columnar tables with an explicitly declared PRIMARY KEY default to the fixed-width diff --git a/tests/SharpCoreDB.Tests/MultiRowInsertBatchingTests.cs b/tests/SharpCoreDB.Tests/MultiRowInsertBatchingTests.cs index 569ea2d0..23089b61 100644 --- a/tests/SharpCoreDB.Tests/MultiRowInsertBatchingTests.cs +++ b/tests/SharpCoreDB.Tests/MultiRowInsertBatchingTests.cs @@ -137,13 +137,19 @@ public void CanUseBatchedInsert_IsFalse_WithCheckConstraint() } [Fact] - public void CanUseBatchedInsert_IsFalse_WithUniqueSecondaryIndex() + public void CanUseBatchedInsert_IsTrue_WithUniqueSecondaryIndex_BecauseTheBatchCoreEnforcesIt() { + // Up to 2.1.4 a unique secondary index forced the per-row loop, because the batch core did not enforce it. The + // batch core now validates unique indexes (Table.UniqueKeys), so the batched path stays on — and still rejects a + // duplicate, keeping none of the statement's rows. var db = Open(NewDir("gate_unique")); db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT NOT NULL, score INTEGER)"); TableOf(db).CreateHashIndex("idx_t_name_unique", "name", isUnique: true); + db.ExecuteSQL("INSERT INTO t (id, name, score) VALUES (1, 'a', 1)"); - Assert.False(TableOf(db).CanUseBatchedInsert); + Assert.True(TableOf(db).CanUseBatchedInsert); + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, name, score) VALUES (2, 'b', 2), (3, 'a', 3)")); + Assert.Single(db.ExecuteQuery("SELECT id FROM t")); (db as IDisposable)?.Dispose(); } diff --git a/tests/SharpCoreDB.Tests/PrimaryKeyUpdateTests.cs b/tests/SharpCoreDB.Tests/PrimaryKeyUpdateTests.cs new file mode 100644 index 00000000..495bfdaa --- /dev/null +++ b/tests/SharpCoreDB.Tests/PrimaryKeyUpdateTests.cs @@ -0,0 +1,126 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// An UPDATE that changes a PRIMARY KEY may not move a row onto a key another row holds, and a rejected UPDATE changes +/// nothing — on columnar and page-based storage, in the session and after a reopen. +/// +/// Up to 2.1.4 the new key was never checked (known issue §8): UPDATE t SET id = 1 WHERE id = 3 with a row 1 +/// present was accepted, the primary-key index was re-pointed at the moved row, and the original row 1 became +/// unreachable — silent data loss. Single-file storage enforces no primary-key uniqueness at all yet (§11). +/// +/// +public sealed class PrimaryKeyUpdateTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_PkUpdate_{Guid.NewGuid():N}"); + + public PrimaryKeyUpdateTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public static TheoryData Modes => new() { "columnar", "page-based" }; + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_PrimaryKeyOntoAnExistingKey_IsRejectedAndLosesNoRow(string mode) + { + var (db, _) = Open(mode); + + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET id = 1 WHERE id = 3")); + Assert.Equal([(1, "one"), (3, "three")], Rows(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_PrimaryKeyOntoAnExistingKey_AfterReopen_LosesNoRow(string mode) + { + var (db, path) = Open(mode); + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET id = 1 WHERE id = 3")); + (db as IDisposable)?.Dispose(); + + var reopened = _factory.Create(path, "pw"); + Assert.Equal([(1, "one"), (3, "three")], Rows(reopened)); + (reopened as IDisposable)?.Dispose(); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_PrimaryKeyToAFreeKey_MovesTheRow(string mode) + { + var (db, _) = Open(mode); + + db.ExecuteSQL("UPDATE t SET id = 7 WHERE id = 3"); + + Assert.Equal([(1, "one"), (7, "three")], Rows(db)); + Assert.Equal("three", db.ExecuteQuery("SELECT name FROM t WHERE id = 7").Single()["name"]); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_PrimaryKeyToItsOwnValue_IsAccepted(string mode) + { + var (db, _) = Open(mode); + + db.ExecuteSQL("UPDATE t SET id = 3, name = 'still three' WHERE id = 3"); + + Assert.Equal([(1, "one"), (3, "still three")], Rows(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_PrimaryKeyOfSeveralRowsToOneKey_IsRejectedAndChangesNothing(string mode) + { + var (db, _) = Open(mode); + + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET id = 9 WHERE id > 0")); + Assert.Equal([(1, "one"), (3, "three")], Rows(db)); + } + + private (IDatabase Db, string Path) Open(string mode) + { + var path = Path.Combine(_root, $"db_{Guid.NewGuid():N}"); + var db = _factory.Create(path, "pw"); + var create = "CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT)"; + db.ExecuteSQL(mode == "page-based" ? create + " STORAGE = PAGE_BASED" : create); + db.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, 'one')"); + db.ExecuteSQL("INSERT INTO t (id, name) VALUES (3, 'three')"); + return (db, path); + } + + private static (int Id, string? Name)[] Rows(IDatabase db) => + [.. db.ExecuteQuery("SELECT id, name FROM t") + .Select(r => (Convert.ToInt32(r["id"], CultureInfo.InvariantCulture), r["name"] as string)) + .OrderBy(r => r.Item1)]; +} diff --git a/tests/SharpCoreDB.Tests/UniqueConstraintTests.cs b/tests/SharpCoreDB.Tests/UniqueConstraintTests.cs new file mode 100644 index 00000000..c4e2ff81 --- /dev/null +++ b/tests/SharpCoreDB.Tests/UniqueConstraintTests.cs @@ -0,0 +1,221 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// A single-column UNIQUE constraint — column-level email TEXT UNIQUE or table-level UNIQUE (email) — +/// is enforced on INSERT, multi-row INSERT and UPDATE, on columnar and page-based storage, and after a reopen; NULLs +/// never conflict, as in SQLite. Single-file storage is not covered yet (the skipped test). +/// +/// Up to 2.1.4 the constraint was parsed and stored but never checked (known issue §9): the INSERT-time validation +/// was a stub, a UNIQUE index was checked only on a single-row INSERT into a columnar table, and a multi-row +/// INSERT or an UPDATE was never checked at all. A UNIQUE column now gets an automatic unique index, as in SQLite +/// (sqlite_autoindex_…), and one validator covers every write path. +/// +/// +public sealed class UniqueConstraintTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_Unique_{Guid.NewGuid():N}"); + + public UniqueConstraintTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public static TheoryData Modes => new() { "columnar", "page-based" }; + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_DuplicateIntoUniqueColumn_IsRejected(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'a@x')")); + Assert.Equal([1], Ids(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_NullsIntoUniqueColumn_AreAllowed(string mode) + { + var (db, _) = Open(mode); + + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, NULL)"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, NULL)"); + + Assert.Equal([1, 2], Ids(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void MultiRowInsert_DuplicateWithinTheStatement_KeepsNoRow(string mode) + { + var (db, _) = Open(mode); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x'), (2, 'b@x'), (3, 'a@x')")); + Assert.Empty(Ids(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void MultiRowInsert_DuplicatingAStoredValue_KeepsNoRowOfTheStatement(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'b@x'), (3, 'a@x')")); + Assert.Equal([1], Ids(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_UniqueColumnToAnotherRowsValue_IsRejectedAndChangesNothing(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'b@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET email = 'a@x' WHERE id = 2")); + Assert.Equal("b@x", Email(db, 2)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_UniqueColumnToItsOwnValue_IsAccepted(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + db.ExecuteSQL("UPDATE t SET email = 'a@x' WHERE id = 1"); + + Assert.Equal("a@x", Email(db, 1)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_UniqueColumnToAFreeValue_IsAccepted(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'b@x')"); + + db.ExecuteSQL("UPDATE t SET email = 'c@x' WHERE id = 2"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (3, 'b@x')"); + + Assert.Equal("c@x", Email(db, 2)); + Assert.Equal([1, 2, 3], Ids(db)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_UniqueColumnOnSeveralRowsToOneValue_IsRejectedAndChangesNothing(string mode) + { + var (db, _) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'b@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET email = 'z@x' WHERE id > 0")); + Assert.Equal("a@x", Email(db, 1)); + Assert.Equal("b@x", Email(db, 2)); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_DuplicateIntoUniqueColumn_AfterReopen_IsRejected(string mode) + { + var (db, path) = Open(mode); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + (db as IDisposable)?.Dispose(); + + var reopened = Reopen(mode, path); + Assert.ThrowsAny(() => reopened.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'a@x')")); + Assert.Equal([1], Ids(reopened)); + (reopened as IDisposable)?.Dispose(); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_DuplicateIntoTableLevelUniqueColumn_IsRejected(string mode) + { + var (db, _) = Open(mode, "CREATE TABLE t (id INTEGER PRIMARY KEY, email TEXT, UNIQUE (email))"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'a@x')")); + Assert.Equal([1], Ids(db)); + } + + [Fact(Skip = "Single-file storage does not enforce UNIQUE yet: its persisted ColumnFlags.Unique is also written for " + + "every inline PRIMARY KEY column (\"PRIMARY KEY implies UNIQUE\"), so a reopened file cannot tell a declared " + + "UNIQUE from a column of an EF Core composite key. Needs the column-entry format decision of known issues " + + "§10/§11.")] + public void Insert_DuplicateIntoUniqueColumn_IsRejectedOnSingleFile() + { + var (db, _) = Open("single-file"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'a@x')")); + Assert.Equal([1], Ids(db)); + } + + [Fact] + public void MultiRowInsert_DuplicatingAUniqueIndex_IsRejected() + { + // A UNIQUE index was checked only on a single-row INSERT; the batched multi-row path skipped it. + var (db, _) = Open("columnar", "CREATE TABLE t (id INTEGER PRIMARY KEY, email TEXT)"); + db.ExecuteSQL("CREATE UNIQUE INDEX ux_email ON t (email)"); + db.ExecuteSQL("INSERT INTO t (id, email) VALUES (1, 'a@x')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, email) VALUES (2, 'b@x'), (3, 'a@x')")); + Assert.Equal([1], Ids(db)); + } + + private (IDatabase Db, string Path) Open(string mode, string? ddl = null) + { + var path = Path.Combine(_root, mode == "single-file" ? $"db_{Guid.NewGuid():N}.scdb" : $"db_{Guid.NewGuid():N}"); + var db = Reopen(mode, path); + var create = ddl ?? "CREATE TABLE t (id INTEGER PRIMARY KEY, email TEXT UNIQUE)"; + db.ExecuteSQL(mode == "page-based" ? create + " STORAGE = PAGE_BASED" : create); + return (db, path); + } + + private IDatabase Reopen(string mode, string path) => mode == "single-file" + ? _factory.CreateWithOptions(path, "pw", DatabaseOptions.CreateSingleFileDefault()) + : _factory.Create(path, "pw"); + + private static int[] Ids(IDatabase db) => + [.. db.ExecuteQuery("SELECT id FROM t").Select(r => Convert.ToInt32(r["id"], CultureInfo.InvariantCulture)).Order()]; + + private static string? Email(IDatabase db, int id) => + db.ExecuteQuery($"SELECT email FROM t WHERE id = {id}").Single()["email"] as string; +}