Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **`SqlFunctions.EvaluateFunction` answered SQLite's date functions only for a `DateTime` argument** (`STRFTIME`, `DATE`; anything else was NULL) and threw for `TIME`, `DATETIME`, `JULIANDAY`, `UNIXEPOCH`. They now go through `SqlDateTimeFunctions` with SQLite's argument order and format language; the legacy `(DateTime, .NET format)` shape keeps its behaviour. `SqlFunctions.StrFTime` formatted with the current culture (th-TH printed 2026 as 2569) and is now invariant. Pinned by `SqlFunctionsDateFamilyTests`.
- **Recorded, not fixed (`docs/sharpcoredb-known-issues.md` §8–§11):** an UPDATE that moves a primary key onto an existing key loses a row on directory storage and duplicates it on single-file storage; a column-level `UNIQUE` is enforced on neither mode; single-file storage ignores a declared `COLLATE`; and single-file storage enforces no PRIMARY KEY uniqueness, which cannot be switched on safely until composite keys are modelled — EF Core declares one inline `PRIMARY KEY` per key column and both modes keep only the last. Turning single-file enforcement on was tried and backed out: `CompleteExampleIntegrationTests.CompositeKey_*` showed existing EF Core databases would reject legitimate inserts. §8 and §9 need the write-path performance gate; §10 and §11 need an owner decision on the column-entry format. Suites: core **2360 / 0 failed (5 skipped: the documented single-file cases)**, VectorSearch 255 / 0, EF Core 116 / 0, linq2DB **96 / 0**, regression canaries 51 / 0.

- **An UPDATE that moved a PRIMARY KEY onto an existing key lost a row (directory storage).** `UPDATE t SET id = 1 WHERE id = 3` with a row 1 present was accepted: the primary-key index was re-pointed at the moved row and the original row 1 became unreachable. Every UPDATE path — `Table.UpdateAffectedCount` (SQL UPDATE), `UpdateMultipleCore` (`ExecuteBatchSQL`, the structured batch API) and `UpdateByPrimaryKey` — now validates new key values before any row is written, counting what earlier operations of a batch claim and release; a SET of the key matching more than one row is rejected too. The check starts with a test that allocates nothing and returns at once for an UPDATE that does not touch a key. Pinned by `PrimaryKeyUpdateTests` (columnar and page-based, in the session and after a reopen). Known issue §8; its single-file half stays with §11.
- **A UNIQUE constraint was never enforced; a UNIQUE index only partly.** A column declared `UNIQUE` (or `UNIQUE (col)`) accepted duplicates on every storage mode: the INSERT-time check was a stub. A UNIQUE *index* was checked only by a single-row INSERT into a columnar table — not on page-based tables, not by a multi-row INSERT, not by an UPDATE — and rejected a second NULL as a duplicate. A single-column UNIQUE now gets an automatic unique hash index on directory storage, as SQLite's `sqlite_autoindex_…` (registered at CREATE TABLE and on every open, so existing tables are covered; rows stored earlier never block an open). One validator covers INSERT, both batch critical sections and every UPDATE path; NULLs never conflict; and a table with a unique index keeps the batched — atomic — multi-row INSERT, which it used to give up for a per-row loop that left earlier rows behind. Pinned by `UniqueConstraintTests` (columnar, page-based) and `MultiRowInsertBatchingTests.CanUseBatchedInsert_IsTrue_WithUniqueSecondaryIndex_BecauseTheBatchCoreEnforcesIt`, which replaces the test that pinned the old per-row fallback. Known issue §9. Still open: single-file storage, whose persisted `ColumnFlags.Unique` is also written for every inline PRIMARY KEY column, so a reopened file cannot tell a declared UNIQUE from an EF Core composite key's column (the format decision of §10/§11; enforcing it was tried and backed out when `CompleteExampleIntegrationTests.CompositeKey_*` failed); and a composite `UNIQUE (a, b)` on INSERT. Suites: core **2392 / 0 failed (6 skipped: documented single-file cases)**, VectorSearch 255 / 0, EF Core 116 / 0, linq2DB 96 / 0, regression canaries 51 / 0; write-path gate: GATE PASSED on a NOISY box (WSearch and Defender need elevation), every metric ok, worst best-of-reps ratio 1.22x on raw UPDATE against a 1.50x limit while its median ratio was 0.60x (faster).

## [2.1.4] - 2026-11-13

> The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema
Expand Down
30 changes: 21 additions & 9 deletions docs/sharpcoredb-known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -421,10 +421,14 @@ attribution before merge.

---

## 8. 🔴 OPEN — An UPDATE that moves a PRIMARY KEY onto an existing key loses or duplicates a row
## 8. ✅ FIXED (directory storage) — An UPDATE that moves a PRIMARY KEY onto an existing key lost a row

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/linq2db-adapter-and-scalar-functions`).** Reproduced on both storage
> modes with the statements below; not fixed in this change.
> ✅ **FIXED (2026-11-17, `fix/pk-update-and-unique-constraints`) on directory storage.** Every UPDATE path —
> `Table.UpdateAffectedCount` (SQL UPDATE), `UpdateMultipleCore` (`ExecuteBatchSQL` and the structured batch API) and
> `UpdateByPrimaryKey` — now validates a new key before any row is written: a key another row holds, or a SET of the key
> that matches more than one row, is rejected and nothing changes. Pinned by `PrimaryKeyUpdateTests` (columnar and
> page-based, in the session and after a reopen). The single-file half — a duplicated key, not a lost row — is the same
> composite-key question as §11 and is tracked there.

Found 2026-11-17. Neither storage mode checks the new key of an UPDATE that changes the primary key:

Expand All @@ -444,10 +448,17 @@ SELECT id FROM t;

---

## 9. 🟠 OPEN — A column-level `UNIQUE` constraint is not enforced
## 9. ✅ FIXED (directory storage) — A column-level `UNIQUE` constraint was not enforced

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/linq2db-adapter-and-scalar-functions`).** Reproduced on both storage
> modes; not fixed in this change.
> ✅ **FIXED (2026-11-17, `fix/pk-update-and-unique-constraints`) on directory storage.** A single-column UNIQUE
> (column-level or table-level `UNIQUE (col)`) gets an automatic unique hash index, as SQLite's `sqlite_autoindex_…` —
> registered at CREATE TABLE and on every open, so existing tables are covered. INSERT, multi-row INSERT (now atomic on
> tables with a unique index) and every UPDATE path are checked; NULLs never conflict. On the way: a UNIQUE *index* was
> enforced only by a single-row INSERT into a columnar table, and a second NULL was rejected as a duplicate. Pinned by
> `UniqueConstraintTests` (columnar and page-based). **Still open:** single-file storage (its persisted
> `ColumnFlags.Unique` is also written for every inline PRIMARY KEY column, so a reopened file cannot tell a declared
> UNIQUE from an EF Core composite key's column — the format decision of §10/§11; tried and backed out when
> `CompleteExampleIntegrationTests.CompositeKey_*` failed) and a composite `UNIQUE (a, b)` on INSERT.

Found 2026-11-17. A column declared `UNIQUE` accepts duplicates on both storage modes:

Expand Down Expand Up @@ -489,7 +500,8 @@ on-disk format extension. A `Locale(…)` collation would still need a name fiel
Found 2026-11-17 while fixing single-file constraint enforcement.

- **Single-file storage does not enforce PRIMARY KEY uniqueness.** `INSERT … VALUES (2, 'y'), (1, 'dup')` into a
table already holding id 1 leaves two rows with id 1.
table already holding id 1 leaves two rows with id 1, and an UPDATE that moves a key onto an existing one (§8's
single-file half) duplicates it.
- **Neither mode models a composite key.** EF Core's `EnsureCreated` (`SharpCoreDBDatabaseCreator`) declares one
inline `PRIMARY KEY` per key column; both storage modes keep only the *last* such column as `PrimaryKeyIndex`.
Directory storage then enforces uniqueness on that one column — a false violation as soon as two rows share it
Expand Down Expand Up @@ -519,7 +531,7 @@ single-file cases in `PrimaryKeyUniquenessTests`.
| 5 | SQL validator `@` prefix | 🟡 Low | Keys without `@` prefix | **Fixed** — `NormalizeKey` strips `@` / `:` |
| 6 | INTEGER → Int32 | 🟡 Low | `BIGINT` columns | **Still the default** — opt in with `UseSqliteIntegerAffinity` |
| 7 | Bound text re-parsed as SQL | 🔴 High | Write through `ITable.Insert` instead of SQL text | **Fixed** — one `SqlLiteralScanner` (SQL/SQLite: a doubled quote is one quote, a backslash is data) behind every scanner; `ParseValue` strips one pair and un-collapses doubling; a tuple with too many values is rejected. `ParameterLiteralRoundTripTests` 74/74 |
| 8 | PK-changing UPDATE loses/duplicates a row | 🔴 High | — | **Open** — directory loses the overwritten row, single-file duplicates the key; write-path gate needed (§8) |
| 9 | Column `UNIQUE` not enforced | 🟠 Medium | — | **Open** — both storage modes accept duplicates (§9) |
| 8 | PK-changing UPDATE loses/duplicates a row | 🔴 High | — | **Fixed on directory storage** — every UPDATE path validates the new key first; single-file part tracked in §11 |
| 9 | Column `UNIQUE` not enforced | 🟠 Medium | — | **Fixed on directory storage** — automatic unique index; single-file (format decision, §10/§11) and composite `UNIQUE (a, b)` still open |
| 10 | Single-file ignores `COLLATE` | 🟡 Low | — | **Open, owner decision** — needs a column-entry format extension (§10) |
| 11 | PK uniqueness / composite keys | 🔴 High | — | **Open, owner decision** — single-file enforces no PK uniqueness; composite keys are modelled as their last column (§11) |
12 changes: 6 additions & 6 deletions src/SharpCoreDB/DataStructures/HashIndex.cs
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ internal void Add(object key, long position)
{
// Unique path: atomic check + add under outer lock
var keyBytes = BuildUnsafeKey(normalizedKey);
if (HasUnsafeRowsForKey(keyBytes))
if (normalizedKey is not DBNull && HasUnsafeRowsForKey(keyBytes))
{
throw new InvalidOperationException(
$"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'");
Expand All @@ -122,7 +122,7 @@ internal void Add(object key, long position)
list = [];
_index[normalizedKey] = list;
}
else if (_isUnique && list.Count > 0)
else if (_isUnique && list.Count > 0 && normalizedKey is not DBNull)
{
throw new InvalidOperationException(
$"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'");
Expand Down Expand Up @@ -463,7 +463,7 @@ private void AddBatchKeysLockedCore(object?[] keys, long[] positions)
{
// Unique path: atomic check + add under outer lock.
var keyBytes = BuildUnsafeKey(normalizedKey);
if (HasUnsafeRowsForKey(keyBytes))
if (normalizedKey is not DBNull && HasUnsafeRowsForKey(keyBytes))
{
throw new InvalidOperationException(
$"Duplicate key value '{keys[i]}' violates unique constraint on index '{_columnName}'");
Expand All @@ -479,7 +479,7 @@ private void AddBatchKeysLockedCore(object?[] keys, long[] positions)
list = [];
_index[normalizedKey] = list;
}
else if (_isUnique && list.Count > 0)
else if (_isUnique && list.Count > 0 && normalizedKey is not DBNull)
{
throw new InvalidOperationException(
$"Duplicate key value '{keys[i]}' violates unique constraint on index '{_columnName}'");
Expand Down Expand Up @@ -552,7 +552,7 @@ public void AddBatch(List<Dictionary<string, object>> rows, long[] positions)
{
// Unique path: atomic check + add under outer lock
var keyBytes = BuildUnsafeKey(normalizedKey);
if (HasUnsafeRowsForKey(keyBytes))
if (normalizedKey is not DBNull && HasUnsafeRowsForKey(keyBytes))
{
throw new InvalidOperationException(
$"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'");
Expand All @@ -567,7 +567,7 @@ public void AddBatch(List<Dictionary<string, object>> rows, long[] positions)
list = [];
_index[normalizedKey] = list;
}
else if (_isUnique && list.Count > 0)
else if (_isUnique && list.Count > 0 && normalizedKey is not DBNull)
{
throw new InvalidOperationException(
$"Duplicate key value '{key}' violates unique constraint on index '{_columnName}'");
Expand Down
54 changes: 28 additions & 26 deletions src/SharpCoreDB/DataStructures/Table.CRUD.cs
Original file line number Diff line number Diff line change
Expand Up @@ -98,21 +98,6 @@ public void Insert(Dictionary<string, object> row)
}
}

// ✅ UNIQUE validation (outside lock)
foreach (var uniqueConstraint in this.UniqueConstraints)
{
if (uniqueConstraint.Count == 1) // Single column unique
{
var colName = uniqueConstraint[0];
var colIndex = this.Columns.IndexOf(colName);
if (colIndex >= 0 && row.TryGetValue(colName, out var value) && value != null && value != DBNull.Value)
{
// Check if value already exists (simplified - would need index lookup in real impl)
// For now, just validate non-null for single column unique
}
}
}

// ✅ CHECK constraint validation (outside lock)
for (int i = 0; i < this.Columns.Count; i++)
{
Expand Down Expand Up @@ -189,7 +174,7 @@ public void Insert(Dictionary<string, object> row)
continue;
}

if (!row.TryGetValue(registeredCol, out var value) || value is null)
if (!row.TryGetValue(registeredCol, out var value) || value is null or DBNull)
{
continue;
}
Expand All @@ -202,6 +187,10 @@ public void Insert(Dictionary<string, object> row)
}
}

// UNIQUE columns and unique indexes, on every storage mode (the check above runs for columnar tables
// only and skips an index it has not loaded). Returns at once for a table without unique indexes.
ValidateUniqueValues(row);

// ✅ NEW: Route through storage engine
var engine = GetOrCreateStorageEngine();

Expand Down Expand Up @@ -278,8 +267,8 @@ public void Insert(Dictionary<string, object> row)
/// this table — i.e. when nothing in the table's schema relies on the per-row-only semantics that the
/// batch core does not implement.
/// <para>
/// What the batch core does NOT do: CHECK-constraint evaluation (per-column and table-level) and non-PK
/// unique-index enforcement. PK and NOT NULL validation <em>are</em> implemented on both paths, and the
/// What the batch core does NOT do: CHECK-constraint evaluation (per-column and table-level). PK, NOT NULL and
/// UNIQUE (column constraints and unique indexes) validation <em>are</em> implemented on both paths, and the
/// batch core joins an open engine transaction rather than nesting one
/// (<c>InsertBatchCriticalSection</c>: <c>needsTransaction = !engine.IsInTransaction</c>).
/// </para>
Expand Down Expand Up @@ -307,14 +296,6 @@ internal bool CanUseBatchedInsert
}
}

foreach (var metadata in this.registeredIndexes.Values)
{
if (metadata.IsUnique)
{
return false;
}
}

return _database?.IsBatchUpdateActive != true;
}
}
Expand Down Expand Up @@ -802,6 +783,9 @@ private long[] InsertBatchCriticalSection(object[][] validatedRows, List<byte[]>
// Validate primary keys (requires lock for index access)
ValidateExistingPrimaryKeys(validatedRows);

// UNIQUE columns and unique indexes: up to 2.1.4 the batched path checked neither.
ValidateUniqueValues(validatedRows);

var engine = GetOrCreateStorageEngine();
bool needsTransaction = !engine.IsInTransaction;

Expand Down Expand Up @@ -882,6 +866,9 @@ private long[] InsertBatchCriticalSection(
List<Dictionary<string, object>> validatedRows,
List<byte[]> serializedRows)
{
// UNIQUE columns and unique indexes: up to 2.1.4 the batched path checked neither.
ValidateUniqueValues(validatedRows);

// Validate primary keys (requires lock for index access)
if (this.PrimaryKeyIndex >= 0)
{
Expand Down Expand Up @@ -1778,6 +1765,13 @@ public int UpdateAffectedCount(string? where, Dictionary<string, object> updates
var rows = ResolveUpdateRows(where);
int affected = 0;

// A new PRIMARY KEY or UNIQUE value must not be another row's, and is checked before any row is written.
// Up to 2.1.4 a primary key moved onto an existing key re-pointed the index and lost the original row.
if (rows.Count > 0 && UpdateTouchesKeyColumns(updates))
{
ValidateKeyChanges([(rows.ConvertAll(r => r.Row), updates)]);
}

foreach (var (rowPos, row) in rows)
{
affected++;
Expand Down Expand Up @@ -2510,6 +2504,9 @@ private int UpdateMultipleCore(
// put an atomic per operation into the loop it measures.
long structuredOps = 0;

// Key changes are validated for the whole batch before any operation is written (known issues §8/§9).
ValidateBatchKeyChanges(operations);

foreach (var (where, structuredKeyColumn, structuredKeyValue, structuredKeyLiteral, updates) in operations)
{
// §5.3 session 2 (2026-09-26): the per-operation head of this loop — key resolution, the
Expand Down Expand Up @@ -5326,6 +5323,11 @@ public bool UpdateByPrimaryKey(object key, Dictionary<string, object> updates)
if (row == null)
return false;

if (UpdateTouchesKeyColumns(updates))
{
ValidateKeyChanges([([row], updates)]);
}

// WP13: capture only what index maintenance needs instead of copying the whole row.
Dictionary<string, object>? oldHashKeys = null;
foreach (var kvp in this.hashIndexes) // NOSONAR:S3267 - deliberate: LINQ Select/Where would allocate per point-update on the hot path
Expand Down
Loading
Loading