From f0990bd0df78d63620115e185042a8d8017207cc Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 14:47:55 -0700 Subject: [PATCH 1/9] Document SQL LIKE semantics: wildcards, ESCAPE, case per engine, cost Adds a LIKE Pattern Matching section to the operations-API SQL reference for harper#3936 (v5.4): % and _ semantics, ESCAPE and its 400s, case sensitivity per engine and how auto can serve either, legacy-only character classes, and the per-value linear matching bound. Defines sql.engine and sql.allowFullScan in the fallback contract, lists LIKE among the engine behavior differences, and adds a 5.4 release note. Refs HarperFast/harper#3936 Dispatch-Task: docs-harper-3936 Co-Authored-By: Claude Opus 5.5 --- reference/operations-api/sql.md | 101 +++++++++++++++++++++++++++++--- release-notes/v5-lincoln/5.4.md | 14 +++++ 2 files changed, 107 insertions(+), 8 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index 4667acff2..f8ac5a480 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -12,7 +12,7 @@ title: SQL :::info Harper 5.2 introduces a new SQL engine -As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the same results but with the older full-scan cost — each fallback is logged (`SQL engine v2 fallback: ...`) with the reason, so you have direct visibility into which queries aren't being planned. See [Query Performance](#query-performance) for how to keep your queries on the fast path. +As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the legacy engine's results with the older full-scan cost — each fallback is logged at the `info` level (`SQL engine v2 fallback: ...`) with the reason, so you have direct visibility into which queries aren't being planned. See [Query Performance](#query-performance) for how to keep your queries on the fast path. For your most performance-critical production read paths, the [REST interface](../rest/overview.md) remains the recommended choice — it provides a stable, cacheable, index-driven contract. SQL is well suited to ad-hoc investigation, administrative queries, joins, and reporting. ::: @@ -131,7 +131,7 @@ These map directly to indexed storage operations: - **Equality and `IN` on an indexed attribute** — `WHERE status = 'active'`, `WHERE breed_id IN (1, 2, 3)`. - **Range predicates on an indexed attribute** — `<`, `<=`, `>`, `>=`, and `BETWEEN` (compiled to a bounded index range). - **`OR` of indexed predicates** — `WHERE a = 1 OR b = 2` is served as a union of index scans, provided **every** branch is an index-driving predicate on an indexed attribute. -- **Prefix `LIKE`** — `WHERE name LIKE 'Har%'` compiles to an indexed `starts_with` scan. +- **Prefix `LIKE`** — `WHERE name LIKE 'Har%'` compiles to an indexed `starts_with` scan. The new engine matches it case-sensitively; see [LIKE Pattern Matching](#like-pattern-matching). - **`ORDER BY` on an indexed attribute, combined with an indexed `WHERE`** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. Without an index-driving `WHERE` condition, an `ORDER BY` on a _secondary_ indexed attribute runs on the legacy path — except when sorting on the table's **primary key**, which drives the scan from the primary key's natural index order even with no `WHERE` clause. - **`GROUP BY` on an indexed attribute, combined with an indexed `WHERE`** — aggregates stream group-by-group with O(1) memory per group (memory is bounded by the group count; every matching row is still read). Without an index-driving `WHERE` condition, a `GROUP BY` currently runs on the legacy path. - **`INNER`/`LEFT JOIN` on an indexed join key** — executed as an index-nested-loop join (stream the outer table, probe the inner by index). Keep the join key indexed on the inner (probed) table. @@ -153,7 +153,15 @@ Note that `UNION`, sub-`SELECT`, and `INSERT … SELECT` are not supported by ei ### The fallback contract -The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't. A query that falls back returns exactly what the legacy engine would — **its results don't change, only its performance profile.** Fallbacks are logged (`SQL engine v2 fallback: ...`) with the reason, which is a useful signal for finding queries worth an added index or a reshape. (For queries the new engine _does_ plan, it corrects a few long-standing legacy quirks — see [Behavior changes from the legacy engine](#behavior-changes-from-the-legacy-engine).) +The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't. A query that falls back returns exactly what the legacy engine would, at the legacy engine's cost. Fallbacks are logged (`SQL engine v2 fallback: ...`) with the reason, which is a useful signal for finding queries worth an added index or a reshape. They are logged at the `info` level, below the default `warn`, so set [`logging.level`](../logging/configuration.md#logginglevel) to `info` to see them. (Where the two engines differ, which engine runs a statement decides its results: for queries the new engine _does_ plan, it corrects a few long-standing legacy quirks, and `LIKE` matches differently — see [Behavior changes from the legacy engine](#behavior-changes-from-the-legacy-engine).) + +The `sql.engine` setting in the [configuration file](../configuration/overview.md) selects the mode, and the `HARPER_SQL_ENGINE` environment variable overrides it: + +- `auto` (default) — runs each statement on the new engine when it can, and on the legacy engine otherwise. +- `new` — runs every statement on the new engine. A statement the new engine can't plan is rejected with a 400 error instead of falling back. +- `legacy` — runs every statement on the legacy engine. + +By default (`sql.allowFullScan: false`), the new engine does not run a query that has no index-driving condition, which is one reason `auto` falls back. With `sql.allowFullScan: true`, the new engine runs such a query as a full scan instead of falling back (under `auto`) or rejecting it (under `new`). A full scan on the new engine is not a fallback, so it is not logged. ### Behavior changes from the legacy engine @@ -163,6 +171,7 @@ For queries the new engine plans (the default under `auto`), a handful of non-st - **`MIN`/`MAX` over text columns** return the lexicographically smallest/largest value. The legacy engine produced no result for string `MIN`/`MAX`. - **`NULL` never equals `NULL` in a join key.** Rows whose join key is `NULL` on either side no longer match each other (three-valued logic). The legacy engine incorrectly joined `NULL` to `NULL`. - **`OFFSET` past the end of a result returns no rows.** The legacy engine could still return rows when the `OFFSET` exceeded the number of matches. +- **`LIKE` is case-sensitive, and `[...]` in a pattern is literal text.** The legacy engine matches case-insensitively and reads `[...]` as a character class. Under `auto`, adding or removing an index, or changing the other conditions in a statement, can move it between engines, which changes the rows a `LIKE` matches, including the rows an `UPDATE` or `DELETE` changes. See [LIKE Pattern Matching](#like-pattern-matching). - **A `NULL`-valued expression result is returned as an explicit `null`** rather than omitted from the row. A computed column or `COALESCE` that evaluates to `NULL` — or `UPPER()` — now yields `null` (the legacy engine dropped the key, and `UPPER` of a `NULL` returned the literal string `"NULL"`). These are deliberate corrections toward standard SQL, applied when the new engine serves the query — not fallbacks. If your application depended on a legacy quirk, adjust for the standard behavior. For how `WHERE` compares `NULL` and omitted values, including the v5.4.0 change to ranges compared with `NULL`, see [NULL and Omitted Values](#null-and-omitted-values). @@ -399,11 +408,87 @@ Geospatial data must be stored using the [GeoJSON standard](https://geojson.org/ ## Logical Operators -| Keyword | Description | -| --------- | ------------------------------------------------ | -| `BETWEEN` | Returns values within a given range (inclusive). | -| `IN` | Specifies multiple values in a WHERE clause. | -| `LIKE` | Searches for a pattern. | +| Keyword | Description | +| --------- | -------------------------------------------------------------------------------------------------------------------------------- | +| `BETWEEN` | Returns values within a given range (inclusive). | +| `IN` | Specifies multiple values in a WHERE clause. | +| `LIKE` | Matches a string against a pattern. Case sensitivity depends on the engine; see [LIKE Pattern Matching](#like-pattern-matching). | + +--- + +## LIKE Pattern Matching + +`value LIKE pattern` is true when the whole value matches the pattern, and `NOT LIKE` negates it. The value is matched as a string. The pattern can contain these elements: + +| Pattern element | Matches | +| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| `%` | Any run of characters, including none, and including line breaks. | +| `_` | Exactly one character other than a line break (`\n`, `\r`, U+2028, U+2029). | +| `[abc]`, `[a-z]`, `[^abc]` | Legacy engine only: one character in the set or range, or with `^`, one character not in it. See the list below. | +| Any other character | Itself. `*`, `\`, `^`, `$`, `.`, `?` and `+` have no special meaning on either engine. | + +`_` matches one UTF-16 code unit, so a character outside the Basic Multilingual Plane, such as most emoji, takes two: `__`. + +```sql +SELECT * FROM dev.dog WHERE dog_name LIKE 'Pen%' -- starts with "Pen" +SELECT * FROM dev.dog WHERE dog_name LIKE '%y' -- ends with "y" +SELECT * FROM dev.dog WHERE dog_name LIKE '_ack' -- "Jack", "Mack", ... +``` + +### Matching `%` and `_` Literally With ESCAPE + + Add `ESCAPE 'c'` to name an escape character. In the pattern, the escape character makes the character after it literal, and writing it twice matches the escape character itself: + +```sql +SELECT * FROM dev.addr WHERE ip LIKE 't0!%' ESCAPE '!' -- exactly "t0%" +SELECT * FROM dev.product WHERE label LIKE '50!%%' ESCAPE '!' -- starts with "50%" +SELECT * FROM dev.addr WHERE ip LIKE 't0!!%' ESCAPE '!' -- starts with "t0!" +``` + +Without `ESCAPE`, a pattern has no escape character: `ip LIKE 't0!%'` matches `t0!` followed by anything. `ESCAPE` works in `SELECT`, `UPDATE` and `DELETE`, on both engines. + +These are rejected with a 400 error, and no rows are changed: + +- A pattern that ends in the escape character, such as `'t0!' ESCAPE '!'`. +- An `ESCAPE` value that is empty (`''`) or longer than one character (`'!!'`, or a character such as an emoji that takes two UTF-16 code units). +- An `ESCAPE` value that is not a string, such as `ESCAPE 1`. + +Before v5.4.0, `ESCAPE` was ignored, so `ip LIKE 't0!%' ESCAPE '!'` matched every value starting with `t0!`, and an `UPDATE` or `DELETE` using it changed those rows. + +### Case Sensitivity Depends on the Engine + +| Engine that runs the statement | `LIKE` is | +| ------------------------------------------------------------------------------------------------------- | --------------------------------------------- | +| New engine: `sql.engine: new`, and statements that `auto` runs on the new engine | Case-sensitive: `'t0%'` does not match `T0b`. | +| Legacy engine: `sql.engine: legacy`, and statements that `auto` [falls back](#the-fallback-contract) on | Case-insensitive: `'t0%'` matches `T0b`. | + +Under the default `sql.engine: auto`, the same pattern can be either. For example, with `ip` indexed and `note` not indexed, and no other condition in the query: + +- `ip LIKE 't0%'` is a prefix pattern on an indexed attribute, so the new engine serves it from the index, and it is case-sensitive. +- `ip LIKE '%T0B'` is not a prefix pattern, so the index can't drive it. `auto` runs it on the legacy engine, and it matches `T0b`. +- `note LIKE 't0%'` has no index to drive it, so `auto` runs it on the legacy engine, and it is case-insensitive. With `sql.allowFullScan: true`, the new engine runs it as a full scan instead, and it is case-sensitive. + +This applies to `UPDATE` and `DELETE` as well: adding an index on `note` moves `DELETE FROM dev.addr WHERE note LIKE 't0%'` to the new engine, and it stops deleting `T0b`. A statement that falls back is logged, which tells you which engine ran it (see [The fallback contract](#the-fallback-contract)). + +To match regardless of case on both engines, lower-case the value: `LOWER(note) LIKE 't0%'`. `LOWER()` uses JavaScript's lower-casing, which treats a few non-ASCII characters differently from the legacy engine's case-insensitive matching. The index can't serve an expression, so combine it with an indexed condition using `AND`. Otherwise `auto` runs it on the legacy engine as a full scan, and `sql.engine: new` rejects it unless `sql.allowFullScan` is `true`. To keep a case-insensitive prefix search on the index, store a lower-cased copy of the value in its own indexed attribute and query that. + +### Character Classes on the Legacy Engine + +The legacy engine also reads `[...]` in a pattern as a character class, matched without regard to case like the rest of the pattern: + +- `[abc]` matches one of `a`, `b` or `c`. +- `[a-z]` matches one character in the range. +- `[^abc]` matches one character that is not in the set. +- A range written backwards, such as `[z-a]`, is rejected with a 400 error. +- A `[` with no closing `]` is a literal. + +The new engine has no character classes: `[` and `]` match themselves. Under `auto`, a pattern such as `'[a]%'` is a class on the legacy engine and literal text on the new engine, so avoid `[` in patterns unless you know which engine runs the statement. + +### Matching Cost + + Testing one value takes time at most proportional to the value's length times the pattern's length, for any mix of `%` and `_`. No pattern makes a single value expensive to test. Before v5.4.0, a pattern such as `'%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z'` against a long value could block a thread. + +The total cost of a `LIKE` is still the number of rows it tests. On an indexed attribute, the index serves a prefix pattern, which is literal text followed by one `%` (`'abc%'`, or `'50!%%' ESCAPE '!'` for values starting with `50%`), and a pattern with no wildcards, which is an equality test. Any other `LIKE` filters the rows that the rest of the query selects, so with no indexed condition it tests every row in the table. Under `auto`, that query runs on the legacy engine. Under `sql.engine: new` with `sql.allowFullScan` off (the default), it is rejected with a 400 error instead of scanning. See [Query Performance](#query-performance). --- diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index fa3976d26..2de3e5454 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -51,6 +51,20 @@ This change is not released yet. This change is not released yet. +## SQL + +### LIKE Honors ESCAPE and Matches in Linear Time + +[Match SQL LIKE in linear time and honor LIKE … ESCAPE, in both SQL engines](https://github.com/HarperFast/harper/pull/3936) fixes two `LIKE` problems on both SQL engines. `LIKE pattern ESCAPE 'c'` now makes the character after `c` literal in `SELECT`, `UPDATE` and `DELETE`. Before, `ESCAPE` was ignored, so `ip LIKE 't0!%' ESCAPE '!'` selected, updated or deleted every value starting with `t0!` instead of only `t0%`. Matching also no longer backtracks: testing a value takes time at most proportional to its length times the pattern's length, where a pattern such as `'%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z'` against a long value could block a thread before. + +**Upgrade impact:** + +- On the new engine, `*` is now a literal (it acted as a regular-expression quantifier), and `%` now matches line breaks, as it already did on the legacy engine. +- On the legacy engine, `\`, `^` and `$` are now literals. They acted as regular-expression syntax, so `'a\d'` matched `a5`. +- On both engines, these are rejected with a 400 error: a pattern ending in the escape character, and an `ESCAPE` value that is not exactly one character or not a string. On the legacy engine, so is a character-class range written backwards, such as `[z-a]`. + +`LIKE` is still case-sensitive on the new engine and case-insensitive on the legacy engine. Under the default `sql.engine: auto`, the case behavior depends on which engine runs the statement, so adding or removing an index can change the rows a `LIKE` in an `UPDATE` or `DELETE` matches. See [LIKE Pattern Matching](/reference/v5/operations-api/sql#like-pattern-matching). + ## Replication ### Sharding From 2a44ed270bc5fad0464514a267f976b7b929854c Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 14:58:22 -0700 Subject: [PATCH 2/9] Qualify LIKE engine routing, legacy error timing and the cost bound - Name what keeps an unindexed LIKE on the new engine under auto: an indexed AND condition, ORDER BY on the primary key, allowFullScan. - Say the legacy engine checks a pattern when it tests a row. - Drop the claim that no pattern is expensive; both lengths multiply. - Scope the full-scan fallback list to auto with allowFullScan off, and note that pairing a suffix LIKE with an indexed condition makes it case-sensitive. Dispatch-Task: docs-harper-3936 Co-Authored-By: Claude Opus 5.5 --- reference/operations-api/sql.md | 21 +++++++++++---------- release-notes/v5-lincoln/5.4.md | 4 ++-- 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index f8ac5a480..99847f7fa 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -12,7 +12,7 @@ title: SQL :::info Harper 5.2 introduces a new SQL engine -As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the legacy engine's results with the older full-scan cost — each fallback is logged at the `info` level (`SQL engine v2 fallback: ...`) with the reason, so you have direct visibility into which queries aren't being planned. See [Query Performance](#query-performance) for how to keep your queries on the fast path. +As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the legacy engine's results (which [differ in a few behaviors](#behavior-changes-from-the-legacy-engine), such as `LIKE` case sensitivity) with the older full-scan cost — each fallback is logged at the `info` level (`SQL engine v2 fallback: ...`) with the reason, so you have direct visibility into which queries aren't being planned. See [Query Performance](#query-performance) for how to keep your queries on the fast path. For your most performance-critical production read paths, the [REST interface](../rest/overview.md) remains the recommended choice — it provides a stable, cacheable, index-driven contract. SQL is well suited to ad-hoc investigation, administrative queries, joins, and reporting. ::: @@ -139,11 +139,11 @@ These map directly to indexed storage operations: ### Queries that fall back to a full scan (avoid on large tables) -These can't be served from an index. They still return correct results, but the engine falls back to the legacy full-scan path, so cost and memory grow with table size: +These can't be served from an index. Under the default `auto` mode with `sql.allowFullScan` off, the engine falls back to the legacy full-scan path, so cost and memory grow with table size. Under `sql.engine: new` they are rejected with a 400 error instead, and with `sql.allowFullScan: true` the new engine scans the table itself (see [The fallback contract](#the-fallback-contract)): - **Filters on un-indexed attributes** — any equality, range, or `IN` on an attribute with no index, _when it's the only predicate_. Combined via `AND` with an indexed predicate, it's applied as a cheap residual filter after the index narrows the scan, and the query stays index-served overall — it's only a standalone un-indexed filter (or one joined by `OR`) that forces a full scan. - **`OR` where a branch isn't index-driving** — e.g. `WHERE a = 1 OR b = 2` when `b` is un-indexed, or `WHERE a = 1 OR name LIKE '%x%'`. A single un-indexed (or non-index-comparator) branch taints the whole disjunction and forces a scan. When **every** branch is an indexed, index-driving predicate, the `OR` stays on the fast path (see above). -- **Suffix / substring `LIKE`** — `LIKE '%x'` and `LIKE '%x%'` (only prefix `LIKE 'x%'` is index-served). A suffix/contains `LIKE` combined (via `AND`) with an indexed predicate is applied as a cheap residual filter on the indexed result, so pair it with an indexed condition. +- **Suffix / substring `LIKE`** — `LIKE '%x'` and `LIKE '%x%'` (only prefix `LIKE 'x%'` is index-served). A suffix/contains `LIKE` combined (via `AND`) with an indexed predicate is applied as a cheap residual filter on the indexed result, so pair it with an indexed condition. That runs the statement on the new engine, where `LIKE` is [case-sensitive](#case-sensitivity-depends-on-the-engine). - **`!=` / `<>` and `NOT` negations** — matching "everything except" a value can't use an index. - **`ORDER BY` with no indexed filter to drive it** — a table-wide ordered scan on an un-indexed sort key. - **`SEARCH_JSON`** — evaluates JSONata over each candidate row's nested JSON; it is not index-backed. Narrow the candidate set with an indexed `WHERE` condition first. @@ -447,7 +447,7 @@ SELECT * FROM dev.addr WHERE ip LIKE 't0!!%' ESCAPE '!' -- starts with " Without `ESCAPE`, a pattern has no escape character: `ip LIKE 't0!%'` matches `t0!` followed by anything. `ESCAPE` works in `SELECT`, `UPDATE` and `DELETE`, on both engines. -These are rejected with a 400 error, and no rows are changed: +These are rejected with a 400 error, and no rows are changed. Under `auto` and `new`, a pattern written in the statement is checked before any row is read. Under `sql.engine: legacy`, it is checked when a row is tested against it, so a statement that tests no rows does not fail. - A pattern that ends in the escape character, such as `'t0!' ESCAPE '!'`. - An `ESCAPE` value that is empty (`''`) or longer than one character (`'!!'`, or a character such as an emoji that takes two UTF-16 code units). @@ -462,15 +462,16 @@ Before v5.4.0, `ESCAPE` was ignored, so `ip LIKE 't0!%' ESCAPE '!'` matched ever | New engine: `sql.engine: new`, and statements that `auto` runs on the new engine | Case-sensitive: `'t0%'` does not match `T0b`. | | Legacy engine: `sql.engine: legacy`, and statements that `auto` [falls back](#the-fallback-contract) on | Case-insensitive: `'t0%'` matches `T0b`. | -Under the default `sql.engine: auto`, the same pattern can be either. For example, with `ip` indexed and `note` not indexed, and no other condition in the query: +Under the default `sql.engine: auto`, the same pattern can be either, and the whole statement decides which. For example, with `ip` indexed and `note` not indexed: - `ip LIKE 't0%'` is a prefix pattern on an indexed attribute, so the new engine serves it from the index, and it is case-sensitive. - `ip LIKE '%T0B'` is not a prefix pattern, so the index can't drive it. `auto` runs it on the legacy engine, and it matches `T0b`. -- `note LIKE 't0%'` has no index to drive it, so `auto` runs it on the legacy engine, and it is case-insensitive. With `sql.allowFullScan: true`, the new engine runs it as a full scan instead, and it is case-sensitive. +- `note LIKE 't0%'` alone has no index to drive it, so `auto` runs it on the legacy engine, and it is case-insensitive. +- `note LIKE 't0%'` is case-sensitive when something else keeps the statement on the new engine: an indexed condition joined with `AND`, an `ORDER BY` on the primary key, or `sql.allowFullScan: true`. This applies to `UPDATE` and `DELETE` as well: adding an index on `note` moves `DELETE FROM dev.addr WHERE note LIKE 't0%'` to the new engine, and it stops deleting `T0b`. A statement that falls back is logged, which tells you which engine ran it (see [The fallback contract](#the-fallback-contract)). -To match regardless of case on both engines, lower-case the value: `LOWER(note) LIKE 't0%'`. `LOWER()` uses JavaScript's lower-casing, which treats a few non-ASCII characters differently from the legacy engine's case-insensitive matching. The index can't serve an expression, so combine it with an indexed condition using `AND`. Otherwise `auto` runs it on the legacy engine as a full scan, and `sql.engine: new` rejects it unless `sql.allowFullScan` is `true`. To keep a case-insensitive prefix search on the index, store a lower-cased copy of the value in its own indexed attribute and query that. +To match regardless of case on both engines, lower-case the value: `LOWER(note) LIKE 't0%'`. `LOWER()` uses JavaScript's lower-casing, which treats a few non-ASCII characters differently from the legacy engine's case-insensitive matching. The index can't serve an expression, so combine it with an indexed condition using `AND`. Otherwise it is a full scan, which `auto` runs on the legacy engine and `sql.engine: new` rejects unless `sql.allowFullScan` is `true`. To keep a case-insensitive prefix search on the index, store a lower-cased copy of the value in its own indexed attribute and query that. ### Character Classes on the Legacy Engine @@ -479,16 +480,16 @@ The legacy engine also reads `[...]` in a pattern as a character class, matched - `[abc]` matches one of `a`, `b` or `c`. - `[a-z]` matches one character in the range. - `[^abc]` matches one character that is not in the set. -- A range written backwards, such as `[z-a]`, is rejected with a 400 error. +- A range written backwards, such as `[z-a]`, is rejected with a 400 error when a row is tested against it. - A `[` with no closing `]` is a literal. The new engine has no character classes: `[` and `]` match themselves. Under `auto`, a pattern such as `'[a]%'` is a class on the legacy engine and literal text on the new engine, so avoid `[` in patterns unless you know which engine runs the statement. ### Matching Cost - Testing one value takes time at most proportional to the value's length times the pattern's length, for any mix of `%` and `_`. No pattern makes a single value expensive to test. Before v5.4.0, a pattern such as `'%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z'` against a long value could block a thread. + Testing one value takes time at most proportional to the value's length times the pattern's length, for any mix of `%` and `_`. Before v5.4.0, a short pattern such as `'%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z'` against a long value could block a thread. Both lengths still multiply, so a pattern thousands of characters long tested against values millions of characters long is slow; limit pattern length if users supply patterns. -The total cost of a `LIKE` is still the number of rows it tests. On an indexed attribute, the index serves a prefix pattern, which is literal text followed by one `%` (`'abc%'`, or `'50!%%' ESCAPE '!'` for values starting with `50%`), and a pattern with no wildcards, which is an equality test. Any other `LIKE` filters the rows that the rest of the query selects, so with no indexed condition it tests every row in the table. Under `auto`, that query runs on the legacy engine. Under `sql.engine: new` with `sql.allowFullScan` off (the default), it is rejected with a 400 error instead of scanning. See [Query Performance](#query-performance). +The total cost of a `LIKE` is still the number of rows it tests. On an indexed attribute, the index serves a prefix pattern, which is literal text followed by one `%` (`'abc%'`, or `'50!%%' ESCAPE '!'` for values starting with `50%`), and a pattern with no wildcards, which is an equality test. Any other `LIKE` filters the rows that the rest of the query selects, so with no indexed condition it tests every row in the table. Under `auto` with `sql.allowFullScan` off, such a query usually runs on the legacy engine (an `ORDER BY` on the primary key keeps it on the new engine). Under `sql.engine: new` with `sql.allowFullScan` off (the default), it is rejected with a 400 error instead of scanning. See [Query Performance](#query-performance). --- diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 2de3e5454..c13c498a3 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -53,7 +53,7 @@ This change is not released yet. ## SQL -### LIKE Honors ESCAPE and Matches in Linear Time +### LIKE Honors ESCAPE and No Longer Backtracks [Match SQL LIKE in linear time and honor LIKE … ESCAPE, in both SQL engines](https://github.com/HarperFast/harper/pull/3936) fixes two `LIKE` problems on both SQL engines. `LIKE pattern ESCAPE 'c'` now makes the character after `c` literal in `SELECT`, `UPDATE` and `DELETE`. Before, `ESCAPE` was ignored, so `ip LIKE 't0!%' ESCAPE '!'` selected, updated or deleted every value starting with `t0!` instead of only `t0%`. Matching also no longer backtracks: testing a value takes time at most proportional to its length times the pattern's length, where a pattern such as `'%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z'` against a long value could block a thread before. @@ -61,7 +61,7 @@ This change is not released yet. - On the new engine, `*` is now a literal (it acted as a regular-expression quantifier), and `%` now matches line breaks, as it already did on the legacy engine. - On the legacy engine, `\`, `^` and `$` are now literals. They acted as regular-expression syntax, so `'a\d'` matched `a5`. -- On both engines, these are rejected with a 400 error: a pattern ending in the escape character, and an `ESCAPE` value that is not exactly one character or not a string. On the legacy engine, so is a character-class range written backwards, such as `[z-a]`. +- On both engines, these are rejected with a 400 error: a pattern ending in the escape character, and an `ESCAPE` value that is not exactly one character or not a string. On the legacy engine, so is a character-class range written backwards, such as `[z-a]`. The legacy engine checks a pattern when it tests a row against it. `LIKE` is still case-sensitive on the new engine and case-insensitive on the legacy engine. Under the default `sql.engine: auto`, the case behavior depends on which engine runs the statement, so adding or removing an index can change the rows a `LIKE` in an `UPDATE` or `DELETE` matches. See [LIKE Pattern Matching](/reference/v5/operations-api/sql#like-pattern-matching). From d2ab45133c142bbac1f15d4bb0a08951bc2d6874 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 15:05:15 -0700 Subject: [PATCH 3/9] State LIKE routing generally instead of listing exceptions Point the scan-routing sentences at the fallback contract rather than enumerating which statements fall back, scan on the new engine, or get a 400, and scope eager pattern checks to statements the new engine plans. Dispatch-Task: docs-harper-3936 Co-Authored-By: Claude Opus 5.5 --- reference/operations-api/sql.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index 99847f7fa..e911e15b4 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -111,7 +111,7 @@ Use `IS NULL` and `IS NOT NULL` to test for `NULL`. REST and Resource API condit ## Query Performance -Harper 5.2's SQL engine maps SQL clauses onto Harper's native indexed storage. When a query can be planned against an index, it runs as a streaming, index-driven operation with memory bounded by the size of the result — not the size of the table. When it can't, the engine falls back to the legacy path, which fetches candidate rows and evaluates the query in memory, with cost and memory growing with the number of rows scanned. +Harper 5.2's SQL engine maps SQL clauses onto Harper's native indexed storage. When a query can be planned against an index, it runs as a streaming, index-driven operation with memory bounded by the size of the result — not the size of the table. When it can't, the engine falls back to the legacy path, which fetches candidate rows and evaluates the query in memory, with cost and memory growing with the number of rows scanned. This section describes the default `auto` mode; see [The fallback contract](#the-fallback-contract) for `sql.engine: new` and `sql.allowFullScan`. On any non-trivial table the difference between these two paths is large. @@ -139,7 +139,7 @@ These map directly to indexed storage operations: ### Queries that fall back to a full scan (avoid on large tables) -These can't be served from an index. Under the default `auto` mode with `sql.allowFullScan` off, the engine falls back to the legacy full-scan path, so cost and memory grow with table size. Under `sql.engine: new` they are rejected with a 400 error instead, and with `sql.allowFullScan: true` the new engine scans the table itself (see [The fallback contract](#the-fallback-contract)): +These can't be served from an index. Under the default `auto` mode with `sql.allowFullScan` off, the engine falls back to the legacy full-scan path, so cost and memory grow with table size. Under `sql.engine: new`, a statement the new engine can't plan is rejected with a 400 error instead, and `sql.allowFullScan: true` lets the new engine scan the table for a filter it can't serve from an index (see [The fallback contract](#the-fallback-contract)): - **Filters on un-indexed attributes** — any equality, range, or `IN` on an attribute with no index, _when it's the only predicate_. Combined via `AND` with an indexed predicate, it's applied as a cheap residual filter after the index narrows the scan, and the query stays index-served overall — it's only a standalone un-indexed filter (or one joined by `OR`) that forces a full scan. - **`OR` where a branch isn't index-driving** — e.g. `WHERE a = 1 OR b = 2` when `b` is un-indexed, or `WHERE a = 1 OR name LIKE '%x%'`. A single un-indexed (or non-index-comparator) branch taints the whole disjunction and forces a scan. When **every** branch is an indexed, index-driving predicate, the `OR` stays on the fast path (see above). @@ -447,7 +447,7 @@ SELECT * FROM dev.addr WHERE ip LIKE 't0!!%' ESCAPE '!' -- starts with " Without `ESCAPE`, a pattern has no escape character: `ip LIKE 't0!%'` matches `t0!` followed by anything. `ESCAPE` works in `SELECT`, `UPDATE` and `DELETE`, on both engines. -These are rejected with a 400 error, and no rows are changed. Under `auto` and `new`, a pattern written in the statement is checked before any row is read. Under `sql.engine: legacy`, it is checked when a row is tested against it, so a statement that tests no rows does not fail. +These are rejected with a 400 error, and no rows are changed. When the new engine plans the statement, a pattern written in it is checked before any row is read. When the legacy engine runs it, the pattern is checked when a row is tested against it, so a statement that tests no rows does not fail. - A pattern that ends in the escape character, such as `'t0!' ESCAPE '!'`. - An `ESCAPE` value that is empty (`''`) or longer than one character (`'!!'`, or a character such as an emoji that takes two UTF-16 code units). @@ -471,7 +471,7 @@ Under the default `sql.engine: auto`, the same pattern can be either, and the wh This applies to `UPDATE` and `DELETE` as well: adding an index on `note` moves `DELETE FROM dev.addr WHERE note LIKE 't0%'` to the new engine, and it stops deleting `T0b`. A statement that falls back is logged, which tells you which engine ran it (see [The fallback contract](#the-fallback-contract)). -To match regardless of case on both engines, lower-case the value: `LOWER(note) LIKE 't0%'`. `LOWER()` uses JavaScript's lower-casing, which treats a few non-ASCII characters differently from the legacy engine's case-insensitive matching. The index can't serve an expression, so combine it with an indexed condition using `AND`. Otherwise it is a full scan, which `auto` runs on the legacy engine and `sql.engine: new` rejects unless `sql.allowFullScan` is `true`. To keep a case-insensitive prefix search on the index, store a lower-cased copy of the value in its own indexed attribute and query that. +To match regardless of case on both engines, lower-case the value: `LOWER(note) LIKE 't0%'`. `LOWER()` uses JavaScript's lower-casing, which treats a few non-ASCII characters differently from the legacy engine's case-insensitive matching. The index can't serve an expression, so combine it with an indexed condition using `AND`. Otherwise the statement is a full scan of the table. To keep a case-insensitive prefix search on the index, store a lower-cased copy of the value in its own indexed attribute and query that. ### Character Classes on the Legacy Engine @@ -489,7 +489,7 @@ The new engine has no character classes: `[` and `]` match themselves. Under `au Testing one value takes time at most proportional to the value's length times the pattern's length, for any mix of `%` and `_`. Before v5.4.0, a short pattern such as `'%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z'` against a long value could block a thread. Both lengths still multiply, so a pattern thousands of characters long tested against values millions of characters long is slow; limit pattern length if users supply patterns. -The total cost of a `LIKE` is still the number of rows it tests. On an indexed attribute, the index serves a prefix pattern, which is literal text followed by one `%` (`'abc%'`, or `'50!%%' ESCAPE '!'` for values starting with `50%`), and a pattern with no wildcards, which is an equality test. Any other `LIKE` filters the rows that the rest of the query selects, so with no indexed condition it tests every row in the table. Under `auto` with `sql.allowFullScan` off, such a query usually runs on the legacy engine (an `ORDER BY` on the primary key keeps it on the new engine). Under `sql.engine: new` with `sql.allowFullScan` off (the default), it is rejected with a 400 error instead of scanning. See [Query Performance](#query-performance). +The total cost of a `LIKE` is still the number of rows it tests. On an indexed attribute, the index serves a prefix pattern, which is literal text followed by one `%` (`'abc%'`, or `'50!%%' ESCAPE '!'` for values starting with `50%`), and a pattern with no wildcards, which is an equality test. Any other `LIKE` filters the rows that the rest of the query selects, so with no indexed condition it tests every row in the table. Whether such a statement falls back to the legacy engine, scans on the new engine, or is rejected with a 400 error depends on `sql.engine`, `sql.allowFullScan` and the rest of the statement; see [The fallback contract](#the-fallback-contract) and [Query Performance](#query-performance). --- From 13308072aff34eea7ba0648f10c84ce6311bd7f2 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 15:07:43 -0700 Subject: [PATCH 4/9] Name the primary-key ORDER BY exception in the fallback contract Dispatch-Task: docs-harper-3936 Co-Authored-By: Claude Opus 5.5 --- reference/operations-api/sql.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index e911e15b4..337016590 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -147,7 +147,7 @@ These can't be served from an index. Under the default `auto` mode with `sql.all - **`!=` / `<>` and `NOT` negations** — matching "everything except" a value can't use an index. - **`ORDER BY` with no indexed filter to drive it** — a table-wide ordered scan on an un-indexed sort key. - **`SEARCH_JSON`** — evaluates JSONata over each candidate row's nested JSON; it is not index-backed. Narrow the candidate set with an indexed `WHERE` condition first. -- **`RIGHT`/`FULL OUTER JOIN`, no-`WHERE` `UPDATE`/`DELETE`, `COUNT(DISTINCT ...)`** — the new engine doesn't plan these yet, so they run on the legacy engine (correct results, legacy cost). +- **`RIGHT`/`FULL OUTER JOIN`, no-`WHERE` `UPDATE`/`DELETE`, `COUNT(DISTINCT ...)`** — the new engine doesn't plan these yet, so they run on the legacy engine (legacy results and cost). Note that `UNION`, sub-`SELECT`, and `INSERT … SELECT` are not supported by either engine — see the [Features Matrix](#features-matrix). @@ -161,7 +161,7 @@ The `sql.engine` setting in the [configuration file](../configuration/overview.m - `new` — runs every statement on the new engine. A statement the new engine can't plan is rejected with a 400 error instead of falling back. - `legacy` — runs every statement on the legacy engine. -By default (`sql.allowFullScan: false`), the new engine does not run a query that has no index-driving condition, which is one reason `auto` falls back. With `sql.allowFullScan: true`, the new engine runs such a query as a full scan instead of falling back (under `auto`) or rejecting it (under `new`). A full scan on the new engine is not a fallback, so it is not logged. +By default (`sql.allowFullScan: false`), the new engine does not run a query that has neither an index-driving condition nor an `ORDER BY` on the primary key, which is one reason `auto` falls back. With `sql.allowFullScan: true`, the new engine runs such a query as a full scan instead of falling back (under `auto`) or rejecting it (under `new`). A full scan on the new engine is not a fallback, so it is not logged. ### Behavior changes from the legacy engine From 0bd7d4e072d56cd829832e8bd1bfaa025ae876e8 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 2 Nov 2026 15:20:14 -0700 Subject: [PATCH 5/9] Put the ESCAPE error list right after its lead-in; split the pattern-length advice Dispatch-Task: docs-harper-3936 Co-Authored-By: Claude Opus 5.5 --- reference/operations-api/sql.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index 337016590..6023225f6 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -447,12 +447,14 @@ SELECT * FROM dev.addr WHERE ip LIKE 't0!!%' ESCAPE '!' -- starts with " Without `ESCAPE`, a pattern has no escape character: `ip LIKE 't0!%'` matches `t0!` followed by anything. `ESCAPE` works in `SELECT`, `UPDATE` and `DELETE`, on both engines. -These are rejected with a 400 error, and no rows are changed. When the new engine plans the statement, a pattern written in it is checked before any row is read. When the legacy engine runs it, the pattern is checked when a row is tested against it, so a statement that tests no rows does not fail. +These are rejected with a 400 error, and no rows are changed: - A pattern that ends in the escape character, such as `'t0!' ESCAPE '!'`. - An `ESCAPE` value that is empty (`''`) or longer than one character (`'!!'`, or a character such as an emoji that takes two UTF-16 code units). - An `ESCAPE` value that is not a string, such as `ESCAPE 1`. +When the new engine plans the statement, a pattern written in it is checked before any row is read. When the legacy engine runs it, the pattern is checked when a row is tested against it, so a statement that tests no rows does not fail. + Before v5.4.0, `ESCAPE` was ignored, so `ip LIKE 't0!%' ESCAPE '!'` matched every value starting with `t0!`, and an `UPDATE` or `DELETE` using it changed those rows. ### Case Sensitivity Depends on the Engine @@ -487,7 +489,7 @@ The new engine has no character classes: `[` and `]` match themselves. Under `au ### Matching Cost - Testing one value takes time at most proportional to the value's length times the pattern's length, for any mix of `%` and `_`. Before v5.4.0, a short pattern such as `'%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z'` against a long value could block a thread. Both lengths still multiply, so a pattern thousands of characters long tested against values millions of characters long is slow; limit pattern length if users supply patterns. + Testing one value takes time at most proportional to the value's length times the pattern's length, for any mix of `%` and `_`. Before v5.4.0, a short pattern such as `'%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z'` against a long value could block a thread. Both lengths still multiply, so a pattern thousands of characters long tested against values millions of characters long is slow. If users supply patterns, limit their length. The total cost of a `LIKE` is still the number of rows it tests. On an indexed attribute, the index serves a prefix pattern, which is literal text followed by one `%` (`'abc%'`, or `'50!%%' ESCAPE '!'` for values starting with `50%`), and a pattern with no wildcards, which is an equality test. Any other `LIKE` filters the rows that the rest of the query selects, so with no indexed condition it tests every row in the table. Whether such a statement falls back to the legacy engine, scans on the new engine, or is rejected with a 400 error depends on `sql.engine`, `sql.allowFullScan` and the rest of the statement; see [The fallback contract](#the-fallback-contract) and [Query Performance](#query-performance). From d3cee1b3e153eabb77f4437d7ad8678f1efd0d9f Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 4 Nov 2026 06:11:44 -0700 Subject: [PATCH 6/9] Document final SQL LIKE rules for harper#3936 - Document that _ matches any one character including line breaks on both SQL engines in v5.4.0. - Document the 10,000-character LIKE segment limit and its 400 error shape on both engines. - Document the legacy-engine 400 error when a pattern with more than 64 _ wildcards uses a character from every reserved noncharacter and Private Use Area pair. - Clarify escaped [ on the legacy engine and literal pattern checking under auto. Dispatch-Task: documentation-786-like-rules-update Co-Authored-By: Gemini --- reference/operations-api/sql.md | 28 +++++++++++++++++++++++----- release-notes/v5-lincoln/5.4.md | 3 ++- 2 files changed, 25 insertions(+), 6 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index 6023225f6..bf7cdfcb8 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -422,12 +422,12 @@ Geospatial data must be stored using the [GeoJSON standard](https://geojson.org/ | Pattern element | Matches | | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -| `%` | Any run of characters, including none, and including line breaks. | -| `_` | Exactly one character other than a line break (`\n`, `\r`, U+2028, U+2029). | +| `%` | Any run of characters, including none, and including line breaks (`\n`, `\r`, U+2028, U+2029). | +| `_` | Exactly one character, including a line break (`\n`, `\r`, U+2028, U+2029). | | `[abc]`, `[a-z]`, `[^abc]` | Legacy engine only: one character in the set or range, or with `^`, one character not in it. See the list below. | | Any other character | Itself. `*`, `\`, `^`, `$`, `.`, `?` and `+` have no special meaning on either engine. | -`_` matches one UTF-16 code unit, so a character outside the Basic Multilingual Plane, such as most emoji, takes two: `__`. +`_` matches one UTF-16 code unit, so a character outside the Basic Multilingual Plane, such as most emoji, takes two: `__`. Before v5.4.0, `_` did not match line breaks on either engine, and `%` did not match line breaks on the new engine. ```sql SELECT * FROM dev.dog WHERE dog_name LIKE 'Pen%' -- starts with "Pen" @@ -453,7 +453,7 @@ These are rejected with a 400 error, and no rows are changed: - An `ESCAPE` value that is empty (`''`) or longer than one character (`'!!'`, or a character such as an emoji that takes two UTF-16 code units). - An `ESCAPE` value that is not a string, such as `ESCAPE 1`. -When the new engine plans the statement, a pattern written in it is checked before any row is read. When the legacy engine runs it, the pattern is checked when a row is tested against it, so a statement that tests no rows does not fail. +The new engine checks a literal pattern before any row is read (including under `auto`, unless the statement falls back earlier on an unsupported clause such as `RIGHT JOIN`). When the legacy engine checks a pattern, or when the pattern comes from an expression rather than a string literal, it is checked when a row is tested against it, so a statement that tests no rows does not fail. Before v5.4.0, `ESCAPE` was ignored, so `ip LIKE 't0!%' ESCAPE '!'` matched every value starting with `t0!`, and an `UPDATE` or `DELETE` using it changed those rows. @@ -483,7 +483,7 @@ The legacy engine also reads `[...]` in a pattern as a character class, matched - `[a-z]` matches one character in the range. - `[^abc]` matches one character that is not in the set. - A range written backwards, such as `[z-a]`, is rejected with a 400 error when a row is tested against it. -- A `[` with no closing `]` is a literal. +- A `[` with no closing `]` is a literal, as is an escaped `[` (such as `![` with `ESCAPE '!'`). The new engine has no character classes: `[` and `]` match themselves. Under `auto`, a pattern such as `'[a]%'` is a class on the legacy engine and literal text on the new engine, so avoid `[` in patterns unless you know which engine runs the statement. @@ -491,6 +491,24 @@ The new engine has no character classes: `[` and `]` match themselves. Under `au Testing one value takes time at most proportional to the value's length times the pattern's length, for any mix of `%` and `_`. Before v5.4.0, a short pattern such as `'%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z'` against a long value could block a thread. Both lengths still multiply, so a pattern thousands of characters long tested against values millions of characters long is slow. If users supply patterns, limit their length. +Each segment of a pattern (the text between `%` wildcards, or the whole pattern when it has no `%`) can be at most 10,000 characters long. Each literal character, `_` wildcard, escaped character (such as `!%` with `ESCAPE '!'`), or legacy `[...]` character class counts as one character toward that limit; a pattern can have multiple segments of up to 10,000 characters separated by `%`. A pattern with a segment longer than 10,000 characters is rejected with a 400 error: + +```json +{ + "error": "SQL engine v2: LIKE pattern must not have more than 10000 characters between % wildcards" +} +``` + +When the legacy engine checks the pattern, the error message omits the `SQL engine v2: ` prefix (`{"error": "LIKE pattern must not have more than 10000 characters between % wildcards"}`). + +On the legacy engine, a pattern with more than 64 `_` wildcards (in segments that also contain literal characters or `[...]` classes) is also rejected with a 400 error in the rare case that the pattern contains at least one code unit from every consecutive two-code-unit pair in the reserved Unicode noncharacter ranges (`U+FDD0` to `U+FDEF`, `U+FFFE` to `U+FFFF`) and Private Use Area (`U+E000` to `U+F8FF`, about 3,200 pairs in all): + +```json +{ + "error": "LIKE pattern with more than 64 _ names too many private-use characters" +} +``` + The total cost of a `LIKE` is still the number of rows it tests. On an indexed attribute, the index serves a prefix pattern, which is literal text followed by one `%` (`'abc%'`, or `'50!%%' ESCAPE '!'` for values starting with `50%`), and a pattern with no wildcards, which is an equality test. Any other `LIKE` filters the rows that the rest of the query selects, so with no indexed condition it tests every row in the table. Whether such a statement falls back to the legacy engine, scans on the new engine, or is rejected with a 400 error depends on `sql.engine`, `sql.allowFullScan` and the rest of the statement; see [The fallback contract](#the-fallback-contract) and [Query Performance](#query-performance). --- diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index c13c498a3..bea2ec003 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -59,9 +59,10 @@ This change is not released yet. **Upgrade impact:** +- On both engines, `_` now matches line breaks (`\n`, `\r`, U+2028, U+2029); previously it matched none on either engine, so a pattern such as `'a_b'` in `SELECT`, `UPDATE` or `DELETE` now also matches `a\nb`. - On the new engine, `*` is now a literal (it acted as a regular-expression quantifier), and `%` now matches line breaks, as it already did on the legacy engine. - On the legacy engine, `\`, `^` and `$` are now literals. They acted as regular-expression syntax, so `'a\d'` matched `a5`. -- On both engines, these are rejected with a 400 error: a pattern ending in the escape character, and an `ESCAPE` value that is not exactly one character or not a string. On the legacy engine, so is a character-class range written backwards, such as `[z-a]`. The legacy engine checks a pattern when it tests a row against it. +- On both engines, these are rejected with a 400 error: a pattern segment between `%` wildcards longer than 10,000 characters, a pattern ending in the escape character, and an `ESCAPE` value that is not a single-character string. On the legacy engine, so is a character-class range written backwards (such as `[z-a]`), or a pattern with more than 64 `_` wildcards that also contains a character from every consecutive pair of reserved Unicode noncharacters (`U+FDD0` to `U+FDEF`, `U+FFFE` to `U+FFFF`) and Private Use Area characters (`U+E000` to `U+F8FF`). The legacy engine checks a pattern when it tests a row against it. `LIKE` is still case-sensitive on the new engine and case-insensitive on the legacy engine. Under the default `sql.engine: auto`, the case behavior depends on which engine runs the statement, so adding or removing an index can change the rows a `LIKE` in an `UPDATE` or `DELETE` matches. See [LIKE Pattern Matching](/reference/v5/operations-api/sql#like-pattern-matching). From e7024b087026e3f044a07a8c45b104356827bd17 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 4 Nov 2026 06:22:47 -0700 Subject: [PATCH 7/9] docs(sql): clarify LIKE lower-case workaround and 64-underscore wording Dispatch-Task: documentation-786-like-rules-update Dispatch-Task: pr-maint-53d171c3888f7402736f60bbd9a4d118 Co-Authored-By: Gemini --- reference/operations-api/sql.md | 12 ++++++------ release-notes/v5-lincoln/5.4.md | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index bf7cdfcb8..0dbe9eeaa 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -12,7 +12,7 @@ title: SQL :::info Harper 5.2 introduces a new SQL engine -As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the legacy engine's results (which [differ in a few behaviors](#behavior-changes-from-the-legacy-engine), such as `LIKE` case sensitivity) with the older full-scan cost — each fallback is logged at the `info` level (`SQL engine v2 fallback: ...`) with the reason, so you have direct visibility into which queries aren't being planned. See [Query Performance](#query-performance) for how to keep your queries on the fast path. +As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the legacy engine's results (which [differ from the new engine's in a few behaviors](#behavior-changes-from-the-legacy-engine), such as `LIKE` case sensitivity) with the older full-scan cost — each fallback is logged at the `info` level (`SQL engine v2 fallback: ...`) with the reason, so you have direct visibility into which queries aren't being planned. See [Query Performance](#query-performance) for how to keep your queries on the fast path. For your most performance-critical production read paths, the [REST interface](../rest/overview.md) remains the recommended choice — it provides a stable, cacheable, index-driven contract. SQL is well suited to ad-hoc investigation, administrative queries, joins, and reporting. ::: @@ -165,7 +165,7 @@ By default (`sql.allowFullScan: false`), the new engine does not run a query tha ### Behavior changes from the legacy engine -For queries the new engine plans (the default under `auto`), a handful of non-standard legacy behaviors are corrected, so you may see different — standard-SQL-conformant — results after upgrading: +For queries the new engine plans (the default under `auto`), `LIKE` is case-sensitive and a handful of non-standard legacy behaviors are corrected, so you may see different results after upgrading: - **Aggregates over an empty result set** return `NULL` for `SUM`, `AVG`, `MIN`, and `MAX` (and `0` for `COUNT`), per the SQL standard. The legacy engine returned `0` for `SUM` and omitted `MIN`/`MAX` from the response entirely. - **`MIN`/`MAX` over text columns** return the lexicographically smallest/largest value. The legacy engine produced no result for string `MIN`/`MAX`. @@ -174,7 +174,7 @@ For queries the new engine plans (the default under `auto`), a handful of non-st - **`LIKE` is case-sensitive, and `[...]` in a pattern is literal text.** The legacy engine matches case-insensitively and reads `[...]` as a character class. Under `auto`, adding or removing an index, or changing the other conditions in a statement, can move it between engines, which changes the rows a `LIKE` matches, including the rows an `UPDATE` or `DELETE` changes. See [LIKE Pattern Matching](#like-pattern-matching). - **A `NULL`-valued expression result is returned as an explicit `null`** rather than omitted from the row. A computed column or `COALESCE` that evaluates to `NULL` — or `UPPER()` — now yields `null` (the legacy engine dropped the key, and `UPPER` of a `NULL` returned the literal string `"NULL"`). -These are deliberate corrections toward standard SQL, applied when the new engine serves the query — not fallbacks. If your application depended on a legacy quirk, adjust for the standard behavior. For how `WHERE` compares `NULL` and omitted values, including the v5.4.0 change to ranges compared with `NULL`, see [NULL and Omitted Values](#null-and-omitted-values). +These differences apply whenever the new engine serves the query — not on fallbacks. If your application depended on the legacy behavior, adjust your queries accordingly. For how `WHERE` compares `NULL` and omitted values, including the v5.4.0 change to ranges compared with `NULL`, see [NULL and Omitted Values](#null-and-omitted-values). ### Practical guidance @@ -471,9 +471,9 @@ Under the default `sql.engine: auto`, the same pattern can be either, and the wh - `note LIKE 't0%'` alone has no index to drive it, so `auto` runs it on the legacy engine, and it is case-insensitive. - `note LIKE 't0%'` is case-sensitive when something else keeps the statement on the new engine: an indexed condition joined with `AND`, an `ORDER BY` on the primary key, or `sql.allowFullScan: true`. -This applies to `UPDATE` and `DELETE` as well: adding an index on `note` moves `DELETE FROM dev.addr WHERE note LIKE 't0%'` to the new engine, and it stops deleting `T0b`. A statement that falls back is logged, which tells you which engine ran it (see [The fallback contract](#the-fallback-contract)). +This applies to `UPDATE` and `DELETE` as well: adding an index on `note` moves `DELETE FROM dev.addr WHERE note LIKE 't0%'` to the new engine, and it stops deleting `T0b`. With `logging.level` set to `info`, a statement that falls back is logged, which tells you which engine ran it (see [The fallback contract](#the-fallback-contract)). -To match regardless of case on both engines, lower-case the value: `LOWER(note) LIKE 't0%'`. `LOWER()` uses JavaScript's lower-casing, which treats a few non-ASCII characters differently from the legacy engine's case-insensitive matching. The index can't serve an expression, so combine it with an indexed condition using `AND`. Otherwise the statement is a full scan of the table. To keep a case-insensitive prefix search on the index, store a lower-cased copy of the value in its own indexed attribute and query that. +To match regardless of case on both engines, lower-case both the value and the pattern: `LOWER(note) LIKE 't0%'` (write the pattern in lower case, or lower-case a user-supplied pattern before sending it). `LOWER()` uses JavaScript's lower-casing, which treats a few non-ASCII characters differently from the legacy engine's case-insensitive matching. The index can't serve an expression, so combine it with an indexed condition using `AND`. Otherwise the statement scans the whole table, or is rejected with a 400 error under `sql.engine: new` with `sql.allowFullScan: false`. To keep a case-insensitive prefix search on the index, store a lower-cased copy of the value in its own indexed attribute and query that. ### Character Classes on the Legacy Engine @@ -501,7 +501,7 @@ Each segment of a pattern (the text between `%` wildcards, or the whole pattern When the legacy engine checks the pattern, the error message omits the `SQL engine v2: ` prefix (`{"error": "LIKE pattern must not have more than 10000 characters between % wildcards"}`). -On the legacy engine, a pattern with more than 64 `_` wildcards (in segments that also contain literal characters or `[...]` classes) is also rejected with a 400 error in the rare case that the pattern contains at least one code unit from every consecutive two-code-unit pair in the reserved Unicode noncharacter ranges (`U+FDD0` to `U+FDEF`, `U+FFFE` to `U+FFFF`) and Private Use Area (`U+E000` to `U+F8FF`, about 3,200 pairs in all): +On the legacy engine, a pattern with more than 64 `_` wildcards across segments that also contain literal characters or `[...]` classes is also rejected with a 400 error in the rare case that the pattern contains at least one code unit from every consecutive two-code-unit pair in the reserved Unicode noncharacter ranges (`U+FDD0` to `U+FDEF`, `U+FFFE` to `U+FFFF`) and Private Use Area (`U+E000` to `U+F8FF`, about 3,200 pairs in all): ```json { diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index bea2ec003..739a54ea3 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -62,7 +62,7 @@ This change is not released yet. - On both engines, `_` now matches line breaks (`\n`, `\r`, U+2028, U+2029); previously it matched none on either engine, so a pattern such as `'a_b'` in `SELECT`, `UPDATE` or `DELETE` now also matches `a\nb`. - On the new engine, `*` is now a literal (it acted as a regular-expression quantifier), and `%` now matches line breaks, as it already did on the legacy engine. - On the legacy engine, `\`, `^` and `$` are now literals. They acted as regular-expression syntax, so `'a\d'` matched `a5`. -- On both engines, these are rejected with a 400 error: a pattern segment between `%` wildcards longer than 10,000 characters, a pattern ending in the escape character, and an `ESCAPE` value that is not a single-character string. On the legacy engine, so is a character-class range written backwards (such as `[z-a]`), or a pattern with more than 64 `_` wildcards that also contains a character from every consecutive pair of reserved Unicode noncharacters (`U+FDD0` to `U+FDEF`, `U+FFFE` to `U+FFFF`) and Private Use Area characters (`U+E000` to `U+F8FF`). The legacy engine checks a pattern when it tests a row against it. +- On both engines, these are rejected with a 400 error: a pattern segment between `%` wildcards longer than 10,000 characters, a pattern ending in the escape character, and an `ESCAPE` value that is not a single-character string. On the legacy engine, so is a character-class range written backwards (such as `[z-a]`), or a pattern with more than 64 `_` wildcards across segments that also contain literal characters or `[...]` classes when it also contains a character from every consecutive pair of reserved Unicode noncharacters (`U+FDD0` to `U+FDEF`, `U+FFFE` to `U+FFFF`) and Private Use Area characters (`U+E000` to `U+F8FF`). The legacy engine checks a pattern when it tests a row against it. `LIKE` is still case-sensitive on the new engine and case-insensitive on the legacy engine. Under the default `sql.engine: auto`, the case behavior depends on which engine runs the statement, so adding or removing an index can change the rows a `LIKE` in an `UPDATE` or `DELETE` matches. See [LIKE Pattern Matching](/reference/v5/operations-api/sql#like-pattern-matching). From 5fa407780ece1b0baaf3044670986ba073c1e9a0 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 4 Nov 2026 18:51:06 -0700 Subject: [PATCH 8/9] docs(sql): qualify LIKE history for prefix patterns, non-literal ESCAPE, and literal characters - A literal prefix followed by one trailing % was already matched as a prefix before 5.4, so `%` and `*` did not change for it. - A non-literal ESCAPE is unsupported on the new engine and falls back under auto. - Only `*`, `\`, `^` and `$` changed meaning; `.`, `?` and `+` were already literal. Co-Authored-By: Claude Sonnet 5.5 Dispatch-Task: pr-maint-53d171c3888f7402736f60bbd9a4d118 --- reference/operations-api/sql.md | 16 ++++++++-------- release-notes/v5-lincoln/5.4.md | 2 +- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index 0dbe9eeaa..f05f476ea 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -420,14 +420,14 @@ Geospatial data must be stored using the [GeoJSON standard](https://geojson.org/ `value LIKE pattern` is true when the whole value matches the pattern, and `NOT LIKE` negates it. The value is matched as a string. The pattern can contain these elements: -| Pattern element | Matches | -| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -| `%` | Any run of characters, including none, and including line breaks (`\n`, `\r`, U+2028, U+2029). | -| `_` | Exactly one character, including a line break (`\n`, `\r`, U+2028, U+2029). | -| `[abc]`, `[a-z]`, `[^abc]` | Legacy engine only: one character in the set or range, or with `^`, one character not in it. See the list below. | -| Any other character | Itself. `*`, `\`, `^`, `$`, `.`, `?` and `+` have no special meaning on either engine. | +| Pattern element | Matches | +| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `%` | Any run of characters, including none, and including line breaks (`\n`, `\r`, U+2028, U+2029). | +| `_` | Exactly one character, including a line break (`\n`, `\r`, U+2028, U+2029). | +| `[abc]`, `[a-z]`, `[^abc]` | Legacy engine only: one character in the set or range, or with `^`, one character not in it. See the list below. | +| Any other character | Itself. `.`, `?` and `+` have no special meaning on either engine; since v5.4.0, neither do `*`, `\`, `^` and `$`. | -`_` matches one UTF-16 code unit, so a character outside the Basic Multilingual Plane, such as most emoji, takes two: `__`. Before v5.4.0, `_` did not match line breaks on either engine, and `%` did not match line breaks on the new engine. +`_` matches one UTF-16 code unit, so a character outside the Basic Multilingual Plane, such as most emoji, takes two: `__`. Before v5.4.0, `_` did not match line breaks on either engine, and `%` did not match line breaks on the new engine, except in a pattern that is a literal prefix followed by one trailing `%`, such as `'a%'`, which was already matched as a prefix. ```sql SELECT * FROM dev.dog WHERE dog_name LIKE 'Pen%' -- starts with "Pen" @@ -453,7 +453,7 @@ These are rejected with a 400 error, and no rows are changed: - An `ESCAPE` value that is empty (`''`) or longer than one character (`'!!'`, or a character such as an emoji that takes two UTF-16 code units). - An `ESCAPE` value that is not a string, such as `ESCAPE 1`. -The new engine checks a literal pattern before any row is read (including under `auto`, unless the statement falls back earlier on an unsupported clause such as `RIGHT JOIN`). When the legacy engine checks a pattern, or when the pattern comes from an expression rather than a string literal, it is checked when a row is tested against it, so a statement that tests no rows does not fail. +The new engine checks a literal pattern before any row is read (including under `auto`, unless the statement falls back earlier on an unsupported clause such as `RIGHT JOIN`). When the legacy engine checks a pattern, or when the pattern comes from an expression rather than a string literal, it is checked when a row is tested against it, so a statement that tests no rows does not fail. The new engine accepts only a string literal as the `ESCAPE` value; a column or expression there is unsupported, so under `auto` the statement runs on the legacy engine, which checks the value when it tests a row. Before v5.4.0, `ESCAPE` was ignored, so `ip LIKE 't0!%' ESCAPE '!'` matched every value starting with `t0!`, and an `UPDATE` or `DELETE` using it changed those rows. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 739a54ea3..215e0aef0 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -60,7 +60,7 @@ This change is not released yet. **Upgrade impact:** - On both engines, `_` now matches line breaks (`\n`, `\r`, U+2028, U+2029); previously it matched none on either engine, so a pattern such as `'a_b'` in `SELECT`, `UPDATE` or `DELETE` now also matches `a\nb`. -- On the new engine, `*` is now a literal (it acted as a regular-expression quantifier), and `%` now matches line breaks, as it already did on the legacy engine. +- On the new engine, `*` is now a literal (it acted as a regular-expression quantifier), and `%` now matches line breaks, as it already did on the legacy engine. A pattern that is a literal prefix followed by one trailing `%`, such as `'a*%'`, was already matched as a prefix, so it does not change. - On the legacy engine, `\`, `^` and `$` are now literals. They acted as regular-expression syntax, so `'a\d'` matched `a5`. - On both engines, these are rejected with a 400 error: a pattern segment between `%` wildcards longer than 10,000 characters, a pattern ending in the escape character, and an `ESCAPE` value that is not a single-character string. On the legacy engine, so is a character-class range written backwards (such as `[z-a]`), or a pattern with more than 64 `_` wildcards across segments that also contain literal characters or `[...]` classes when it also contains a character from every consecutive pair of reserved Unicode noncharacters (`U+FDD0` to `U+FDEF`, `U+FFFE` to `U+FFFF`) and Private Use Area characters (`U+E000` to `U+F8FF`). The legacy engine checks a pattern when it tests a row against it. From b85698eccd398b5305214d5a0d9d46bca6c388f7 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 4 Nov 2026 18:53:28 -0700 Subject: [PATCH 9/9] docs(sql): limit the prefix-pattern exception to patterns an index served A prefix pattern evaluated as a residual filter used the regular-expression matcher before 5.4, so its matches did change. Co-Authored-By: Claude Sonnet 5.5 Dispatch-Task: pr-maint-53d171c3888f7402736f60bbd9a4d118 --- reference/operations-api/sql.md | 2 +- release-notes/v5-lincoln/5.4.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index f05f476ea..b3309ada8 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -427,7 +427,7 @@ Geospatial data must be stored using the [GeoJSON standard](https://geojson.org/ | `[abc]`, `[a-z]`, `[^abc]` | Legacy engine only: one character in the set or range, or with `^`, one character not in it. See the list below. | | Any other character | Itself. `.`, `?` and `+` have no special meaning on either engine; since v5.4.0, neither do `*`, `\`, `^` and `$`. | -`_` matches one UTF-16 code unit, so a character outside the Basic Multilingual Plane, such as most emoji, takes two: `__`. Before v5.4.0, `_` did not match line breaks on either engine, and `%` did not match line breaks on the new engine, except in a pattern that is a literal prefix followed by one trailing `%`, such as `'a%'`, which was already matched as a prefix. +`_` matches one UTF-16 code unit, so a character outside the Basic Multilingual Plane, such as most emoji, takes two: `__`. Before v5.4.0, `_` did not match line breaks on either engine, and `%` did not match line breaks on the new engine, except in a pattern that is a literal prefix followed by one trailing `%`, such as `'a%'`, when an index on the attribute served it as a prefix scan. ```sql SELECT * FROM dev.dog WHERE dog_name LIKE 'Pen%' -- starts with "Pen" diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 215e0aef0..9f5615d80 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -60,7 +60,7 @@ This change is not released yet. **Upgrade impact:** - On both engines, `_` now matches line breaks (`\n`, `\r`, U+2028, U+2029); previously it matched none on either engine, so a pattern such as `'a_b'` in `SELECT`, `UPDATE` or `DELETE` now also matches `a\nb`. -- On the new engine, `*` is now a literal (it acted as a regular-expression quantifier), and `%` now matches line breaks, as it already did on the legacy engine. A pattern that is a literal prefix followed by one trailing `%`, such as `'a*%'`, was already matched as a prefix, so it does not change. +- On the new engine, `*` is now a literal (it acted as a regular-expression quantifier), and `%` now matches line breaks, as it already did on the legacy engine. A pattern that is a literal prefix followed by one trailing `%`, such as `'a*%'`, already behaved this way when an index on the attribute served it as a prefix scan. - On the legacy engine, `\`, `^` and `$` are now literals. They acted as regular-expression syntax, so `'a\d'` matched `a5`. - On both engines, these are rejected with a 400 error: a pattern segment between `%` wildcards longer than 10,000 characters, a pattern ending in the escape character, and an `ESCAPE` value that is not a single-character string. On the legacy engine, so is a character-class range written backwards (such as `[z-a]`), or a pattern with more than 64 `_` wildcards across segments that also contain literal characters or `[...]` classes when it also contains a character from every consecutive pair of reserved Unicode noncharacters (`U+FDD0` to `U+FDEF`, `U+FFFE` to `U+FFFF`) and Private Use Area characters (`U+E000` to `U+F8FF`). The legacy engine checks a pattern when it tests a row against it.