diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index 4667acff2..b3309ada8 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -12,7 +12,7 @@ title: SQL :::info Harper 5.2 introduces a new SQL engine -As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the same results but with the older full-scan cost — each fallback is logged (`SQL engine v2 fallback: ...`) with the reason, so you have direct visibility into which queries aren't being planned. See [Query Performance](#query-performance) for how to keep your queries on the fast path. +As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the legacy engine's results (which [differ from the new engine's in a few behaviors](#behavior-changes-from-the-legacy-engine), such as `LIKE` case sensitivity) with the older full-scan cost — each fallback is logged at the `info` level (`SQL engine v2 fallback: ...`) with the reason, so you have direct visibility into which queries aren't being planned. See [Query Performance](#query-performance) for how to keep your queries on the fast path. For your most performance-critical production read paths, the [REST interface](../rest/overview.md) remains the recommended choice — it provides a stable, cacheable, index-driven contract. SQL is well suited to ad-hoc investigation, administrative queries, joins, and reporting. ::: @@ -111,7 +111,7 @@ Use `IS NULL` and `IS NOT NULL` to test for `NULL`. REST and Resource API condit ## Query Performance -Harper 5.2's SQL engine maps SQL clauses onto Harper's native indexed storage. When a query can be planned against an index, it runs as a streaming, index-driven operation with memory bounded by the size of the result — not the size of the table. When it can't, the engine falls back to the legacy path, which fetches candidate rows and evaluates the query in memory, with cost and memory growing with the number of rows scanned. +Harper 5.2's SQL engine maps SQL clauses onto Harper's native indexed storage. When a query can be planned against an index, it runs as a streaming, index-driven operation with memory bounded by the size of the result — not the size of the table. When it can't, the engine falls back to the legacy path, which fetches candidate rows and evaluates the query in memory, with cost and memory growing with the number of rows scanned. This section describes the default `auto` mode; see [The fallback contract](#the-fallback-contract) for `sql.engine: new` and `sql.allowFullScan`. On any non-trivial table the difference between these two paths is large. @@ -131,7 +131,7 @@ These map directly to indexed storage operations: - **Equality and `IN` on an indexed attribute** — `WHERE status = 'active'`, `WHERE breed_id IN (1, 2, 3)`. - **Range predicates on an indexed attribute** — `<`, `<=`, `>`, `>=`, and `BETWEEN` (compiled to a bounded index range). - **`OR` of indexed predicates** — `WHERE a = 1 OR b = 2` is served as a union of index scans, provided **every** branch is an index-driving predicate on an indexed attribute. -- **Prefix `LIKE`** — `WHERE name LIKE 'Har%'` compiles to an indexed `starts_with` scan. +- **Prefix `LIKE`** — `WHERE name LIKE 'Har%'` compiles to an indexed `starts_with` scan. The new engine matches it case-sensitively; see [LIKE Pattern Matching](#like-pattern-matching). - **`ORDER BY` on an indexed attribute, combined with an indexed `WHERE`** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. Without an index-driving `WHERE` condition, an `ORDER BY` on a _secondary_ indexed attribute runs on the legacy path — except when sorting on the table's **primary key**, which drives the scan from the primary key's natural index order even with no `WHERE` clause. - **`GROUP BY` on an indexed attribute, combined with an indexed `WHERE`** — aggregates stream group-by-group with O(1) memory per group (memory is bounded by the group count; every matching row is still read). Without an index-driving `WHERE` condition, a `GROUP BY` currently runs on the legacy path. - **`INNER`/`LEFT JOIN` on an indexed join key** — executed as an index-nested-loop join (stream the outer table, probe the inner by index). Keep the join key indexed on the inner (probed) table. @@ -139,33 +139,42 @@ These map directly to indexed storage operations: ### Queries that fall back to a full scan (avoid on large tables) -These can't be served from an index. They still return correct results, but the engine falls back to the legacy full-scan path, so cost and memory grow with table size: +These can't be served from an index. Under the default `auto` mode with `sql.allowFullScan` off, the engine falls back to the legacy full-scan path, so cost and memory grow with table size. Under `sql.engine: new`, a statement the new engine can't plan is rejected with a 400 error instead, and `sql.allowFullScan: true` lets the new engine scan the table for a filter it can't serve from an index (see [The fallback contract](#the-fallback-contract)): - **Filters on un-indexed attributes** — any equality, range, or `IN` on an attribute with no index, _when it's the only predicate_. Combined via `AND` with an indexed predicate, it's applied as a cheap residual filter after the index narrows the scan, and the query stays index-served overall — it's only a standalone un-indexed filter (or one joined by `OR`) that forces a full scan. - **`OR` where a branch isn't index-driving** — e.g. `WHERE a = 1 OR b = 2` when `b` is un-indexed, or `WHERE a = 1 OR name LIKE '%x%'`. A single un-indexed (or non-index-comparator) branch taints the whole disjunction and forces a scan. When **every** branch is an indexed, index-driving predicate, the `OR` stays on the fast path (see above). -- **Suffix / substring `LIKE`** — `LIKE '%x'` and `LIKE '%x%'` (only prefix `LIKE 'x%'` is index-served). A suffix/contains `LIKE` combined (via `AND`) with an indexed predicate is applied as a cheap residual filter on the indexed result, so pair it with an indexed condition. +- **Suffix / substring `LIKE`** — `LIKE '%x'` and `LIKE '%x%'` (only prefix `LIKE 'x%'` is index-served). A suffix/contains `LIKE` combined (via `AND`) with an indexed predicate is applied as a cheap residual filter on the indexed result, so pair it with an indexed condition. That runs the statement on the new engine, where `LIKE` is [case-sensitive](#case-sensitivity-depends-on-the-engine). - **`!=` / `<>` and `NOT` negations** — matching "everything except" a value can't use an index. - **`ORDER BY` with no indexed filter to drive it** — a table-wide ordered scan on an un-indexed sort key. - **`SEARCH_JSON`** — evaluates JSONata over each candidate row's nested JSON; it is not index-backed. Narrow the candidate set with an indexed `WHERE` condition first. -- **`RIGHT`/`FULL OUTER JOIN`, no-`WHERE` `UPDATE`/`DELETE`, `COUNT(DISTINCT ...)`** — the new engine doesn't plan these yet, so they run on the legacy engine (correct results, legacy cost). +- **`RIGHT`/`FULL OUTER JOIN`, no-`WHERE` `UPDATE`/`DELETE`, `COUNT(DISTINCT ...)`** — the new engine doesn't plan these yet, so they run on the legacy engine (legacy results and cost). Note that `UNION`, sub-`SELECT`, and `INSERT … SELECT` are not supported by either engine — see the [Features Matrix](#features-matrix). ### The fallback contract -The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't. A query that falls back returns exactly what the legacy engine would — **its results don't change, only its performance profile.** Fallbacks are logged (`SQL engine v2 fallback: ...`) with the reason, which is a useful signal for finding queries worth an added index or a reshape. (For queries the new engine _does_ plan, it corrects a few long-standing legacy quirks — see [Behavior changes from the legacy engine](#behavior-changes-from-the-legacy-engine).) +The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't. A query that falls back returns exactly what the legacy engine would, at the legacy engine's cost. Fallbacks are logged (`SQL engine v2 fallback: ...`) with the reason, which is a useful signal for finding queries worth an added index or a reshape. They are logged at the `info` level, below the default `warn`, so set [`logging.level`](../logging/configuration.md#logginglevel) to `info` to see them. (Where the two engines differ, which engine runs a statement decides its results: for queries the new engine _does_ plan, it corrects a few long-standing legacy quirks, and `LIKE` matches differently — see [Behavior changes from the legacy engine](#behavior-changes-from-the-legacy-engine).) + +The `sql.engine` setting in the [configuration file](../configuration/overview.md) selects the mode, and the `HARPER_SQL_ENGINE` environment variable overrides it: + +- `auto` (default) — runs each statement on the new engine when it can, and on the legacy engine otherwise. +- `new` — runs every statement on the new engine. A statement the new engine can't plan is rejected with a 400 error instead of falling back. +- `legacy` — runs every statement on the legacy engine. + +By default (`sql.allowFullScan: false`), the new engine does not run a query that has neither an index-driving condition nor an `ORDER BY` on the primary key, which is one reason `auto` falls back. With `sql.allowFullScan: true`, the new engine runs such a query as a full scan instead of falling back (under `auto`) or rejecting it (under `new`). A full scan on the new engine is not a fallback, so it is not logged. ### Behavior changes from the legacy engine -For queries the new engine plans (the default under `auto`), a handful of non-standard legacy behaviors are corrected, so you may see different — standard-SQL-conformant — results after upgrading: +For queries the new engine plans (the default under `auto`), `LIKE` is case-sensitive and a handful of non-standard legacy behaviors are corrected, so you may see different results after upgrading: - **Aggregates over an empty result set** return `NULL` for `SUM`, `AVG`, `MIN`, and `MAX` (and `0` for `COUNT`), per the SQL standard. The legacy engine returned `0` for `SUM` and omitted `MIN`/`MAX` from the response entirely. - **`MIN`/`MAX` over text columns** return the lexicographically smallest/largest value. The legacy engine produced no result for string `MIN`/`MAX`. - **`NULL` never equals `NULL` in a join key.** Rows whose join key is `NULL` on either side no longer match each other (three-valued logic). The legacy engine incorrectly joined `NULL` to `NULL`. - **`OFFSET` past the end of a result returns no rows.** The legacy engine could still return rows when the `OFFSET` exceeded the number of matches. +- **`LIKE` is case-sensitive, and `[...]` in a pattern is literal text.** The legacy engine matches case-insensitively and reads `[...]` as a character class. Under `auto`, adding or removing an index, or changing the other conditions in a statement, can move it between engines, which changes the rows a `LIKE` matches, including the rows an `UPDATE` or `DELETE` changes. See [LIKE Pattern Matching](#like-pattern-matching). - **A `NULL`-valued expression result is returned as an explicit `null`** rather than omitted from the row. A computed column or `COALESCE` that evaluates to `NULL` — or `UPPER()` — now yields `null` (the legacy engine dropped the key, and `UPPER` of a `NULL` returned the literal string `"NULL"`). -These are deliberate corrections toward standard SQL, applied when the new engine serves the query — not fallbacks. If your application depended on a legacy quirk, adjust for the standard behavior. For how `WHERE` compares `NULL` and omitted values, including the v5.4.0 change to ranges compared with `NULL`, see [NULL and Omitted Values](#null-and-omitted-values). +These differences apply whenever the new engine serves the query — not on fallbacks. If your application depended on the legacy behavior, adjust your queries accordingly. For how `WHERE` compares `NULL` and omitted values, including the v5.4.0 change to ranges compared with `NULL`, see [NULL and Omitted Values](#null-and-omitted-values). ### Practical guidance @@ -399,11 +408,108 @@ Geospatial data must be stored using the [GeoJSON standard](https://geojson.org/ ## Logical Operators -| Keyword | Description | -| --------- | ------------------------------------------------ | -| `BETWEEN` | Returns values within a given range (inclusive). | -| `IN` | Specifies multiple values in a WHERE clause. | -| `LIKE` | Searches for a pattern. | +| Keyword | Description | +| --------- | -------------------------------------------------------------------------------------------------------------------------------- | +| `BETWEEN` | Returns values within a given range (inclusive). | +| `IN` | Specifies multiple values in a WHERE clause. | +| `LIKE` | Matches a string against a pattern. Case sensitivity depends on the engine; see [LIKE Pattern Matching](#like-pattern-matching). | + +--- + +## LIKE Pattern Matching + +`value LIKE pattern` is true when the whole value matches the pattern, and `NOT LIKE` negates it. The value is matched as a string. The pattern can contain these elements: + +| Pattern element | Matches | +| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `%` | Any run of characters, including none, and including line breaks (`\n`, `\r`, U+2028, U+2029). | +| `_` | Exactly one character, including a line break (`\n`, `\r`, U+2028, U+2029). | +| `[abc]`, `[a-z]`, `[^abc]` | Legacy engine only: one character in the set or range, or with `^`, one character not in it. See the list below. | +| Any other character | Itself. `.`, `?` and `+` have no special meaning on either engine; since v5.4.0, neither do `*`, `\`, `^` and `$`. | + +`_` matches one UTF-16 code unit, so a character outside the Basic Multilingual Plane, such as most emoji, takes two: `__`. Before v5.4.0, `_` did not match line breaks on either engine, and `%` did not match line breaks on the new engine, except in a pattern that is a literal prefix followed by one trailing `%`, such as `'a%'`, when an index on the attribute served it as a prefix scan. + +```sql +SELECT * FROM dev.dog WHERE dog_name LIKE 'Pen%' -- starts with "Pen" +SELECT * FROM dev.dog WHERE dog_name LIKE '%y' -- ends with "y" +SELECT * FROM dev.dog WHERE dog_name LIKE '_ack' -- "Jack", "Mack", ... +``` + +### Matching `%` and `_` Literally With ESCAPE + + Add `ESCAPE 'c'` to name an escape character. In the pattern, the escape character makes the character after it literal, and writing it twice matches the escape character itself: + +```sql +SELECT * FROM dev.addr WHERE ip LIKE 't0!%' ESCAPE '!' -- exactly "t0%" +SELECT * FROM dev.product WHERE label LIKE '50!%%' ESCAPE '!' -- starts with "50%" +SELECT * FROM dev.addr WHERE ip LIKE 't0!!%' ESCAPE '!' -- starts with "t0!" +``` + +Without `ESCAPE`, a pattern has no escape character: `ip LIKE 't0!%'` matches `t0!` followed by anything. `ESCAPE` works in `SELECT`, `UPDATE` and `DELETE`, on both engines. + +These are rejected with a 400 error, and no rows are changed: + +- A pattern that ends in the escape character, such as `'t0!' ESCAPE '!'`. +- An `ESCAPE` value that is empty (`''`) or longer than one character (`'!!'`, or a character such as an emoji that takes two UTF-16 code units). +- An `ESCAPE` value that is not a string, such as `ESCAPE 1`. + +The new engine checks a literal pattern before any row is read (including under `auto`, unless the statement falls back earlier on an unsupported clause such as `RIGHT JOIN`). When the legacy engine checks a pattern, or when the pattern comes from an expression rather than a string literal, it is checked when a row is tested against it, so a statement that tests no rows does not fail. The new engine accepts only a string literal as the `ESCAPE` value; a column or expression there is unsupported, so under `auto` the statement runs on the legacy engine, which checks the value when it tests a row. + +Before v5.4.0, `ESCAPE` was ignored, so `ip LIKE 't0!%' ESCAPE '!'` matched every value starting with `t0!`, and an `UPDATE` or `DELETE` using it changed those rows. + +### Case Sensitivity Depends on the Engine + +| Engine that runs the statement | `LIKE` is | +| ------------------------------------------------------------------------------------------------------- | --------------------------------------------- | +| New engine: `sql.engine: new`, and statements that `auto` runs on the new engine | Case-sensitive: `'t0%'` does not match `T0b`. | +| Legacy engine: `sql.engine: legacy`, and statements that `auto` [falls back](#the-fallback-contract) on | Case-insensitive: `'t0%'` matches `T0b`. | + +Under the default `sql.engine: auto`, the same pattern can be either, and the whole statement decides which. For example, with `ip` indexed and `note` not indexed: + +- `ip LIKE 't0%'` is a prefix pattern on an indexed attribute, so the new engine serves it from the index, and it is case-sensitive. +- `ip LIKE '%T0B'` is not a prefix pattern, so the index can't drive it. `auto` runs it on the legacy engine, and it matches `T0b`. +- `note LIKE 't0%'` alone has no index to drive it, so `auto` runs it on the legacy engine, and it is case-insensitive. +- `note LIKE 't0%'` is case-sensitive when something else keeps the statement on the new engine: an indexed condition joined with `AND`, an `ORDER BY` on the primary key, or `sql.allowFullScan: true`. + +This applies to `UPDATE` and `DELETE` as well: adding an index on `note` moves `DELETE FROM dev.addr WHERE note LIKE 't0%'` to the new engine, and it stops deleting `T0b`. With `logging.level` set to `info`, a statement that falls back is logged, which tells you which engine ran it (see [The fallback contract](#the-fallback-contract)). + +To match regardless of case on both engines, lower-case both the value and the pattern: `LOWER(note) LIKE 't0%'` (write the pattern in lower case, or lower-case a user-supplied pattern before sending it). `LOWER()` uses JavaScript's lower-casing, which treats a few non-ASCII characters differently from the legacy engine's case-insensitive matching. The index can't serve an expression, so combine it with an indexed condition using `AND`. Otherwise the statement scans the whole table, or is rejected with a 400 error under `sql.engine: new` with `sql.allowFullScan: false`. To keep a case-insensitive prefix search on the index, store a lower-cased copy of the value in its own indexed attribute and query that. + +### Character Classes on the Legacy Engine + +The legacy engine also reads `[...]` in a pattern as a character class, matched without regard to case like the rest of the pattern: + +- `[abc]` matches one of `a`, `b` or `c`. +- `[a-z]` matches one character in the range. +- `[^abc]` matches one character that is not in the set. +- A range written backwards, such as `[z-a]`, is rejected with a 400 error when a row is tested against it. +- A `[` with no closing `]` is a literal, as is an escaped `[` (such as `![` with `ESCAPE '!'`). + +The new engine has no character classes: `[` and `]` match themselves. Under `auto`, a pattern such as `'[a]%'` is a class on the legacy engine and literal text on the new engine, so avoid `[` in patterns unless you know which engine runs the statement. + +### Matching Cost + + Testing one value takes time at most proportional to the value's length times the pattern's length, for any mix of `%` and `_`. Before v5.4.0, a short pattern such as `'%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z'` against a long value could block a thread. Both lengths still multiply, so a pattern thousands of characters long tested against values millions of characters long is slow. If users supply patterns, limit their length. + +Each segment of a pattern (the text between `%` wildcards, or the whole pattern when it has no `%`) can be at most 10,000 characters long. Each literal character, `_` wildcard, escaped character (such as `!%` with `ESCAPE '!'`), or legacy `[...]` character class counts as one character toward that limit; a pattern can have multiple segments of up to 10,000 characters separated by `%`. A pattern with a segment longer than 10,000 characters is rejected with a 400 error: + +```json +{ + "error": "SQL engine v2: LIKE pattern must not have more than 10000 characters between % wildcards" +} +``` + +When the legacy engine checks the pattern, the error message omits the `SQL engine v2: ` prefix (`{"error": "LIKE pattern must not have more than 10000 characters between % wildcards"}`). + +On the legacy engine, a pattern with more than 64 `_` wildcards across segments that also contain literal characters or `[...]` classes is also rejected with a 400 error in the rare case that the pattern contains at least one code unit from every consecutive two-code-unit pair in the reserved Unicode noncharacter ranges (`U+FDD0` to `U+FDEF`, `U+FFFE` to `U+FFFF`) and Private Use Area (`U+E000` to `U+F8FF`, about 3,200 pairs in all): + +```json +{ + "error": "LIKE pattern with more than 64 _ names too many private-use characters" +} +``` + +The total cost of a `LIKE` is still the number of rows it tests. On an indexed attribute, the index serves a prefix pattern, which is literal text followed by one `%` (`'abc%'`, or `'50!%%' ESCAPE '!'` for values starting with `50%`), and a pattern with no wildcards, which is an equality test. Any other `LIKE` filters the rows that the rest of the query selects, so with no indexed condition it tests every row in the table. Whether such a statement falls back to the legacy engine, scans on the new engine, or is rejected with a 400 error depends on `sql.engine`, `sql.allowFullScan` and the rest of the statement; see [The fallback contract](#the-fallback-contract) and [Query Performance](#query-performance). --- diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index fa3976d26..9f5615d80 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -51,6 +51,21 @@ This change is not released yet. This change is not released yet. +## SQL + +### LIKE Honors ESCAPE and No Longer Backtracks + +[Match SQL LIKE in linear time and honor LIKE … ESCAPE, in both SQL engines](https://github.com/HarperFast/harper/pull/3936) fixes two `LIKE` problems on both SQL engines. `LIKE pattern ESCAPE 'c'` now makes the character after `c` literal in `SELECT`, `UPDATE` and `DELETE`. Before, `ESCAPE` was ignored, so `ip LIKE 't0!%' ESCAPE '!'` selected, updated or deleted every value starting with `t0!` instead of only `t0%`. Matching also no longer backtracks: testing a value takes time at most proportional to its length times the pattern's length, where a pattern such as `'%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z'` against a long value could block a thread before. + +**Upgrade impact:** + +- On both engines, `_` now matches line breaks (`\n`, `\r`, U+2028, U+2029); previously it matched none on either engine, so a pattern such as `'a_b'` in `SELECT`, `UPDATE` or `DELETE` now also matches `a\nb`. +- On the new engine, `*` is now a literal (it acted as a regular-expression quantifier), and `%` now matches line breaks, as it already did on the legacy engine. A pattern that is a literal prefix followed by one trailing `%`, such as `'a*%'`, already behaved this way when an index on the attribute served it as a prefix scan. +- On the legacy engine, `\`, `^` and `$` are now literals. They acted as regular-expression syntax, so `'a\d'` matched `a5`. +- On both engines, these are rejected with a 400 error: a pattern segment between `%` wildcards longer than 10,000 characters, a pattern ending in the escape character, and an `ESCAPE` value that is not a single-character string. On the legacy engine, so is a character-class range written backwards (such as `[z-a]`), or a pattern with more than 64 `_` wildcards across segments that also contain literal characters or `[...]` classes when it also contains a character from every consecutive pair of reserved Unicode noncharacters (`U+FDD0` to `U+FDEF`, `U+FFFE` to `U+FFFF`) and Private Use Area characters (`U+E000` to `U+F8FF`). The legacy engine checks a pattern when it tests a row against it. + +`LIKE` is still case-sensitive on the new engine and case-insensitive on the legacy engine. Under the default `sql.engine: auto`, the case behavior depends on which engine runs the statement, so adding or removing an index can change the rows a `LIKE` in an `UPDATE` or `DELETE` matches. See [LIKE Pattern Matching](/reference/v5/operations-api/sql#like-pattern-matching). + ## Replication ### Sharding