From 0d915d5718b558426a0099a9565a54051806d568 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 31 Oct 2026 12:36:02 -0600 Subject: [PATCH 01/12] Document full-text query boosts and optional scoring --- reference/full-text-search/querying.md | 51 ++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 4980eb120..ebacffccb 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -83,6 +83,57 @@ Full-text results use descending relevance order. The only explicit full-text so Full-text queries cannot provide an exact total after authorization, structured filtering, and current-record validation. On a REST interface with `exactCount` enabled, `Prefer: count=exact` reports an unavailable total; otherwise it is downgraded to an estimated count. See [pagination and total count](../rest/querying.md#pagination-and-total-count). `Table.getRecordCount()` counts table records, not matches for a full-text query. +## Query-time relevance controls + + + +Availability: pending release. These controls require a Harper build with query-scoring support and a matching fulltext package. Published fulltext 0.8.0 does not provide them. + +Use `should` to reward additional text matches while retaining the parent condition's required matches: + +```javascript +const target = { + conditions: [ + { + attribute: 'catalogSearch', + comparator: 'matches_all', + value: 'waterproof boots', + should: [ + { + comparator: 'matches_phrase', + value: 'waterproof boots', + fields: ['name'], + boost: 3, + }, + ], + }, + ], + select: ['id', 'name', '$score'], + limit: 20, +}; +const products = Product.search(target); +``` + +A product that matches the required text remains eligible without the optional phrase. A product that matches only an optional clause is excluded. Optional scores add to the required score; they do not change which records match. + +Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, and the supported fuzzy controls. It searches the parent index: omit `attribute`. The list must be nonempty. Nested `should` lists, negated optional descriptors and scoring controls on condition groups, the query target, negated conditions or ordinary record conditions are rejected. + +Set `boost` on the parent condition to multiply its required text score, or on an optional descriptor to multiply that bonus. Boosts multiply existing configured source-field weights; they do not replace those weights or require an index rebuild. Values round to float32 and must then be finite between 0 and 100 inclusive. Negative zero is rejected. Zero preserves matching while contributing zero score. + +Every referenced source must be readable, including optional fields used only for ranking. Omitting `fields` searches all sources and requires permission for all of them. Existing query capacity, fuzzy policy, deadline and result limits also apply to optional clauses. Requested highlights include matching positive required and optional clauses. + +Use the same structured target through HTTP `QUERY`: + +```javascript +const response = await fetch('/Product/', { + method: 'QUERY', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(target), +}); +``` + +Include the application's normal authentication credentials. Operations `search_by_conditions` accepts the same `conditions` with its usual `schema`, `table` and `get_attributes` properties. These controls add no FIQL syntax. + ## Highlights Highlighting returns source-field fragments and matching character spans. The index must configure highlighting and mark at least one source with `highlight: true`. A query can also search sources without that option; those sources can affect matching and ranking but are omitted from `$highlights`. From 96f5f753a2f02515ad962ee1768694518e379b64 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 31 Oct 2026 12:48:16 -0600 Subject: [PATCH 02/12] docs: align query relevance API references --- reference/full-text-search/querying.md | 12 ++++++------ reference/operations-api/operations.md | 2 +- reference/resources/resource-api.md | 6 ++++-- reference/rest/querying.md | 2 +- 4 files changed, 12 insertions(+), 10 deletions(-) diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index ebacffccb..aa399178f 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -17,7 +17,7 @@ const products = Product.search({ `Table.search()` returns an async iterable. Consume it with `for await...of`, or return it directly from a custom Resource to stream the results. -Results are ordered by BM25 relevance. The `FullText` field, not one of its source fields, identifies the search target. +Results are ordered by relevance: BM25 with any supplied [query-time boosts and optional-clause contributions](#query-time-relevance-controls). The `FullText` field, not one of its source fields, identifies the search target. ## Match modes @@ -69,7 +69,7 @@ Direct calls through `tables` or `databases` run in a trusted server-side contex ## Score and ordering -Select `$score` to include the BM25 score: +Select `$score` to include the relevance score. Without query-time controls, this is the BM25 score; [boosts and optional clauses](#query-time-relevance-controls) adjust it when supplied: ```javascript const products = Product.search({ @@ -116,11 +116,11 @@ const products = Product.search(target); A product that matches the required text remains eligible without the optional phrase. A product that matches only an optional clause is excluded. Optional scores add to the required score; they do not change which records match. -Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, and the supported fuzzy controls. It searches the parent index: omit `attribute`. The list must be nonempty. Nested `should` lists, negated optional descriptors and scoring controls on condition groups, the query target, negated conditions or ordinary record conditions are rejected. +Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`: `fuzzyDistance` must be an integer from 0 to 2, and `fuzzyTranspositions` must be a boolean. It searches the parent index: omit `attribute`. The list must be nonempty. Nested `should` lists, negated optional descriptors and scoring controls on condition groups, the query target, negated conditions or ordinary record conditions are rejected. Set `boost` on the parent condition to multiply its required text score, or on an optional descriptor to multiply that bonus. Boosts multiply existing configured source-field weights; they do not replace those weights or require an index rebuild. Values round to float32 and must then be finite between 0 and 100 inclusive. Negative zero is rejected. Zero preserves matching while contributing zero score. -Every referenced source must be readable, including optional fields used only for ranking. Omitting `fields` searches all sources and requires permission for all of them. Existing query capacity, fuzzy policy, deadline and result limits also apply to optional clauses. Requested highlights include matching positive required and optional clauses. +Every referenced source must be readable, including optional fields used only for ranking. Omitting `fields` searches all sources and requires permission for all of them. Server-configured full-text query limits, including clause limits, permitted fuzzy modes and edit distances, execution deadlines and result limits, also apply to optional clauses. Requested highlights include matching positive required and optional clauses. Use the same structured target through HTTP `QUERY`: @@ -277,11 +277,11 @@ Repeat the index parameter to combine the required positive and negated conditio GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather&limit(20) ``` -REST can return configured highlights by selecting `$highlights`. Use `Table.search()` or `search_by_conditions` for explicit `fields`, condition-level `includeHighlights`, or freshness controls. +REST can return configured highlights by selecting `$highlights`. Use [HTTP `QUERY`](../rest/overview.md#tables-and-their-automatic-endpoints), `Table.search()` or `search_by_conditions` for explicit `fields`, condition-level `includeHighlights`, freshness controls, or [query-time relevance controls](#query-time-relevance-controls). ## Operations API -`search_by_conditions` uses the same condition shape: +`search_by_conditions` uses the same condition shape, including the pending-release [`should` and `boost` controls](#query-time-relevance-controls): ```json { diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 981103058..ff24b4d3c 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -317,7 +317,7 @@ Returns records matching one or more conditions. Supports `operator` (`and`/`or` Full-text conditions (Added in: v5.3.0): -Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). +Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. It also preserves the pending-release [`should` and `boost` relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). --- diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 7bc962206..043eaf94e 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1035,11 +1035,13 @@ Array of condition objects to filter records. Each condition: | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | | `fields` | Full-text source fields to search; every name must belong to the selected full-text index | +| `boost` | Optional full-text score multiplier (`number`, default `1`); see pending-release [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | +| `should` | Optional nonempty array of additional scoring clauses; omitted by default; see pending-release [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | | `includeHighlights` | Include configured full-text highlights | | `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; see [freshness controls](../full-text-search/querying.md#freshness-controls) | | `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage; see [freshness controls](../full-text-search/querying.md#freshness-controls) | -The final four properties apply to full-text conditions (v5.3.0). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. +The `fields`, `includeHighlights`, `maxIndexLagMilliseconds` and `waitForIndexMilliseconds` properties apply to full-text conditions (v5.3.0). Availability and constraints for `boost` and `should` are described in [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. Each comparator also has a `not_` form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. A negated condition requires a positive full-text condition on the same index. See [Full-text query execution](./query-optimization.md#full-text-search-work-and-paging) for paging costs and work limits. @@ -1098,7 +1100,7 @@ Special properties: Full-text queries add two special properties: -- `$score`: When a query uses a full-text index, returns its BM25 relevance score. +- `$score`: When a query uses a full-text index, returns its relevance score, including any [query-time boosts and optional-clause contributions](../full-text-search/querying.md#query-time-relevance-controls). - `$highlights`: Enables and returns full-text fragments and matching spans when the index has highlighting configured. See [Highlights](../full-text-search/querying.md#highlights). #### Selecting related records diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 72faec949..23071454f 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -96,7 +96,7 @@ REST's `limit(start,end)` uses an exclusive end. In v5.4, this prefix query skip GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(100,110) ``` -Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. +Use [HTTP `QUERY`](./overview.md#tables-and-their-automatic-endpoints), `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, freshness controls, or [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries can pass that native window within request budgets; source-version checks and filters run before records count toward the offset and limit. Each page is a new query and can see concurrent changes. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Full-text queries cannot provide an exact total count. On a REST interface with `exactCount: true`, a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). Without that opt-in, Harper applies `count=estimated` instead and can return an estimate. From 0d08bda0188a37fffefb713e9d8fb551464b51ae Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 31 Oct 2026 12:49:30 -0600 Subject: [PATCH 03/12] docs: clarify pending scoring contract and validation --- reference/full-text-search/querying.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index aa399178f..fe2b03c2f 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -85,9 +85,7 @@ Full-text queries cannot provide an exact total after authorization, structured ## Query-time relevance controls - - -Availability: pending release. These controls require a Harper build with query-scoring support and a matching fulltext package. Published fulltext 0.8.0 does not provide them. +Availability: pending release; see [Expose full-text query boosts and optional scoring clauses through Table.search and REST (#3843)](https://github.com/HarperFast/harper/issues/3843). These controls require a Harper build with query-scoring support and a compatible fulltext wrapper. The currently published wrapper does not support them. A Harper build with this validation rejects scoring requests with HTTP 400 when the installed wrapper lacks the required query capability. Use `should` to reward additional text matches while retaining the parent condition's required matches: @@ -116,23 +114,23 @@ const products = Product.search(target); A product that matches the required text remains eligible without the optional phrase. A product that matches only an optional clause is excluded. Optional scores add to the required score; they do not change which records match. -Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`: `fuzzyDistance` must be an integer from 0 to 2, and `fuzzyTranspositions` must be a boolean. It searches the parent index: omit `attribute`. The list must be nonempty. Nested `should` lists, negated optional descriptors and scoring controls on condition groups, the query target, negated conditions or ordinary record conditions are rejected. +Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`: `fuzzyDistance` must be an integer from 0 to 2, and `fuzzyTranspositions` must be a boolean. Each descriptor searches the parent index; an `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The group object itself cannot. Nested `should` lists, negated optional descriptors and scoring controls on the query target, negated conditions or ordinary record conditions are rejected with HTTP 400. -Set `boost` on the parent condition to multiply its required text score, or on an optional descriptor to multiply that bonus. Boosts multiply existing configured source-field weights; they do not replace those weights or require an index rebuild. Values round to float32 and must then be finite between 0 and 100 inclusive. Negative zero is rejected. Zero preserves matching while contributing zero score. +Set `boost` on the parent condition to multiply its required text score, or on an optional descriptor to multiply that bonus. Boosts multiply existing configured source-field weights; they do not replace those weights or require an index rebuild. `boost` is a number and defaults to `1` when omitted. Values round to float32 and must then be finite between 0 and 100 inclusive; invalid values, including negative zero, return HTTP 400. Zero preserves matching while contributing zero score. For a standalone required condition without optional clauses, a positive uniform boost rescales scores without changing relative ranking, apart from floating-point ties; zero makes those scores tie. Every referenced source must be readable, including optional fields used only for ranking. Omitting `fields` searches all sources and requires permission for all of them. Server-configured full-text query limits, including clause limits, permitted fuzzy modes and edit distances, execution deadlines and result limits, also apply to optional clauses. Requested highlights include matching positive required and optional clauses. Use the same structured target through HTTP `QUERY`: ```javascript -const response = await fetch('/Product/', { +const response = await fetch('http://localhost:9926/Product/', { method: 'QUERY', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(target), }); ``` -Include the application's normal authentication credentials. Operations `search_by_conditions` accepts the same `conditions` with its usual `schema`, `table` and `get_attributes` properties. These controls add no FIQL syntax. +Include the application's normal authentication credentials. Operations `search_by_conditions` accepts the same `conditions` with its usual `database`, `table` and `get_attributes` properties. These controls add no FIQL syntax. ## Highlights From 645fe3d255cddf56c7ddfd0daf2c046c4d49c878 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 31 Oct 2026 12:52:55 -0600 Subject: [PATCH 04/12] docs: qualify ranking and tuning guidance --- reference/full-text-search/overview.md | 2 +- reference/resources/query-optimization.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 39d9df04b..21045c943 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -89,7 +89,7 @@ Retry rejected writes with backoff rather than immediately adding more load. Ins ## Ranking and result loading -The native index ranks matching documents with BM25. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. +The native index ranks matching documents with BM25, adjusted by any supplied [query-time boosts and optional-clause contributions](./querying.md#query-time-relevance-controls) when those pending-release controls are available. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. Full-text search can be combined with structured filters. Harper checks conditions against current source records before returning them. In v5.4, additional ranked batches can fill a filtered page within request budgets; exhausting a budget returns an error rather than a budget-truncated page. diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 3da6e188b..5ac2fa0a2 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -52,9 +52,9 @@ When a query is executed, conditions are evaluated against the database. Indexed -A condition using a `FullText` field declared with `@fullText` is executed by the native full-text engine and ranked with BM25. Compatible `and` filters narrow the matches. Harper loads current source records and removes stale or deleted candidates before returning them. +A condition using a `FullText` field declared with `@fullText` is executed by the native full-text engine and ranked with BM25, adjusted by any supplied [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) when those pending-release controls are available. Compatible `and` filters narrow the matches. Harper loads current source records and removes stale or deleted candidates before returning them. -Keep result pages bounded. Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded queries can traverse beyond that window within request budgets; an unbounded query can still fail when it exceeds a native window or configured limit. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Use source-field weights to tune ranking, and enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). +Keep result pages bounded. Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded queries can traverse beyond that window within request budgets; an unbounded query can still fail when it exceeds a native window or configured limit. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Use source-field weights to tune index-level ranking; [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls), when available, adjust scores per request without an index rebuild. Enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). ### Indexed vs. Non-Indexed Fields From 45e5eb7e82b9894bacbf8c87fbe0de6ee34be3b5 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 31 Oct 2026 13:17:55 -0600 Subject: [PATCH 05/12] docs: narrow pending scoring guidance after review --- reference/full-text-search/querying.md | 18 ++++++++++++++---- reference/rest/querying.md | 2 +- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index fe2b03c2f..13ee1c5d3 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -85,7 +85,11 @@ Full-text queries cannot provide an exact total after authorization, structured ## Query-time relevance controls -Availability: pending release; see [Expose full-text query boosts and optional scoring clauses through Table.search and REST (#3843)](https://github.com/HarperFast/harper/issues/3843). These controls require a Harper build with query-scoring support and a compatible fulltext wrapper. The currently published wrapper does not support them. A Harper build with this validation rejects scoring requests with HTTP 400 when the installed wrapper lacks the required query capability. +:::note Pending release + +These controls describe the pending implementation tracked by [Expose full-text query boosts and optional scoring clauses through Table.search and REST (#3843)](https://github.com/HarperFast/harper/issues/3843). Use them only with a Harper build and bundled fulltext wrapper that implement query scoring. Do not send these keys to other builds: a successful request alone does not prove scoring was applied. In the pending implementation, an incompatible wrapper returns HTTP 400. + +::: Use `should` to reward additional text matches while retaining the parent condition's required matches: @@ -114,11 +118,17 @@ const products = Product.search(target); A product that matches the required text remains eligible without the optional phrase. A product that matches only an optional clause is excluded. Optional scores add to the required score; they do not change which records match. -Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`: `fuzzyDistance` must be an integer from 0 to 2, and `fuzzyTranspositions` must be a boolean. Each descriptor searches the parent index; an `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The group object itself cannot. Nested `should` lists, negated optional descriptors and scoring controls on the query target, negated conditions or ordinary record conditions are rejected with HTTP 400. +Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`: `fuzzyDistance` must be an integer from 0 to 2, and `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. The server setting `storage.fullText.query.maxFuzzyDistance` defaults to `1` and caps accepted distances; distance `2` requires that setting to be `2`. Each descriptor searches the parent index; an `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The following shapes return HTTP 400: + +- `should` inside an optional descriptor; +- a `not_*` comparator inside `should`; +- `boost` or `should` on the query target, a group object, a negated full-text condition, or an ordinary record condition. + +A positive full-text condition with scoring controls can still combine with negated full-text conditions on the same index and ordinary record filters under `and`. Full-text `or` groups cannot mix full-text and ordinary record conditions. Set `boost` on the parent condition to multiply its required text score, or on an optional descriptor to multiply that bonus. Boosts multiply existing configured source-field weights; they do not replace those weights or require an index rebuild. `boost` is a number and defaults to `1` when omitted. Values round to float32 and must then be finite between 0 and 100 inclusive; invalid values, including negative zero, return HTTP 400. Zero preserves matching while contributing zero score. For a standalone required condition without optional clauses, a positive uniform boost rescales scores without changing relative ranking, apart from floating-point ties; zero makes those scores tie. -Every referenced source must be readable, including optional fields used only for ranking. Omitting `fields` searches all sources and requires permission for all of them. Server-configured full-text query limits, including clause limits, permitted fuzzy modes and edit distances, execution deadlines and result limits, also apply to optional clauses. Requested highlights include matching positive required and optional clauses. +Every referenced source must be readable, including optional fields used only for ranking. Omitting `fields` searches all sources and requires permission for all of them. The same server query limits and result limits apply to optional clauses. Requested highlights include matching positive required and optional clauses. Use the same structured target through HTTP `QUERY`: @@ -275,7 +285,7 @@ Repeat the index parameter to combine the required positive and negated conditio GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather&limit(20) ``` -REST can return configured highlights by selecting `$highlights`. Use [HTTP `QUERY`](../rest/overview.md#tables-and-their-automatic-endpoints), `Table.search()` or `search_by_conditions` for explicit `fields`, condition-level `includeHighlights`, freshness controls, or [query-time relevance controls](#query-time-relevance-controls). +REST can return configured highlights by selecting `$highlights`. Use `Table.search()` or `search_by_conditions` for explicit `fields`, condition-level `includeHighlights`, or freshness controls. For pending [query-time relevance controls](#query-time-relevance-controls), use HTTP `QUERY`, `Table.search()` or `search_by_conditions`. ## Operations API diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 23071454f..616990c0f 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -96,7 +96,7 @@ REST's `limit(start,end)` uses an exclusive end. In v5.4, this prefix query skip GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(100,110) ``` -Use [HTTP `QUERY`](./overview.md#tables-and-their-automatic-endpoints), `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, freshness controls, or [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. +Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. For pending [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls), use HTTP `QUERY`, `Table.search()` or `search_by_conditions`. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries can pass that native window within request budgets; source-version checks and filters run before records count toward the offset and limit. Each page is a new query and can see concurrent changes. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Full-text queries cannot provide an exact total count. On a REST interface with `exactCount: true`, a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). Without that opt-in, Harper applies `count=estimated` instead and can return an estimate. From 1e0f05e54d1ac6a2dda54927646e7485774adb0f Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 31 Oct 2026 13:51:30 -0600 Subject: [PATCH 06/12] docs: separate relevance limitations and validation rules --- reference/full-text-search/querying.md | 6 +++--- reference/resources/query-optimization.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 13ee1c5d3..025eb6fa5 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -87,7 +87,7 @@ Full-text queries cannot provide an exact total after authorization, structured :::note Pending release -These controls describe the pending implementation tracked by [Expose full-text query boosts and optional scoring clauses through Table.search and REST (#3843)](https://github.com/HarperFast/harper/issues/3843). Use them only with a Harper build and bundled fulltext wrapper that implement query scoring. Do not send these keys to other builds: a successful request alone does not prove scoring was applied. In the pending implementation, an incompatible wrapper returns HTTP 400. +These controls describe the pending implementation tracked by [Expose full-text query boosts and optional scoring clauses through Table.search and REST (#3843)](https://github.com/HarperFast/harper/issues/3843). Use them only with a Harper build and bundled fulltext wrapper that implement query scoring. Do not send these keys to other builds. A successful request alone does not prove scoring was applied. In the pending implementation, an incompatible wrapper returns HTTP 400. ::: @@ -118,7 +118,7 @@ const products = Product.search(target); A product that matches the required text remains eligible without the optional phrase. A product that matches only an optional clause is excluded. Optional scores add to the required score; they do not change which records match. -Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`: `fuzzyDistance` must be an integer from 0 to 2, and `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. The server setting `storage.fullText.query.maxFuzzyDistance` defaults to `1` and caps accepted distances; distance `2` requires that setting to be `2`. Each descriptor searches the parent index; an `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The following shapes return HTTP 400: +Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`. `fuzzyDistance` must be an integer from 0 to 2. `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. The server setting `storage.fullText.query.maxFuzzyDistance` defaults to `1` and caps accepted distances. Distance `2` requires that setting to be `2`. Each descriptor searches the parent index. An `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The following shapes return HTTP 400: - `should` inside an optional descriptor; - a `not_*` comparator inside `should`; @@ -126,7 +126,7 @@ Each optional descriptor accepts a positive full-text `comparator`, a nonempty s A positive full-text condition with scoring controls can still combine with negated full-text conditions on the same index and ordinary record filters under `and`. Full-text `or` groups cannot mix full-text and ordinary record conditions. -Set `boost` on the parent condition to multiply its required text score, or on an optional descriptor to multiply that bonus. Boosts multiply existing configured source-field weights; they do not replace those weights or require an index rebuild. `boost` is a number and defaults to `1` when omitted. Values round to float32 and must then be finite between 0 and 100 inclusive; invalid values, including negative zero, return HTTP 400. Zero preserves matching while contributing zero score. For a standalone required condition without optional clauses, a positive uniform boost rescales scores without changing relative ranking, apart from floating-point ties; zero makes those scores tie. +Set `boost` on the parent condition to multiply its required text score, or on an optional descriptor to multiply that bonus. Boosts multiply existing configured source-field weights; they do not replace those weights or require an index rebuild. `boost` is a number and defaults to `1` when omitted. Values round to float32 and must then be finite between 0 and 100 inclusive. Invalid values, including negative zero, return HTTP 400. Zero preserves matching while contributing zero score. For a standalone required condition without optional clauses, a positive uniform boost rescales scores without changing relative ranking, apart from floating-point ties. Zero makes those scores tie. Every referenced source must be readable, including optional fields used only for ranking. Omitting `fields` searches all sources and requires permission for all of them. The same server query limits and result limits apply to optional clauses. Requested highlights include matching positive required and optional clauses. diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 5ac2fa0a2..6c2b10a0c 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -54,7 +54,7 @@ When a query is executed, conditions are evaluated against the database. Indexed A condition using a `FullText` field declared with `@fullText` is executed by the native full-text engine and ranked with BM25, adjusted by any supplied [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) when those pending-release controls are available. Compatible `and` filters narrow the matches. Harper loads current source records and removes stale or deleted candidates before returning them. -Keep result pages bounded. Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded queries can traverse beyond that window within request budgets; an unbounded query can still fail when it exceeds a native window or configured limit. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Use source-field weights to tune index-level ranking; [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls), when available, adjust scores per request without an index rebuild. Enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). +Keep result pages bounded. Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded queries can traverse beyond that window within request budgets. An unbounded query can still fail when it exceeds a native window or configured limit. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Use source-field weights to tune index-level ranking. [Query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls), when available, adjust scores per request without an index rebuild. Enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). ### Indexed vs. Non-Indexed Fields From 3feb5a849d24a65e73d771cd83779df9183cf781 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 31 Oct 2026 15:25:53 -0600 Subject: [PATCH 07/12] docs: clarify pending fuzzy and scoring contracts --- reference/full-text-search/querying.md | 56 ++++++++++++++++++++------ reference/resources/resource-api.md | 2 +- 2 files changed, 45 insertions(+), 13 deletions(-) diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 025eb6fa5..feea638c4 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -17,18 +17,18 @@ const products = Product.search({ `Table.search()` returns an async iterable. Consume it with `for await...of`, or return it directly from a custom Resource to stream the results. -Results are ordered by relevance: BM25 with any supplied [query-time boosts and optional-clause contributions](#query-time-relevance-controls). The `FullText` field, not one of its source fields, identifies the search target. +Results are ordered by BM25 relevance. When the pending [query-time relevance controls](#query-time-relevance-controls) are available, supplied boosts and optional clauses adjust that ranking. The `FullText` field, not one of its source fields, identifies the search target. ## Match modes -| Comparator | Behavior | Required index option | -| ---------------------- | --------------------------------------------------------------------- | --------------------- | -| `matches` | Matches any analyzed query term. | — | -| `matches_all` | Requires every analyzed query term. | — | -| `matches_phrase` | Matches analyzed terms in order as a phrase. | `positions: true` | -| `matches_prefix` | Matches records whose analyzed terms start with the final query term. | `surfaceTerms: true` | -| `matches_fuzzy` | Matches terms within one edit, including a transposition. | — | -| `matches_fuzzy_prefix` | Combines one-edit fuzzy matching with a final-term prefix. | `surfaceTerms: true` | +| Comparator | Behavior | Required index option | +| ---------------------- | ---------------------------------------------------------------------- | --------------------- | +| `matches` | Matches any analyzed query term. | — | +| `matches_all` | Requires every analyzed query term. | — | +| `matches_phrase` | Matches analyzed terms in order as a phrase. | `positions: true` | +| `matches_prefix` | Matches records whose analyzed terms start with the final query term. | `surfaceTerms: true` | +| `matches_fuzzy` | By default, matches terms within one edit, including a transposition. | — | +| `matches_fuzzy_prefix` | By default, combines one-edit fuzzy matching with a final-term prefix. | `surfaceTerms: true` | Every comparator also has a negated form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. @@ -69,7 +69,7 @@ Direct calls through `tables` or `databases` run in a trusted server-side contex ## Score and ordering -Select `$score` to include the relevance score. Without query-time controls, this is the BM25 score; [boosts and optional clauses](#query-time-relevance-controls) adjust it when supplied: +Select `$score` to include the relevance score. Without query-time controls, this is the BM25 score. When the pending [boosts and optional clauses](#query-time-relevance-controls) are available, they adjust it when supplied: ```javascript const products = Product.search({ @@ -118,7 +118,11 @@ const products = Product.search(target); A product that matches the required text remains eligible without the optional phrase. A product that matches only an optional clause is excluded. Optional scores add to the required score; they do not change which records match. -Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`. `fuzzyDistance` must be an integer from 0 to 2. `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. The server setting `storage.fullText.query.maxFuzzyDistance` defaults to `1` and caps accepted distances. Distance `2` requires that setting to be `2`. Each descriptor searches the parent index. An `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The following shapes return HTTP 400: +Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. + +In this pending structured contract, `fuzzyDistance` and `fuzzyTranspositions` can be set on the required positive condition as well as on each optional descriptor. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`. `fuzzyDistance` must be an integer from 0 to 2. `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. The server setting `storage.fullText.query.maxFuzzyDistance` defaults to `1` and caps accepted distances. Distance `2` requires that setting to be `2` and remains preview. Fuzzy-prefix is also preview. Qualify the workload before enabling preview fuzzy options. See [full-text configuration](../configuration/options.md#storage) for the surrounding request budgets. + +Each descriptor searches the parent index. An `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The following shapes return HTTP 400: - `should` inside an optional descriptor; - a `not_*` comparator inside `should`; @@ -289,7 +293,7 @@ REST can return configured highlights by selecting `$highlights`. Use `Table.sea ## Operations API -`search_by_conditions` uses the same condition shape, including the pending-release [`should` and `boost` controls](#query-time-relevance-controls): +`search_by_conditions` uses the same condition shape: ```json { @@ -312,6 +316,34 @@ REST can return configured highlights by selecting `$highlights`. Use `Table.sea } ``` +For the pending [query-time relevance controls](#query-time-relevance-controls), use the following shape only with a compatible Harper build and bundled wrapper: + +```json +{ + "operation": "search_by_conditions", + "database": "catalog", + "table": "Product", + "limit": 20, + "get_attributes": ["id", "name", "$score"], + "conditions": [ + { + "attribute": "catalogSearch", + "comparator": "matches_all", + "value": "waterproof boots", + "boost": 0.5, + "should": [ + { + "comparator": "matches_phrase", + "value": "waterproof boots", + "fields": ["name"], + "boost": 3 + } + ] + } + ] +} +``` + If an index is `unknown`, `needs-rebuild`, or `rebuilding`, the query returns `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. An index in terminal `unavailable` state returns a generic, non-retryable `503`; inspect readiness and logs instead of retrying it as a rebuild. Other busy paths can also return a generic `503`, so clients should branch on the code rather than the status alone. Unreadable source fields return `403`. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. For paging-budget and native response-envelope failures, see [paging and prefix result windows](#paging-and-prefix-result-windows). diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 043eaf94e..f4c9b9b05 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1100,7 +1100,7 @@ Special properties: Full-text queries add two special properties: -- `$score`: When a query uses a full-text index, returns its relevance score, including any [query-time boosts and optional-clause contributions](../full-text-search/querying.md#query-time-relevance-controls). +- `$score`: When a query uses a full-text index, returns its BM25 relevance score. When the pending [query-time boosts and optional clauses](../full-text-search/querying.md#query-time-relevance-controls) are available, their contributions are included. - `$highlights`: Enables and returns full-text fragments and matching spans when the index has highlighting configured. See [Highlights](../full-text-search/querying.md#highlights). #### Selecting related records From c7e9a5b803aece59e702b20811757c1c514b50fa Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 31 Oct 2026 16:03:48 -0600 Subject: [PATCH 08/12] docs: clarify scoring clause limits and fuzzy caps --- reference/full-text-search/querying.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index feea638c4..ce74f76b7 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -120,9 +120,13 @@ A product that matches the required text remains eligible without the optional p Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. -In this pending structured contract, `fuzzyDistance` and `fuzzyTranspositions` can be set on the required positive condition as well as on each optional descriptor. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`. `fuzzyDistance` must be an integer from 0 to 2. `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. The server setting `storage.fullText.query.maxFuzzyDistance` defaults to `1` and caps accepted distances. Distance `2` requires that setting to be `2` and remains preview. Fuzzy-prefix is also preview. Qualify the workload before enabling preview fuzzy options. See [full-text configuration](../configuration/options.md#storage) for the surrounding request budgets. +In this pending structured contract, `fuzzyDistance` and `fuzzyTranspositions` can be set on the required positive condition as well as on each optional descriptor. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`. Supplying them on another match mode returns HTTP 400. `fuzzyDistance` must be an integer from 0 to 2. `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. -Each descriptor searches the parent index. An `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The following shapes return HTTP 400: +The server setting `storage.fullText.query.maxFuzzyDistance` accepts integers from `0` to `2`, defaults to `1`, and caps accepted distances. With a cap of `0`, a fuzzy condition must explicitly set `fuzzyDistance: 0`. Omitting it still defaults to `1` and returns HTTP 400. Distance `2` requires a cap of `2`. + +Qualify the selected fuzzy distance and fuzzy-prefix workload against the actual corpus and concurrency before exposing them to callers. See [full-text configuration](../configuration/options.md#storage) for the surrounding request budgets. + +Each descriptor searches the parent index. An `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty and contain at most `storage.fullText.query.maxQueryClauses` descriptors (default `256`). This is a coarse host precheck: native compilation also counts required clauses, Boolean structure, boosts, analyzed terms and field expansion, so a smaller list can still exceed the compiled-clause budget and return HTTP 400. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The following shapes return HTTP 400: - `should` inside an optional descriptor; - a `not_*` comparator inside `should`; From a5563aa32f085767d380bdd1e92990cd84436251 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 3 Nov 2026 07:02:19 -0700 Subject: [PATCH 09/12] docs: align full-text upgrade guidance with 0.9.0 --- reference/full-text-search/overview.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 21045c943..9e0124d42 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -10,7 +10,7 @@ Harper can build a BM25-ranked full-text index from one or more fields on a tabl Use full-text search when users need relevance-ranked matching across product names, descriptions, tags, articles, or other natural-language fields. It supports term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching. - Starting in Harper v5.4, the native engine is **`@harperfast/fulltext` 0.7.0, embedding Tantivy 0.27.0**. Existing fulltext 0.5.0 and prerelease 0.6 indexes require rebuilding. See [Upgrading existing indexes](#upgrading-existing-indexes) before upgrading a node. + Starting in Harper v5.4, the native engine is **`@harperfast/fulltext` 0.9.0, embedding Tantivy 0.27.0**. Existing fulltext 0.5.0 and prerelease 0.6 indexes require rebuilding. See [Upgrading existing indexes](#upgrading-existing-indexes) before upgrading a node. ## Requirements @@ -59,11 +59,17 @@ Use `describe_table` to inspect each index's declaration and readiness. See [Ins -Harper 5.3.1–5.3.4 ship `@harperfast/fulltext` 0.5.0. Upgrading their current field-based indexes to the 0.7.0 engine in Harper v5.4 requires a full local rebuild. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; plan and validate that migration separately. Prerelease installations using fulltext 0.6 also require rebuilding. This procedure applies to the current field-based declarations; earlier beta declarations and indexes have no supported migration path. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed. +Harper 5.3.1–5.3.4 ship `@harperfast/fulltext` 0.5.0. Upgrading their current field-based indexes to the 0.9.0 engine in Harper v5.4 requires a full local rebuild. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; plan and validate that migration separately. Prerelease installations using fulltext 0.6 also require rebuilding. This procedure applies to the current field-based declarations; earlier beta declarations and indexes have no supported migration path. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed. + +### Upgrading from fulltext 0.7.0 or 0.8.0 + +Current text-only indexes written by fulltext 0.7.0 or 0.8.0 remain compatible with 0.9.0 and can resume from their saved checkpoint without rebuilding. This upgrade adds query controls without changing those indexes' stored identity or adding native filter metadata. Stop all processes sharing a local index before switching versions; do not mix facade/native package versions or run different engines against one live directory. Check readiness, current coverage and representative searches before restoring traffic, as described in the checklist below. + +Returning to fulltext 0.7.0 or 0.8.0 after 0.9.0 has written to the index is not a qualified operator rollback procedure. If returning to Harper 5.4.0-alpha.1 or alpha.2, remove an explicitly configured `storage.fullText.query` block: those versions reject it. Removing that configuration alone does not qualify a downgrade. ### Matching and ranking changes -Tantivy 0.27 changes English stemming even though the schema analyzer remains `english@2`. For example, `added` now stems to `add` instead of `ad`. Queries can match different records or rank them differently after rebuilding. Review representative searches and domain vocabulary before restoring production traffic; the upgrade does not add new Harper query modes or change the declared field options. +Tantivy 0.27 changes English stemming even though the schema analyzer remains `english@2`. For example, `added` now stems to `add` instead of `ad`. Queries can match different records or rank them differently after rebuilding. Review representative searches and domain vocabulary before restoring production traffic; the engine migration does not change the declared field options. Query-time relevance controls are described separately in [Querying](./querying.md#query-time-relevance-controls). ### Upgrade checklist @@ -79,7 +85,7 @@ See [freshness controls](./querying.md#freshness-controls) for the condition fie Keep full-text traffic drained, preserve authoritative data and logs, and correct the reported package, storage or coordination failure before restarting the same v5.4 Harper version. An interrupted rebuild can recover from authoritative records. Do not force readiness or edit, move or delete native index files by hand. -**Direct downgrade to fulltext 0.5.x is unsupported.** That engine refuses both opening and resetting a 0.7 index identity. If an older version has already been started, stop it and restore the 0.7.0-based Harper version before checking readiness and catch-up again. This is recovery with the current engine, not a supported operator downgrade procedure. If recovery also fails, keep traffic drained and investigate the readiness reason and node logs. +**Direct downgrade to fulltext 0.5.x is unsupported.** That engine refuses both opening and resetting the current Tantivy 0.27 index identity. If an older version has already been started, stop it and restore the 0.9.0-based Harper version before checking readiness and catch-up again. This is recovery with the current engine, not a supported operator downgrade procedure. If recovery also fails, keep traffic drained and investigate the readiness reason and node logs. ## Write backpressure From 023f6a9ddeb96130571d8126f63568e0a26a2bb8 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 3 Nov 2026 07:15:03 -0700 Subject: [PATCH 10/12] docs: reconcile full-text versions and query policy across references --- reference/configuration/options.md | 5 +++++ reference/full-text-search/configuration.md | 2 +- reference/full-text-search/overview.md | 8 ++++---- reference/full-text-search/querying.md | 2 +- reference/operations-api/operations.md | 2 +- reference/resources/query-optimization.md | 4 ++-- reference/resources/resource-api.md | 2 ++ release-notes/v5-lincoln/5.4.md | 10 ++++++++-- 8 files changed, 24 insertions(+), 11 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index ee62df608..c7651e18c 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -316,6 +316,9 @@ storage: - `maxRawHitsPerRequest` — Maximum fetched hits per request, including bounded prefix replay when traversal is needed; default `10000`. - `maxResultsPerRequest` — Maximum retained offset plus limit. A request must also fit the separate raw-hit budget; default `10000`. - `requestBudgetMilliseconds` — Execution budget across native batches, source checks and highlights, capped by the native execution limit; default `5000` ms. Reader acquisition and freshness waiting have separate bounds. + - `query.maxQueryTerms` — Maximum analyzed terms per full-text leaf; integer `1`–`1024`, default `64`, also bounded by the loaded native runtime. + - `query.maxQueryClauses` — Maximum compiled clauses, including Boolean structure and field expansion; integer `1`–`4096`, default `256`, also bounded by the loaded native runtime. + - `query.maxFuzzyDistance` — Maximum accepted fuzzy edit distance; integer `0`–`2`, default `1`. A cap of `0` requires explicit `fuzzyDistance: 0` on fuzzy conditions; the omitted distance still defaults to `1` and is rejected. Distance `2` requires operator opt-in and workload qualification. - `runtime.maxResidentIndexes` — Maximum resident native indexes; default `128`. - `runtime.maxIndexingThreads` — Maximum aggregate indexing-thread reservations across resident native indexes; default `128`. - `runtime.maxSearchThreads` — Maximum aggregate search-thread reservations across resident native indexes; default `256`. @@ -323,6 +326,8 @@ storage: - `runtime.maxQueuedBytes` — Aggregate queued-write reservations across resident native indexes; default `8589934592` (8 GiB). - `runtime.maxExpensiveSearches` — Maximum concurrent expensive native searches; default `8`. + The `query` policy is captured for the whole process and inherited by application and HTTP workers. A full Harper process restart is required; restarting only workers does not change it. Omitted keys keep their defaults. Requests cannot raise the policy. It changes query admission, not persisted index identity. Harper 5.4.0-alpha.1/alpha.2 reject an explicit `storage.fullText.query` block. + The runtime budgets apply together and are not hard bounds on total process memory. See [Set Configuration](./operations.md#set-configuration) for when request and native runtime changes take effect. Use the usual underscore-separated parameter names, such as `storage_fullText_maxRawHitsPerRequest`. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for costs; see [readiness and recovery](../resources/query-optimization.md#readiness-and-recovery) for failures. - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md index f7434c676..ffb04df2d 100644 --- a/reference/full-text-search/configuration.md +++ b/reference/full-text-search/configuration.md @@ -147,7 +147,7 @@ type Document @table(audit: true) { ## Schema changes and rebuilds -The `english@2` name remains unchanged in v5.4, but Tantivy 0.27 changes some English stems. Existing indexes must rebuild, and matching or ranking can change. See [upgrading existing indexes](./overview.md#upgrading-existing-indexes). +The `english@2` name remains unchanged in v5.4, but Tantivy 0.27 changes some English stems. Indexes from fulltext 0.5.0 or prerelease 0.6 must rebuild, and matching or ranking can change. Current Harper indexes written by fulltext 0.7.0 or 0.8.0 already use Tantivy 0.27 and can reuse compatible checkpoints with 0.9.0. See [upgrading existing indexes](./overview.md#upgrading-existing-indexes). Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the `fields` list in a stable order when its meaning has not changed. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 9e0124d42..babed5164 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -63,18 +63,18 @@ Harper 5.3.1–5.3.4 ship `@harperfast/fulltext` 0.5.0. Upgrading their current ### Upgrading from fulltext 0.7.0 or 0.8.0 -Current text-only indexes written by fulltext 0.7.0 or 0.8.0 remain compatible with 0.9.0 and can resume from their saved checkpoint without rebuilding. This upgrade adds query controls without changing those indexes' stored identity or adding native filter metadata. Stop all processes sharing a local index before switching versions; do not mix facade/native package versions or run different engines against one live directory. Check readiness, current coverage and representative searches before restoring traffic, as described in the checklist below. +Harper 5.4.0-alpha.1 and alpha.2 ship fulltext 0.7.0. Their existing indexes, and indexes created by development builds using fulltext 0.8.0, remain compatible with 0.9.0 and can resume from their saved checkpoint without rebuilding. These are indexes created through Harper's `@fullText` declaration; Harper does not create native filter metadata. Query controls do not change their stored identity. Stop all processes sharing a local index before switching versions; do not mix facade/native package versions or run different engines against one live directory. Check readiness, current coverage and representative searches before restoring traffic, as described in the checklist below. -Returning to fulltext 0.7.0 or 0.8.0 after 0.9.0 has written to the index is not a qualified operator rollback procedure. If returning to Harper 5.4.0-alpha.1 or alpha.2, remove an explicitly configured `storage.fullText.query` block: those versions reject it. Removing that configuration alone does not qualify a downgrade. +Returning to fulltext 0.7.0 or 0.8.0 after 0.9.0 has written to the index is not a qualified operator rollback procedure. If returning to Harper 5.4.0-alpha.1 or alpha.2, remove an explicitly configured `storage.fullText.query` block: those versions reject it. Removing that configuration alone does not qualify a downgrade. If an older build was already started, stop it, restore the 0.9.0-based Harper build, and check readiness and current coverage before restoring full-text traffic. ### Matching and ranking changes -Tantivy 0.27 changes English stemming even though the schema analyzer remains `english@2`. For example, `added` now stems to `add` instead of `ad`. Queries can match different records or rank them differently after rebuilding. Review representative searches and domain vocabulary before restoring production traffic; the engine migration does not change the declared field options. Query-time relevance controls are described separately in [Querying](./querying.md#query-time-relevance-controls). +When upgrading from fulltext 0.5.0 or prerelease 0.6, Tantivy 0.27 changes English stemming even though the schema analyzer remains `english@2`. For example, `added` now stems to `add` instead of `ad`. Queries can match different records or rank them differently after rebuilding. Review representative searches and domain vocabulary before restoring production traffic; the engine migration does not change the declared field options. Query-time relevance controls are described separately in [Querying](./querying.md#query-time-relevance-controls). ### Upgrade checklist 1. Drain the node's full-text requests and stop all old Harper processes that share its storage, including background workers. Do not run old and new engine versions against one local index. -2. Confirm authoritative records and recovery backups are available. Reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. Measure rebuild duration and disk growth with representative data before setting the rollout window. +2. Confirm authoritative records and recovery backups are available. For fulltext 0.5.0 or prerelease 0.6, reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. Compatible 0.7/0.8 indexes normally resume from checkpoint; missing, corrupt or unrecoverably lagging files can still require rebuilding. Plan rebuild capacity for that recovery case. 3. Start the v5.4 Harper version. Use [`describe_table`](../operations-api/operations.md#describe_table) for each affected table, using an authorized operator account that already has read access to every indexed source field, and check every `full_text_indexes[].readiness.state`. Metadata is permission-filtered; a missing entry does not prove that no index exists. Keep full-text traffic drained until all required indexes are `ready`. 4. Run representative searches through the same REST or permission-checked application path and roles used in production, with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed on the queried node before the search; they do not wait for replication delivery, and readiness alone does not establish that coverage. Check response bodies for terminal stream errors as well as HTTP status. A JSON-array response can start with HTTP 200 and end with an `error` element. 5. Retry write responses carrying `DERIVED_INDEX_LAGGING` with bounded backoff while replay catches up. Restore full-text traffic only after readiness and current-coverage searches succeed. A node-by-node upgrade can use other ready replicas only if the application's routing supports it; the rebuilding node remains unavailable for full-text search. During a mixed-version rollout, matching and ranking can differ between replicas. Pre-v5.4 replicas also reject prefix requests whose `offset + limit` exceeds 100 with `400`. Route related searches consistently and check the application's behavior before using this approach. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index ce74f76b7..15fdcd069 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -120,7 +120,7 @@ A product that matches the required text remains eligible without the optional p Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. -In this pending structured contract, `fuzzyDistance` and `fuzzyTranspositions` can be set on the required positive condition as well as on each optional descriptor. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`. Supplying them on another match mode returns HTTP 400. `fuzzyDistance` must be an integer from 0 to 2. `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. +In this pending structured contract, `fuzzyDistance` and `fuzzyTranspositions` can be set on the required positive condition as well as on each optional descriptor. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`. Supplying them on another match mode or a condition group returns HTTP 400. Values with the wrong type also return HTTP 400. `fuzzyDistance` must be an integer from 0 to 2. `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. The server setting `storage.fullText.query.maxFuzzyDistance` accepts integers from `0` to `2`, defaults to `1`, and caps accepted distances. With a cap of `0`, a fuzzy condition must explicitly set `fuzzyDistance: 0`. Omitting it still defaults to `1` and returns HTTP 400. Distance `2` requires a cap of `2`. diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index ff24b4d3c..c369db028 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -317,7 +317,7 @@ Returns records matching one or more conditions. Supports `operator` (`and`/`or` Full-text conditions (Added in: v5.3.0): -Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. It also preserves the pending-release [`should` and `boost` relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). +Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. It also preserves the pending-release [`should`, `boost`, `fuzzyDistance` and `fuzzyTranspositions` relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). --- diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 6c2b10a0c..f4a4d81f5 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -115,7 +115,7 @@ Replacement workers keep the captured budgets after a configuration change. See Full-text indexes require audit logging. New schema tables must explicitly declare `@table(audit: true)` when adding `@fullText`, even if global audit logging is enabled; an existing audited table can add an index without restating `audit`. Setting `@table(audit: false)` while an index is declared is rejected during schema registration. -Harper v5.4 uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Existing 0.5.0 and prerelease 0.6 indexes using current field-based declarations need an automatic local rebuild after this engine upgrade. Follow [upgrading existing indexes](../full-text-search/overview.md#upgrading-existing-indexes) for affected versions, draining traffic, readiness checks, matching changes and current-engine recovery. Earlier beta declarations and indexes have no supported migration path. See [write backpressure](../full-text-search/overview.md#write-backpressure) for local write refusal and retry guidance. +Harper v5.4 uses `@harperfast/fulltext` 0.9.0 with Tantivy 0.27.0. Existing 0.5.0 and prerelease 0.6 indexes using current field-based declarations need an automatic local rebuild after this engine upgrade. Follow [upgrading existing indexes](../full-text-search/overview.md#upgrading-existing-indexes) for affected versions, draining traffic, readiness checks, matching changes and current-engine recovery. Earlier beta declarations and indexes have no supported migration path. See [write backpressure](../full-text-search/overview.md#write-backpressure) for local write refusal and retry guidance. | Failure | HTTP status | Code and retry guidance | | ------------------------------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -136,7 +136,7 @@ Concurrent requests each have their own work budget. These per-request budgets d `INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Use [`describe_table`](../operations-api/operations.md#describe_table) to inspect `full_text_indexes[].readiness`, including its state and reason. Stop aggressive transaction-log cleanup and correct the reported configuration or storage problem before restarting the same Harper version to retry activation. Keep full-text traffic drained until readiness and required coverage recover; do not edit native files. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. -Native page, response-byte and highlight limits still apply. The fulltext 0.7.0 library limits each analyzed query leaf to 64 terms and the compiled query to 256 clauses. Native invalid-query validation returns 400 before response headers; the streaming-error guidance above still applies. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.7.0/docs/limits.md) for page, byte and highlight bounds. +Native page, response-byte and highlight limits still apply. The fulltext 0.9.0 defaults are 64 analyzed terms per leaf and 256 compiled clauses per query. The separate `storage.fullText.query` policy configures those ceilings and the maximum fuzzy distance; see [storage configuration](../configuration/options.md#storage). Native invalid-query validation returns 400 before response headers; the streaming-error guidance above still applies. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.9.0/docs/limits.md) for page, byte and highlight bounds. ## Relationships and Joins diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index f4c9b9b05..6f9b94048 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1037,6 +1037,8 @@ Array of condition objects to filter records. Each condition: | `fields` | Full-text source fields to search; every name must belong to the selected full-text index | | `boost` | Optional full-text score multiplier (`number`, default `1`); see pending-release [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | | `should` | Optional nonempty array of additional scoring clauses; omitted by default; see pending-release [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | +| `fuzzyDistance` | Integer edit distance for a positive `matches_fuzzy` or `matches_fuzzy_prefix` condition; see the [pending fuzzy contract](../full-text-search/querying.md#query-time-relevance-controls) and server cap. | +| `fuzzyTranspositions` | Boolean controlling whether adjacent transpositions count as one edit on a positive fuzzy condition; see the [pending fuzzy contract](../full-text-search/querying.md#query-time-relevance-controls). | | `includeHighlights` | Include configured full-text highlights | | `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; see [freshness controls](../full-text-search/querying.md#freshness-controls) | | `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage; see [freshness controls](../full-text-search/querying.md#freshness-controls) | diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 916e1b5af..a34be3265 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -150,9 +150,15 @@ Harper Pro's CPU profiler used to sample every worker continuously, so each work ### Tantivy 0.27 and Existing Index Rebuilds -Harper now uses `@harperfast/fulltext` 0.7.0 with Tantivy 0.27.0. Harper 5.3.1–5.3.4 ship fulltext 0.5.0. Existing 0.5.0 indexes using current field-based declarations, and prerelease 0.6 indexes, rebuild automatically from authoritative records. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; validate that migration separately. Earlier beta declarations and indexes have no supported migration path. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported. +Harper now uses `@harperfast/fulltext` 0.9.0 with Tantivy 0.27.0. Harper 5.3.1–5.3.4 ship fulltext 0.5.0. Existing 0.5.0 indexes using current field-based declarations, and prerelease 0.6 indexes, rebuild automatically from authoritative records. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; validate that migration separately. Earlier beta declarations and indexes have no supported migration path. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported. -English stemming changes despite the unchanged `english@2` analyzer name: `added` now stems to `add` rather than `ad`. Matching and ranking can change. Query modes and schema options are unchanged; benchmark results from the wrapper do not establish application performance parity. Review representative searches and see [upgrading existing indexes](/reference/v5/full-text-search/overview#upgrading-existing-indexes) before restoring traffic. +For the 0.5/0.6 rebuild path, English stemming changes despite the unchanged `english@2` analyzer name: `added` now stems to `add` rather than `ad`. Matching and ranking can change. Query modes and schema options are unchanged; benchmark results from the wrapper do not establish application performance parity. Review representative searches and see [upgrading existing indexes](/reference/v5/full-text-search/overview#upgrading-existing-indexes) before restoring traffic. + +Harper 5.4.0-alpha.1/alpha.2 indexes using fulltext 0.7.0, and current Harper indexes written by development builds with 0.8.0, can reuse compatible checkpoints with 0.9.0. See [upgrading from fulltext 0.7.0 or 0.8.0](/reference/v5/full-text-search/overview#upgrading-from-fulltext-070-or-080) for process coordination and the unqualified rollback caveat. + +### Pending Query-Time Relevance Controls + +The pending [boost and optional-clause contract](/reference/v5/full-text-search/querying#query-time-relevance-controls) reserves `boost` and `should` as query-object keys on all tables. Misplaced controls return HTTP 400, even without a full-text index; applications that previously passed ignored custom keys must rename them. An attribute named `boost` or `should` can still be queried normally through the `attribute` property. ### Paging and Search Work Limits From 91b33a02c2733b635978c34912ccbe219d2f44e9 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 3 Nov 2026 07:28:11 -0700 Subject: [PATCH 11/12] docs: distinguish fuzzy availability and full-process query-policy restart --- reference/configuration/operations.md | 4 ++-- reference/full-text-search/querying.md | 16 ++++++++++++---- reference/operations-api/operations.md | 4 ++-- reference/resources/query-optimization.md | 4 ++-- reference/resources/resource-api.md | 6 +++--- release-notes/v5-lincoln/5.4.md | 6 +----- 6 files changed, 22 insertions(+), 18 deletions(-) diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index faea62523..520fca429 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -18,7 +18,7 @@ For the full list of configurable options, see [Configuration Options](./options Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`). A change to `replication.threads` takes effect only on a full `restart`. -Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). +Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.query` or `storage.fullText.runtime` require a full Harper process restart; `restart_service` replaces neither the captured query policy nor native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). `operation` _(required)_ — must be `set_configuration` @@ -26,7 +26,7 @@ Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage Additional properties correspond to configuration keys in underscore-separated format (e.g. `logging_level` for `logging.level`, `clustering_enabled` for `clustering.enabled`). -For full-text settings, use individual parameter names such as `storage_fullText_maxRawHitsPerRequest` or `storage_fullText_runtime_maxSearchThreads`. `set_configuration` does not accept a whole `storage_fullText` object; use `storage.fullText` as the section name only in the YAML configuration file. +For full-text settings, use individual parameter names such as `storage_fullText_maxRawHitsPerRequest`, `storage_fullText_query_maxQueryTerms`, `storage_fullText_query_maxQueryClauses`, `storage_fullText_query_maxFuzzyDistance`, or `storage_fullText_runtime_maxSearchThreads`. `set_configuration` does not accept a whole `storage_fullText` object; use `storage.fullText` as the section name only in the YAML configuration file. :::warning Replicate cluster-appropriate parameters only A replicated call sends the exact same values to every node. Do not include node-local parameters — ports, `node.hostname`, `rootPath`, storage/log file paths, TLS certificate or key paths, or `replication.hostname`/`url`/`routes` — as they would overwrite each peer's local values. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 15fdcd069..101675a6a 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -83,6 +83,16 @@ Full-text results use descending relevance order. The only explicit full-text so Full-text queries cannot provide an exact total after authorization, structured filtering, and current-record validation. On a REST interface with `exactCount` enabled, `Prefer: count=exact` reports an unavailable total; otherwise it is downgraded to an estimated count. See [pagination and total count](../rest/querying.md#pagination-and-total-count). `Table.getRecordCount()` counts table records, not matches for a full-text query. +## Fuzzy controls in v5.4 + + + +`fuzzyDistance` and `fuzzyTranspositions` can be set on structured `matches_fuzzy` and `matches_fuzzy_prefix` conditions, including their negated forms. Supplying them on another match mode or a group object returns HTTP 400; put the controls on its individual fuzzy conditions. Values with the wrong type also return HTTP 400. `fuzzyDistance` must be an integer from 0 to 2. `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. + +The server setting `storage.fullText.query.maxFuzzyDistance` accepts integers from `0` to `2`, defaults to `1`, and caps accepted distances. With a cap of `0`, a fuzzy condition must explicitly set `fuzzyDistance: 0`. Omitting it still defaults to `1` and returns HTTP 400. Distance `2` requires a cap of `2`. + +Qualify the selected fuzzy distance and fuzzy-prefix workload against the actual corpus and concurrency before exposing them to callers. See [full-text configuration](../configuration/options.md#storage) for the surrounding request budgets. + ## Query-time relevance controls :::note Pending release @@ -120,11 +130,9 @@ A product that matches the required text remains eligible without the optional p Each optional descriptor accepts a positive full-text `comparator`, a nonempty string `value`, optional `fields`, `boost`, `fuzzyDistance`, and `fuzzyTranspositions`. -In this pending structured contract, `fuzzyDistance` and `fuzzyTranspositions` can be set on the required positive condition as well as on each optional descriptor. The fuzzy controls apply only to `matches_fuzzy` and `matches_fuzzy_prefix`. Supplying them on another match mode or a condition group returns HTTP 400. Values with the wrong type also return HTTP 400. `fuzzyDistance` must be an integer from 0 to 2. `fuzzyTranspositions` must be a boolean. Omitted values default to `1` and `true`, respectively. +Optional descriptors use the same [fuzzy controls](#fuzzy-controls-in-v54) as required fuzzy conditions, including their configured distance cap. Optional descriptors remain positive-only. -The server setting `storage.fullText.query.maxFuzzyDistance` accepts integers from `0` to `2`, defaults to `1`, and caps accepted distances. With a cap of `0`, a fuzzy condition must explicitly set `fuzzyDistance: 0`. Omitting it still defaults to `1` and returns HTTP 400. Distance `2` requires a cap of `2`. - -Qualify the selected fuzzy distance and fuzzy-prefix workload against the actual corpus and concurrency before exposing them to callers. See [full-text configuration](../configuration/options.md#storage) for the surrounding request budgets. +`boost` and `should` are reserved query-object keys on all tables when these pending controls are available. Misplaced controls return HTTP 400 even without a full-text index; applications that previously passed ignored custom keys must rename them. An attribute named `boost` or `should` can still be queried through the `attribute` property. Each descriptor searches the parent index. An `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty and contain at most `storage.fullText.query.maxQueryClauses` descriptors (default `256`). This is a coarse host precheck: native compilation also counts required clauses, Boolean structure, boosts, analyzed terms and field expansion, so a smaller list can still exceed the compiled-clause budget and return HTTP 400. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The following shapes return HTTP 400: diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index c369db028..a2bac7e9f 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -317,7 +317,7 @@ Returns records matching one or more conditions. Supports `operator` (`and`/`or` Full-text conditions (Added in: v5.3.0): -Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. It also preserves the pending-release [`should`, `boost`, `fuzzyDistance` and `fuzzyTranspositions` relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). +Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. It preserves the [v5.4 fuzzy controls](../full-text-search/querying.md#fuzzy-controls-in-v54), `fuzzyDistance` and `fuzzyTranspositions`, and the pending-release [`should` and `boost` relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). --- @@ -1845,7 +1845,7 @@ Detailed documentation: [Configuration Overview](../configuration/overview.md) Updates configuration parameters in `harper-config.yaml`. Most changes require a restart (`restart` or `restart_service`) to take effect. See [Set Configuration](../configuration/operations.md#set-configuration) for full-text restart requirements. -Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags) or `storage.fullText.runtime.*` take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. +Supports `"replicated": true` to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time); `restart_service` restarts only worker threads, though, so changes to [`threads.v8Flags`](../configuration/options.md#applying-v8-flags), `storage.fullText.query.*` or `storage.fullText.runtime.*` take effect only after a `restart` sent to each node in turn (`restart` does not replicate). See [Configuration Operations](../configuration/operations.md#set-configuration) for details. ```json { diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index f4a4d81f5..319d3fefd 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -103,7 +103,7 @@ Use a bounded `limit` for large result sets. An unbounded search that exceeds it The fetched-hit budget includes the initial batch replay when Harper switches to native traversal. It counts hits returned by the native library, not every record Tantivy visits while scoring. Exhausting that budget throws a 400 error; exhausting the execution budget throws 503. A budget failure is a failed query, not a completed short page. If a REST response has already started streaming, a later query error ends that stream instead of changing its HTTP status. JSON-array clients must check for a terminal object shaped like `{ "error": "ErrorName: message" }` and treat the response as failed. For other formats, detect stream or decoding failures; status 200 and count headers alone do not prove completion. See [REST paging](../rest/querying.md#pagination-and-total-count). When there are no further native matches, Harper returns the eligible records found, which may be an empty result. This differs from reaching a budget while more matches remain. -Configure the three request limits and six native process budgets under [`storage.fullText`](../configuration/options.md#storage). See [Set Configuration](../configuration/operations.md#set-configuration) for restart requirements. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. +Configure the request work limits, query admission policy and native process budgets under [`storage.fullText`](../configuration/options.md#storage). See [Set Configuration](../configuration/operations.md#set-configuration) for restart requirements. The execution budget is not a hard response-time guarantee: reader acquisition, query construction and some native operations can take additional time. Freshness waiting and reader acquisition use separate bounds. ### Native runtime configuration @@ -136,7 +136,7 @@ Concurrent requests each have their own work budget. These per-request budgets d `INDEX_REBUILDING` and `DERIVED_INDEX_LAGGING` identify retryable readiness or coverage failures. Retry with backoff while the index makes progress. Transaction history retained under [`logging.auditRetention`](../logging/configuration.md#loggingauditretention) (default `3d`) is used for catch-up. Lost history can require a rebuild; retries alone do not repair an index that cannot catch up. Use [`describe_table`](../operations-api/operations.md#describe_table) to inspect `full_text_indexes[].readiness`, including its state and reason. Stop aggressive transaction-log cleanup and correct the reported configuration or storage problem before restarting the same Harper version to retry activation. Keep full-text traffic drained until readiness and required coverage recover; do not edit native files. Budget and native-busy failures do not carry those readiness codes. A native busy response also returns 503; reduce concurrency or retry with backoff. For an execution-budget failure, narrow the query, reduce the offset or adjust the work budget instead of repeatedly submitting the same expensive query. -Native page, response-byte and highlight limits still apply. The fulltext 0.9.0 defaults are 64 analyzed terms per leaf and 256 compiled clauses per query. The separate `storage.fullText.query` policy configures those ceilings and the maximum fuzzy distance; see [storage configuration](../configuration/options.md#storage). Native invalid-query validation returns 400 before response headers; the streaming-error guidance above still applies. Raising Harper's work budgets does not raise those native limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.9.0/docs/limits.md) for page, byte and highlight bounds. +Native page, response-byte and highlight limits still apply. The fulltext 0.9.0 defaults are 64 analyzed terms per leaf and 256 compiled clauses per query. The separate `storage.fullText.query` policy configures those ceilings and the maximum fuzzy distance; see [storage configuration](../configuration/options.md#storage). Native invalid-query validation returns 400 before response headers; the streaming-error guidance above still applies. Changing request work budgets does not change the query policy or the native page, byte and highlight limits. See the [published native limits](https://github.com/HarperFast/fulltext/blob/v0.9.0/docs/limits.md) for page, byte and highlight bounds. ## Relationships and Joins diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 6f9b94048..45cb2c292 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1037,13 +1037,13 @@ Array of condition objects to filter records. Each condition: | `fields` | Full-text source fields to search; every name must belong to the selected full-text index | | `boost` | Optional full-text score multiplier (`number`, default `1`); see pending-release [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | | `should` | Optional nonempty array of additional scoring clauses; omitted by default; see pending-release [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | -| `fuzzyDistance` | Integer edit distance for a positive `matches_fuzzy` or `matches_fuzzy_prefix` condition; see the [pending fuzzy contract](../full-text-search/querying.md#query-time-relevance-controls) and server cap. | -| `fuzzyTranspositions` | Boolean controlling whether adjacent transpositions count as one edit on a positive fuzzy condition; see the [pending fuzzy contract](../full-text-search/querying.md#query-time-relevance-controls). | +| `fuzzyDistance` | Integer edit distance for a fuzzy condition, including negated forms; see [v5.4 fuzzy controls](../full-text-search/querying.md#fuzzy-controls-in-v54) and the server cap. | +| `fuzzyTranspositions` | Boolean controlling whether adjacent transpositions count as one edit on a fuzzy condition; see [v5.4 fuzzy controls](../full-text-search/querying.md#fuzzy-controls-in-v54). | | `includeHighlights` | Include configured full-text highlights | | `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; see [freshness controls](../full-text-search/querying.md#freshness-controls) | | `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage; see [freshness controls](../full-text-search/querying.md#freshness-controls) | -The `fields`, `includeHighlights`, `maxIndexLagMilliseconds` and `waitForIndexMilliseconds` properties apply to full-text conditions (v5.3.0). Availability and constraints for `boost` and `should` are described in [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. +The `fields`, `includeHighlights`, `maxIndexLagMilliseconds` and `waitForIndexMilliseconds` properties apply to full-text conditions (v5.3.0). The `fuzzyDistance` and `fuzzyTranspositions` properties apply to fuzzy conditions in v5.4. Availability and constraints for `boost` and `should` are described in [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. Each comparator also has a `not_` form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. A negated condition requires a positive full-text condition on the same index. See [Full-text query execution](./query-optimization.md#full-text-search-work-and-paging) for paging costs and work limits. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index a34be3265..5d10626b7 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -156,13 +156,9 @@ For the 0.5/0.6 rebuild path, English stemming changes despite the unchanged `en Harper 5.4.0-alpha.1/alpha.2 indexes using fulltext 0.7.0, and current Harper indexes written by development builds with 0.8.0, can reuse compatible checkpoints with 0.9.0. See [upgrading from fulltext 0.7.0 or 0.8.0](/reference/v5/full-text-search/overview#upgrading-from-fulltext-070-or-080) for process coordination and the unqualified rollback caveat. -### Pending Query-Time Relevance Controls - -The pending [boost and optional-clause contract](/reference/v5/full-text-search/querying#query-time-relevance-controls) reserves `boost` and `should` as query-object keys on all tables. Misplaced controls return HTTP 400, even without a full-text index; applications that previously passed ignored custom keys must rename them. An attribute named `boost` or `should` can still be queried normally through the `attribute` property. - ### Paging and Search Work Limits -Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. The `storage.fullText.runtime` section also provides process-wide native-capacity budgets; changes to those budgets require a full Harper process restart. Separate page requests run independently and can see changes to records or index contents. +Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. The `storage.fullText.runtime` section also provides process-wide native-capacity budgets; changes to those budgets require a full Harper process restart. The `storage.fullText.query` policy configures analyzed terms per leaf, compiled clause count and maximum fuzzy distance; it also requires a full Harper process restart. Defaults remain 64 terms, 256 clauses and distance one. Harper 5.4.0-alpha.1/alpha.2 reject an explicit query-policy block. Separate page requests run independently and can see changes to records or index contents. See [Full-text query execution](/reference/v5/resources/query-optimization#full-text-search-work-and-paging) and [storage configuration](/reference/v5/configuration/options#storage) for limits, restart requirements and recovery guidance. From 7fbade6f44c663f48bd1f9fa2234a4b91c39a2e3 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 3 Nov 2026 07:58:05 -0700 Subject: [PATCH 12/12] docs: assign full-text scoring controls to Harper 5.4 --- reference/full-text-search/overview.md | 2 +- reference/full-text-search/querying.md | 16 +++++++--------- reference/operations-api/operations.md | 2 +- reference/resources/resource-api.md | 8 ++++---- reference/rest/querying.md | 2 +- release-notes/v5-lincoln/5.4.md | 6 ++++++ 6 files changed, 20 insertions(+), 16 deletions(-) diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index babed5164..b026967d9 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -95,7 +95,7 @@ Retry rejected writes with backoff rather than immediately adding more load. Ins ## Ranking and result loading -The native index ranks matching documents with BM25, adjusted by any supplied [query-time boosts and optional-clause contributions](./querying.md#query-time-relevance-controls) when those pending-release controls are available. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. +The native index ranks matching documents with BM25, adjusted by any supplied [query-time boosts and optional-clause contributions](./querying.md#query-time-relevance-controls) in v5.4. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. Full-text search can be combined with structured filters. Harper checks conditions against current source records before returning them. In v5.4, additional ranked batches can fill a filtered page within request budgets; exhausting a budget returns an error rather than a budget-truncated page. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 101675a6a..20a85fb93 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -17,7 +17,7 @@ const products = Product.search({ `Table.search()` returns an async iterable. Consume it with `for await...of`, or return it directly from a custom Resource to stream the results. -Results are ordered by BM25 relevance. When the pending [query-time relevance controls](#query-time-relevance-controls) are available, supplied boosts and optional clauses adjust that ranking. The `FullText` field, not one of its source fields, identifies the search target. +Results are ordered by BM25 relevance. In v5.4, supplied [query-time boosts and optional clauses](#query-time-relevance-controls) adjust that ranking. The `FullText` field, not one of its source fields, identifies the search target. ## Match modes @@ -69,7 +69,7 @@ Direct calls through `tables` or `databases` run in a trusted server-side contex ## Score and ordering -Select `$score` to include the relevance score. Without query-time controls, this is the BM25 score. When the pending [boosts and optional clauses](#query-time-relevance-controls) are available, they adjust it when supplied: +Select `$score` to include the relevance score. Without query-time controls, this is the BM25 score. In v5.4, [boosts and optional clauses](#query-time-relevance-controls) adjust it when supplied: ```javascript const products = Product.search({ @@ -95,11 +95,9 @@ Qualify the selected fuzzy distance and fuzzy-prefix workload against the actual ## Query-time relevance controls -:::note Pending release - -These controls describe the pending implementation tracked by [Expose full-text query boosts and optional scoring clauses through Table.search and REST (#3843)](https://github.com/HarperFast/harper/issues/3843). Use them only with a Harper build and bundled fulltext wrapper that implement query scoring. Do not send these keys to other builds. A successful request alone does not prove scoring was applied. In the pending implementation, an incompatible wrapper returns HTTP 400. + -::: +These controls are available in Harper 5.4 with its bundled fulltext wrapper. Earlier 5.4 prerelease builds may not include them. Do not send these keys to builds without query-scoring support: a successful request alone does not prove scoring was applied. A scoring-capable Harper build with an incompatible wrapper returns HTTP 400. Use `should` to reward additional text matches while retaining the parent condition's required matches: @@ -132,7 +130,7 @@ Each optional descriptor accepts a positive full-text `comparator`, a nonempty s Optional descriptors use the same [fuzzy controls](#fuzzy-controls-in-v54) as required fuzzy conditions, including their configured distance cap. Optional descriptors remain positive-only. -`boost` and `should` are reserved query-object keys on all tables when these pending controls are available. Misplaced controls return HTTP 400 even without a full-text index; applications that previously passed ignored custom keys must rename them. An attribute named `boost` or `should` can still be queried through the `attribute` property. +`boost` and `should` are reserved query-object keys on all tables in v5.4. Misplaced controls return HTTP 400 even without a full-text index; applications that previously passed ignored custom keys must rename them. An attribute named `boost` or `should` can still be queried through the `attribute` property. Each descriptor searches the parent index. An `attribute` property is rejected even if it repeats the parent index name. The list must be nonempty and contain at most `storage.fullText.query.maxQueryClauses` descriptors (default `256`). This is a coarse host precheck: native compilation also counts required clauses, Boolean structure, boosts, analyzed terms and field expansion, so a smaller list can still exceed the compiled-clause budget and return HTTP 400. A positive full-text leaf condition inside an `and` or `or` group can use these controls. The following shapes return HTTP 400: @@ -301,7 +299,7 @@ Repeat the index parameter to combine the required positive and negated conditio GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather&limit(20) ``` -REST can return configured highlights by selecting `$highlights`. Use `Table.search()` or `search_by_conditions` for explicit `fields`, condition-level `includeHighlights`, or freshness controls. For pending [query-time relevance controls](#query-time-relevance-controls), use HTTP `QUERY`, `Table.search()` or `search_by_conditions`. +REST can return configured highlights by selecting `$highlights`. Use `Table.search()` or `search_by_conditions` for explicit `fields`, condition-level `includeHighlights`, or freshness controls. For v5.4 [query-time relevance controls](#query-time-relevance-controls), use HTTP `QUERY`, `Table.search()` or `search_by_conditions`. ## Operations API @@ -328,7 +326,7 @@ REST can return configured highlights by selecting `$highlights`. Use `Table.sea } ``` -For the pending [query-time relevance controls](#query-time-relevance-controls), use the following shape only with a compatible Harper build and bundled wrapper: +For v5.4 [query-time relevance controls](#query-time-relevance-controls), use the following shape: ```json { diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index a2bac7e9f..95c99f382 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -317,7 +317,7 @@ Returns records matching one or more conditions. Supports `operator` (`and`/`or` Full-text conditions (Added in: v5.3.0): -Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. It preserves the [v5.4 fuzzy controls](../full-text-search/querying.md#fuzzy-controls-in-v54), `fuzzyDistance` and `fuzzyTranspositions`, and the pending-release [`should` and `boost` relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). +Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. It preserves the [v5.4 fuzzy controls](../full-text-search/querying.md#fuzzy-controls-in-v54), `fuzzyDistance` and `fuzzyTranspositions`, and the v5.4 [`should` and `boost` relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). --- diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 45cb2c292..96b72708f 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1035,15 +1035,15 @@ Array of condition objects to filter records. Each condition: | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | | `fields` | Full-text source fields to search; every name must belong to the selected full-text index | -| `boost` | Optional full-text score multiplier (`number`, default `1`); see pending-release [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | -| `should` | Optional nonempty array of additional scoring clauses; omitted by default; see pending-release [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | +| `boost` | Optional full-text score multiplier (`number`, default `1`); see v5.4 [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | +| `should` | Optional nonempty array of additional scoring clauses; omitted by default; see v5.4 [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls) | | `fuzzyDistance` | Integer edit distance for a fuzzy condition, including negated forms; see [v5.4 fuzzy controls](../full-text-search/querying.md#fuzzy-controls-in-v54) and the server cap. | | `fuzzyTranspositions` | Boolean controlling whether adjacent transpositions count as one edit on a fuzzy condition; see [v5.4 fuzzy controls](../full-text-search/querying.md#fuzzy-controls-in-v54). | | `includeHighlights` | Include configured full-text highlights | | `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; see [freshness controls](../full-text-search/querying.md#freshness-controls) | | `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage; see [freshness controls](../full-text-search/querying.md#freshness-controls) | -The `fields`, `includeHighlights`, `maxIndexLagMilliseconds` and `waitForIndexMilliseconds` properties apply to full-text conditions (v5.3.0). The `fuzzyDistance` and `fuzzyTranspositions` properties apply to fuzzy conditions in v5.4. Availability and constraints for `boost` and `should` are described in [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. +The `fields`, `includeHighlights`, `maxIndexLagMilliseconds` and `waitForIndexMilliseconds` properties apply to full-text conditions (v5.3.0). The `fuzzyDistance` and `fuzzyTranspositions` properties apply to fuzzy conditions in v5.4. The `boost` and `should` controls are available in v5.4; their constraints are described in [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. Full-text conditions use `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, or `matches_fuzzy_prefix` against a declared full-text index. Each comparator also has a `not_` form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. A negated condition requires a positive full-text condition on the same index. See [Full-text query execution](./query-optimization.md#full-text-search-work-and-paging) for paging costs and work limits. @@ -1102,7 +1102,7 @@ Special properties: Full-text queries add two special properties: -- `$score`: When a query uses a full-text index, returns its BM25 relevance score. When the pending [query-time boosts and optional clauses](../full-text-search/querying.md#query-time-relevance-controls) are available, their contributions are included. +- `$score`: When a query uses a full-text index, returns its BM25 relevance score. In v5.4, supplied [query-time boosts and optional clauses](../full-text-search/querying.md#query-time-relevance-controls) contribute to this score. - `$highlights`: Enables and returns full-text fragments and matching spans when the index has highlighting configured. See [Highlights](../full-text-search/querying.md#highlights). #### Selecting related records diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 616990c0f..3d3e6dc79 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -96,7 +96,7 @@ REST's `limit(start,end)` uses an exclusive end. In v5.4, this prefix query skip GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(100,110) ``` -Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. For pending [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls), use HTTP `QUERY`, `Table.search()` or `search_by_conditions`. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. +Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. For v5.4 [query-time relevance controls](../full-text-search/querying.md#query-time-relevance-controls), use HTTP `QUERY`, `Table.search()` or `search_by_conditions`. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries can pass that native window within request budgets; source-version checks and filters run before records count toward the offset and limit. Each page is a new query and can see concurrent changes. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Full-text queries cannot provide an exact total count. On a REST interface with `exactCount: true`, a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). Without that opt-in, Harper applies `count=estimated` instead and can return an estimate. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 5d10626b7..c7dfec93c 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -12,6 +12,12 @@ All patch release notes for 5.4.x are available on the [releases page](https://g Full-text indexes using fulltext 0.5.0 with current field-based declarations, or prerelease 0.6, rebuild locally after this upgrade. Plan for search unavailability and review the [Tantivy upgrade requirements](#tantivy-027-and-existing-index-rebuilds) before deploying; direct downgrade to fulltext 0.5.x is unsupported. +## Full-Text Query Boosts and Optional Scoring + +Full-text conditions now accept `boost` to scale a required text score and `should` to add optional scoring clauses without expanding the matching records. Use these controls through `Table.search()`, HTTP `QUERY`, or `search_by_conditions`; select `$score` to return the adjusted relevance score. Optional clauses search the same index and require read access to their source fields. No index rebuild is needed for query-time scoring. See [Query-Time Relevance Controls](/reference/v5/full-text-search/querying#query-time-relevance-controls). + +**Upgrade impact:** `boost` and `should` are now reserved query-object keys on all tables. Misplaced controls return HTTP 400 instead of being ignored; rename custom keys that previously used these names. Attributes named `boost` or `should` remain queryable through the `attribute` property. + ## Nested Relationship Constraints [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) and [Fix wrong rows, unfiltered selects and ignored sorts in relationship queries](https://github.com/HarperFast/harper/pull/3775) correct REST and programmatic relationship queries that accepted nested constraints, non-leading relationship conditions, or top-level relationship sorts without applying them completely. Filters, ordering, offset, and limit inside `select` apply per parent in collection and single-record reads, and parents remain present when the nested selection has no matches. Outer relationship filters — whether they lead the query or follow another condition — constrain the selected children as well as the parents, so an identical filter does not need to be repeated inside the nested selection. When a top-level sort across a to-many relationship (such as `sort(products.name)`) runs in memory — when the target attribute is not `@indexed`, a more selective condition on another attribute leads the query, or `enforceExecutionOrder: true` is set — Harper now orders matched parent records by the list of each parent's related values in relationship order, compared element by element, instead of ignoring the sort; when an `@indexed` target attribute leads the query, Harper continues to order via the related table's index join.