Document full-text boosts and optional scoring

reference/configuration/operations.md

@@ -18,15 +18,15 @@ For the full list of configurable options, see [Configuration Options](./options
 
 Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`, [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) and [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents)). A change to `replication.threads` takes effect only on a full `restart`.
 
-Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging).
+Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.query` or `storage.fullText.runtime` require a full Harper process restart; `restart_service` replaces neither the captured query policy nor native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging).
 
 `operation` _(required)_ — must be `set_configuration`
 
 `replicated` _(optional)_ <VersionBadge version="v5.2.0" /> — set to `true` to apply the same configuration change to all cluster nodes in a single call. The origin node applies the change first; if its local write fails, nothing is sent to peers. Requires replication to be configured.
 
 Additional properties correspond to configuration keys in underscore-separated format (e.g. `logging_level` for `logging.level`, `clustering_enabled` for `clustering.enabled`).
 
-For full-text settings, use individual parameter names such as `storage_fullText_maxRawHitsPerRequest` or `storage_fullText_runtime_maxSearchThreads`. `set_configuration` does not accept a whole `storage_fullText` object; use `storage.fullText` as the section name only in the YAML configuration file.
+For full-text settings, use individual parameter names such as `storage_fullText_maxRawHitsPerRequest`, `storage_fullText_query_maxQueryTerms`, `storage_fullText_query_maxQueryClauses`, `storage_fullText_query_maxFuzzyDistance`, or `storage_fullText_runtime_maxSearchThreads`. `set_configuration` does not accept a whole `storage_fullText` object; use `storage.fullText` as the section name only in the YAML configuration file.
 
 :::warning Replicate cluster-appropriate parameters only
 A replicated call sends the exact same values to every node. Do not include node-local parameters — ports, `node.hostname`, `rootPath`, storage/log file paths, TLS certificate or key paths, or `replication.hostname`/`url`/`routes` — as they would overwrite each peer's local values.

reference/configuration/options.md

@@ -320,13 +320,18 @@ storage:
   - `maxRawHitsPerRequest` — Maximum fetched hits per request, including bounded prefix replay when traversal is needed; default `10000`.
   - `maxResultsPerRequest` — Maximum retained offset plus limit. A request must also fit the separate raw-hit budget; default `10000`.
   - `requestBudgetMilliseconds` — Execution budget across native batches, source checks and highlights, capped by the native execution limit; default `5000` ms. Reader acquisition and freshness waiting have separate bounds.
+  - `query.maxQueryTerms` — Maximum analyzed terms per full-text leaf; integer `1`–`1024`, default `64`, also bounded by the loaded native runtime.
+  - `query.maxQueryClauses` — Maximum compiled clauses, including Boolean structure and field expansion; integer `1`–`4096`, default `256`, also bounded by the loaded native runtime.
+  - `query.maxFuzzyDistance` — Maximum accepted fuzzy edit distance; integer `0`–`2`, default `1`. A cap of `0` requires explicit `fuzzyDistance: 0` on fuzzy conditions; the omitted distance still defaults to `1` and is rejected. Distance `2` requires operator opt-in and workload qualification.
   - `runtime.maxResidentIndexes` — Maximum resident native indexes; default `128`.
   - `runtime.maxIndexingThreads` — Maximum aggregate indexing-thread reservations across resident native indexes; default `128`.
   - `runtime.maxSearchThreads` — Maximum aggregate search-thread reservations across resident native indexes; default `256`.
   - `runtime.maxWriterMemoryBytes` — Aggregate writer-memory reservations across resident native indexes; default `4294967296` (4 GiB).
   - `runtime.maxQueuedBytes` — Aggregate queued-write reservations across resident native indexes; default `8589934592` (8 GiB).
   - `runtime.maxExpensiveSearches` — Maximum concurrent expensive native searches; default `8`.
 
+  The `query` policy is captured for the whole process and inherited by application and HTTP workers. A full Harper process restart is required; restarting only workers does not change it. Omitted keys keep their defaults. Requests cannot raise the policy. It changes query admission, not persisted index identity. Harper 5.4.0-alpha.1/alpha.2 reject an explicit `storage.fullText.query` block.
+
   The runtime budgets apply together and are not hard bounds on total process memory. See [Set Configuration](./operations.md#set-configuration) for when request and native runtime changes take effect. Use the usual underscore-separated parameter names, such as `storage_fullText_maxRawHitsPerRequest`. See [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for costs; see [readiness and recovery](../resources/query-optimization.md#readiness-and-recovery) for failures.
 
 - `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false`

reference/full-text-search/configuration.md

@@ -147,7 +147,7 @@ type Document @table(audit: true) {
 
 ## Schema changes and rebuilds
 
-The `english@2` name remains unchanged in v5.4, but Tantivy 0.27 changes some English stems. Existing indexes must rebuild, and matching or ranking can change. See [upgrading existing indexes](./overview.md#upgrading-existing-indexes).
+The `english@2` name remains unchanged in v5.4, but Tantivy 0.27 changes some English stems. Indexes from fulltext 0.5.0 or prerelease 0.6 must rebuild, and matching or ranking can change. Current Harper indexes written by fulltext 0.7.0 or 0.8.0 already use Tantivy 0.27 and can reuse compatible checkpoints with 0.9.0. See [upgrading existing indexes](./overview.md#upgrading-existing-indexes).
 
 Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the `fields` list in a stable order when its meaning has not changed.
 

reference/full-text-search/overview.md

@@ -10,7 +10,7 @@ Harper can build a BM25-ranked full-text index from one or more fields on a tabl
 
 Use full-text search when users need relevance-ranked matching across product names, descriptions, tags, articles, or other natural-language fields. It supports term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching.
 
-<VersionBadge type="changed" version="v5.4.0" /> Starting in Harper v5.4, the native engine is **`@harperfast/fulltext` 0.7.0, embedding Tantivy 0.27.0**. Existing fulltext 0.5.0 and prerelease 0.6 indexes require rebuilding. See [Upgrading existing indexes](#upgrading-existing-indexes) before upgrading a node.
+<VersionBadge type="changed" version="v5.4.0" /> Starting in Harper v5.4, the native engine is **`@harperfast/fulltext` 0.9.0, embedding Tantivy 0.27.0**. Existing fulltext 0.5.0 and prerelease 0.6 indexes require rebuilding. See [Upgrading existing indexes](#upgrading-existing-indexes) before upgrading a node.
 
 ## Requirements
 
@@ -59,16 +59,22 @@ Use `describe_table` to inspect each index's declaration and readiness. See [Ins
 
 <VersionBadge type="changed" version="v5.4.0" />
 
-Harper 5.3.1–5.3.4 ship `@harperfast/fulltext` 0.5.0. Upgrading their current field-based indexes to the 0.7.0 engine in Harper v5.4 requires a full local rebuild. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; plan and validate that migration separately. Prerelease installations using fulltext 0.6 also require rebuilding. This procedure applies to the current field-based declarations; earlier beta declarations and indexes have no supported migration path. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed.
+Harper 5.3.1–5.3.4 ship `@harperfast/fulltext` 0.5.0. Upgrading their current field-based indexes to the 0.9.0 engine in Harper v5.4 requires a full local rebuild. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; plan and validate that migration separately. Prerelease installations using fulltext 0.6 also require rebuilding. This procedure applies to the current field-based declarations; earlier beta declarations and indexes have no supported migration path. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed.
+
+### Upgrading from fulltext 0.7.0 or 0.8.0
+
+Harper 5.4.0-alpha.1 and alpha.2 ship fulltext 0.7.0. Their existing indexes, and indexes created by development builds using fulltext 0.8.0, remain compatible with 0.9.0 and can resume from their saved checkpoint without rebuilding. These are indexes created through Harper's `@fullText` declaration; Harper does not create native filter metadata. Query controls do not change their stored identity. Stop all processes sharing a local index before switching versions; do not mix facade/native package versions or run different engines against one live directory. Check readiness, current coverage and representative searches before restoring traffic, as described in the checklist below.
+
+Returning to fulltext 0.7.0 or 0.8.0 after 0.9.0 has written to the index is not a qualified operator rollback procedure. If returning to Harper 5.4.0-alpha.1 or alpha.2, remove an explicitly configured `storage.fullText.query` block: those versions reject it. Removing that configuration alone does not qualify a downgrade. If an older build was already started, stop it, restore the 0.9.0-based Harper build, and check readiness and current coverage before restoring full-text traffic.
 
 ### Matching and ranking changes
 
-Tantivy 0.27 changes English stemming even though the schema analyzer remains `english@2`. For example, `added` now stems to `add` instead of `ad`. Queries can match different records or rank them differently after rebuilding. Review representative searches and domain vocabulary before restoring production traffic; the upgrade does not add new Harper query modes or change the declared field options.
+When upgrading from fulltext 0.5.0 or prerelease 0.6, Tantivy 0.27 changes English stemming even though the schema analyzer remains `english@2`. For example, `added` now stems to `add` instead of `ad`. Queries can match different records or rank them differently after rebuilding. Review representative searches and domain vocabulary before restoring production traffic; the engine migration does not change the declared field options. Query-time relevance controls are described separately in [Querying](./querying.md#query-time-relevance-controls).
 
 ### Upgrade checklist
 
 1. Drain the node's full-text requests and stop all old Harper processes that share its storage, including background workers. Do not run old and new engine versions against one local index.
-2. Confirm authoritative records and recovery backups are available. Reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. Measure rebuild duration and disk growth with representative data before setting the rollout window.
+2. Confirm authoritative records and recovery backups are available. For fulltext 0.5.0 or prerelease 0.6, reserve disk headroom and time for rebuilding every local full-text index, including temporarily retained retired files. Compatible 0.7/0.8 indexes normally resume from checkpoint; missing, corrupt or unrecoverably lagging files can still require rebuilding. Plan rebuild capacity for that recovery case.
 3. Start the v5.4 Harper version. Use [`describe_table`](../operations-api/operations.md#describe_table) for each affected table, using an authorized operator account that already has read access to every indexed source field, and check every `full_text_indexes[].readiness.state`. Metadata is permission-filtered; a missing entry does not prove that no index exists. Keep full-text traffic drained until all required indexes are `ready`.
 4. Run representative searches through the same REST or permission-checked application path and roles used in production, with `maxIndexLagMilliseconds: 0` and `waitForIndexMilliseconds: 30000` on each full-text condition. These wait for coverage of writes committed on the queried node before the search; they do not wait for replication delivery, and readiness alone does not establish that coverage. Check response bodies for terminal stream errors as well as HTTP status. A JSON-array response can start with HTTP 200 and end with an `error` element.
 5. Retry write responses carrying `DERIVED_INDEX_LAGGING` with bounded backoff while replay catches up. Restore full-text traffic only after readiness and current-coverage searches succeed. A node-by-node upgrade can use other ready replicas only if the application's routing supports it; the rebuilding node remains unavailable for full-text search. During a mixed-version rollout, matching and ranking can differ between replicas. Pre-v5.4 replicas also reject prefix requests whose `offset + limit` exceeds 100 with `400`. Route related searches consistently and check the application's behavior before using this approach.
@@ -79,7 +85,7 @@ See [freshness controls](./querying.md#freshness-controls) for the condition fie
 
 Keep full-text traffic drained, preserve authoritative data and logs, and correct the reported package, storage or coordination failure before restarting the same v5.4 Harper version. An interrupted rebuild can recover from authoritative records. Do not force readiness or edit, move or delete native index files by hand.
 
-**Direct downgrade to fulltext 0.5.x is unsupported.** That engine refuses both opening and resetting a 0.7 index identity. If an older version has already been started, stop it and restore the 0.7.0-based Harper version before checking readiness and catch-up again. This is recovery with the current engine, not a supported operator downgrade procedure. If recovery also fails, keep traffic drained and investigate the readiness reason and node logs.
+**Direct downgrade to fulltext 0.5.x is unsupported.** That engine refuses both opening and resetting the current Tantivy 0.27 index identity. If an older version has already been started, stop it and restore the 0.9.0-based Harper version before checking readiness and catch-up again. This is recovery with the current engine, not a supported operator downgrade procedure. If recovery also fails, keep traffic drained and investigate the readiness reason and node logs.
 
 ## Write backpressure
 
@@ -89,7 +95,7 @@ Retry rejected writes with backoff rather than immediately adding more load. Ins
 
 ## Ranking and result loading
 
-The native index ranks matching documents with BM25. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result.
+The native index ranks matching documents with BM25, adjusted by any supplied [query-time boosts and optional-clause contributions](./querying.md#query-time-relevance-controls) in v5.4. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result.
 
 Full-text search can be combined with structured filters. Harper checks conditions against current source records before returning them. In v5.4, additional ranked batches can fill a filtered page within request budgets; exhausting a budget returns an error rather than a budget-truncated page.