diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 90ceb4beb..528177153 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -349,9 +349,9 @@ Check `available` and `collectionError` before interpreting observations; `obser `complete: false` flags lost final-handle counts. The next successful live observation starts a new `series` and restores complete coverage; use that observation as a new rate baseline. -Pending rows coalesce in a 128-row buffer per worker, dropping the oldest by first pending insertion on overflow; updating a pending row retains its position. While a group still has live handles, dropping a pending row preserves its cumulative counters for the next observation. After its last handle closes, losing its final pending row can lose those final totals; no later observation or `complete: false` row is guaranteed. `droppedRows` counts buffer evictions per worker and is repeated on its rows; do not sum it across indexes or roles. Workers and the receiving main thread enforce their current collection policy independently. During configuration transitions, the receiver can discard exported rows; those policy-filtered rows are not included in `droppedRows`. +Pending rows coalesce in a 128-row buffer per worker, on overflow evicting the least recently refreshed pending row from a still-live group before evicting a final row. Updating a pending row refreshes its position. While a group still has live handles, dropping a pending row preserves its cumulative counters for the next observation. After its last handle closes, losing its final pending row can lose those final totals; no later observation or `complete: false` row is guaranteed. `droppedRows` counts buffer evictions per worker and is repeated on its rows; do not sum it across indexes or roles. Graceful HTTP/job-worker exit drains pending rows and takes final observations after owner cleanup, including still-open handles. Main waits for raw-store commits during graceful exit; this adds no disk-sync guarantee. Forced termination can lose reports. Workers and the receiving main thread enforce their current collection policy independently. During configuration transitions, the receiver can discard exported rows; those policy-filtered rows are not included in `droppedRows`. -Null can mean unavailable data, an empty queue's oldest age, or the unbounded final histogram bucket. Histogram buckets are noncumulative; their changes cover only intervals when histogram recording was enabled. Harper does not invent percentiles. +Null can mean unavailable data, an empty queue's oldest age, or the unbounded final histogram bucket. Histogram buckets are noncumulative; their changes cover only intervals when histogram recording was enabled. Harper does not invent percentiles. `describe_metric` omits attributes whose latest value is null when inferring their types; `get_analytics` retains the null values. Reader reopen and overlapping retired readers carry counters forward while their logical series remains alive. Reader document/segment/generation gauges use the newest eligible observation rather than summing duplicate snapshots. Queue occupancy (`commands` and `bytes`), capacities (`maxCommands` and `maxBytes`) and operation `dispatched` gauges sum live handles of the same role; oldest age uses their maximum. During a reader reopen, overlapping live handles can therefore increase the reported capacity. Shared queue capacities also overlap between ordinary and expensive search, so do not add those capacities or interpret them as a process-wide budget. @@ -419,7 +419,7 @@ Type: `object` Default: omitted; operator collection is disabled -Enable and tune optional full-text collection with the [`analytics.fullText` options](../configuration/options.md#analyticsfulltext). Defaults are implied; the block is absent from the shipped default configuration. Changes apply dynamically without reopening indexes. Removing the configuration file reapplies `HARPER_DEFAULT_CONFIG`, `HARPER_CONFIG` and `HARPER_SET_CONFIG` overlays to an empty file, so collection requested only by that file stops; one of those overlays can keep it enabled. +Enable and tune optional full-text collection with the [`analytics.fullText` options](../configuration/options.md#analyticsfulltext). Defaults are implied; the block is absent from the shipped default configuration. Changes apply dynamically without reopening indexes. A missing or invalid file retains the last valid telemetry policy and overrides. Disable collection with a valid configuration edit, removing any environment or in-process setting that keeps it enabled; file deletion alone does not disable it. ### `analytics.logging` diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index 2e9b089b3..778e5ee47 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -18,7 +18,7 @@ For the full list of configurable options, see [Configuration Options](./options Modifies one or more Harper configuration parameters. **Most changes require a [restart](../operations-api/operations.md#restart) or [restart_service](../operations-api/operations.md#restart_service) to take effect** (a [restart](../operations-api/operations.md#restart) for `threads.v8Flags`, [`storage.rocks.transactionLogAbortMarkers`](../database/storage-tuning.md#storagerockstransactionlogabortmarkers) and [`storage.rocks.transactionLogWriteIntents`](../database/storage-tuning.md#storagerockstransactionlogwriteintents)). A change to `replication.threads` takes effect only on a full `restart`. -Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` require a full Harper process restart; `restart_service` does not replace the native process budget. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). +Full-text request work limits (`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds`) take effect in HTTP workers after `restart_service` or a full restart. Changes to `storage.fullText.runtime` or `storage.fullText.query` require a full Harper process restart; `restart_service` does not replace the captured process budgets or query policy. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). `operation` _(required)_ — must be `set_configuration` diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 684065725..649b1439a 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -320,6 +320,12 @@ storage: - `maxRawHitsPerRequest` — Maximum fetched hits per request, including bounded prefix replay when traversal is needed; default `10000`. - `maxResultsPerRequest` — Maximum retained offset plus limit. A request must also fit the separate raw-hit budget; default `10000`. - `requestBudgetMilliseconds` — Execution budget across native batches, source checks and highlights, capped by the native execution limit; default `5000` ms. Reader acquisition and freshness waiting have separate bounds. + - `query.maxQueryTerms` — Maximum analyzed terms per text leaf; integer `1`–`1024`, default `64`. + - `query.maxFuzzyDistance` — Maximum fuzzy edit distance; integer `0`–`2`, default `1`. Distance `2` is preview and needs workload qualification; admission and deadlines are not hard process-memory bounds. With `0`, fuzzy and fuzzy-prefix conditions that omit `fuzzyDistance: 0` are rejected with HTTP 400; omitted distances remain `1`. + - `query.maxQueryClauses` — Maximum compiled clauses per query, including Boolean structure, NOT and field expansion; integer `1`–`4096`, default `256`. + + Query settings are captured for the process and inherited by workers. Changes require a full Harper process restart; `restart_service` alone does not apply them. Omitted values retain defaults, and request work budgets cannot raise these limits. + - `runtime.maxResidentIndexes` — Maximum resident native indexes; default `128`. - `runtime.maxIndexingThreads` — Maximum aggregate indexing-thread reservations across resident native indexes; default `128`. - `runtime.maxSearchThreads` — Maximum aggregate search-thread reservations across resident native indexes; default `256`. @@ -414,7 +420,7 @@ analytics: | `histograms` | `boolean` | `false` | Record optional latency histograms. | | `snapshotBytes` | `boolean` | `false` | Request cached reader-byte observations; enabling does not trigger index publication or reader reload. | -Set `enabled: false` or remove the block to stop operator collection and export. Invalid edits retain the last valid policy. Deleting the configuration file removes the file-based `analytics.fullText` setting and reapplies `HARPER_DEFAULT_CONFIG`, `HARPER_CONFIG` and `HARPER_SET_CONFIG` overlays to an empty file; collection remains enabled if one of those overlays still requests it. An in-process override is retired if deletion changes its matching configuration value after those overlays are applied; recreating the file does not restore retired overrides. Internal implementation hooks can independently keep collection active without enabling persistence; they are not a public SDK or operator prerequisite. With operator export enabled, persisted fields reflect the union of operator and internal-consumer recording requests. A handle already closing retains its last options until close settles. +Set `enabled: false` or remove the block to stop operator collection and export, unless environment overlays or in-process overrides still enable it. Invalid edits retain the last valid policy. A missing configuration file retains the last valid telemetry policy and in-process overrides; recreating an unchanged file preserves them. To disable collection, make a valid file edit and remove any environment or in-process setting that keeps it enabled. Valid edits retire overrides only where matching effective values change after environment overlays. Internal implementation hooks can independently keep collection active without enabling persistence; they are not a public SDK or operator prerequisite. With operator export enabled, persisted fields reflect the union of operator and internal-consumer recording requests. A handle already closing retains its last options until close settles. The interval controls periodic live sampling. Successful handle closes also collect final counters, so it does not cap total snapshot or export frequency. Qualify reader churn as well as index count and sampling cadence. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md index 7bde113ce..049e4c0c3 100644 --- a/reference/full-text-search/overview.md +++ b/reference/full-text-search/overview.md @@ -10,7 +10,7 @@ Harper can build a BM25-ranked full-text index from one or more fields on a tabl Use full-text search when users need relevance-ranked matching across product names, descriptions, tags, articles, or other natural-language fields. It supports term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching. - Starting in Harper v5.4, the native engine is **`@harperfast/fulltext` 0.9.0, embedding Tantivy 0.27.0**. Existing fulltext 0.5.0 and prerelease 0.6 indexes require rebuilding. See [Upgrading existing indexes](#upgrading-existing-indexes) before upgrading a node. Existing text-only 0.7.0 and 0.8.0 checkpoints with compatible identities are reused by 0.9.0 without another rebuild. + Starting in Harper v5.4, the native engine is **`@harperfast/fulltext` 0.9.0, embedding Tantivy 0.27.0**. Existing fulltext 0.5.0 and prerelease 0.6 indexes require rebuilding. See [Upgrading existing indexes](#upgrading-existing-indexes) before upgrading a node. Nodes that already ran v5.4 prereleases with fulltext 0.7.0 or 0.8.0 can reuse compatible text-only checkpoints with 0.9.0 without another rebuild. Optional [full-text analytics](../analytics/overview.md#full-text-analytics) can report operation counters, queues and cached reader observations without reopening indexes. Collection is off by default. @@ -61,7 +61,7 @@ Use `describe_table` to inspect each index's declaration and readiness. See [Ins -Harper 5.3.1–5.3.4 ship `@harperfast/fulltext` 0.5.0. Upgrading their current field-based indexes to the 0.9.0 engine in Harper v5.4 requires a full local rebuild. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; plan and validate that migration separately. Prerelease installations using fulltext 0.6 also require rebuilding. This procedure applies to the current field-based declarations; earlier beta declarations and indexes have no supported migration path. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed. The 0.5/0.6 rebuild requirement does not apply to compatible text-only 0.7.0 and 0.8.0 checkpoints reused by 0.9.0; other older declarations remain outside this qualified migration. +Harper 5.3.1–5.3.4 ship `@harperfast/fulltext` 0.5.0. Upgrading their current field-based indexes to the 0.9.0 engine in Harper v5.4 requires a full local rebuild. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; plan and validate that migration separately. Prerelease installations using fulltext 0.6 also require rebuilding. This procedure applies to the current field-based declarations; earlier beta declarations and indexes have no supported migration path. Harper initiates the rebuild automatically when the wrapper refuses the old persisted identity; no manual rebuild command or file editing is needed. The 0.5/0.6 rebuild requirement does not apply to compatible text-only checkpoints from nodes that already ran v5.4 prereleases with fulltext 0.7.0 or 0.8.0; other older declarations remain outside this qualified migration. ### Matching and ranking changes diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md index 4980eb120..2ca866ea5 100644 --- a/reference/full-text-search/querying.md +++ b/reference/full-text-search/querying.md @@ -21,14 +21,16 @@ Results are ordered by BM25 relevance. The `FullText` field, not one of its sour ## Match modes -| Comparator | Behavior | Required index option | -| ---------------------- | --------------------------------------------------------------------- | --------------------- | -| `matches` | Matches any analyzed query term. | — | -| `matches_all` | Requires every analyzed query term. | — | -| `matches_phrase` | Matches analyzed terms in order as a phrase. | `positions: true` | -| `matches_prefix` | Matches records whose analyzed terms start with the final query term. | `surfaceTerms: true` | -| `matches_fuzzy` | Matches terms within one edit, including a transposition. | — | -| `matches_fuzzy_prefix` | Combines one-edit fuzzy matching with a final-term prefix. | `surfaceTerms: true` | +| Comparator | Behavior | Required index option | +| ---------------------- | ----------------------------------------------------------------------- | --------------------- | +| `matches` | Matches any analyzed query term. | — | +| `matches_all` | Requires every analyzed query term. | — | +| `matches_phrase` | Matches analyzed terms in order as a phrase. | `positions: true` | +| `matches_prefix` | Matches records whose analyzed terms start with the final query term. | `surfaceTerms: true` | +| `matches_fuzzy` | Matches terms within one edit by default, including a transposition. | — | +| `matches_fuzzy_prefix` | Combines fuzzy matching (one edit by default) with a final-term prefix. | `surfaceTerms: true` | + + On a `Table.search()` fuzzy or fuzzy-prefix condition, `fuzzyDistance` accepts an integer `0`–`2` and defaults to `1`. It must not exceed `storage.fullText.query.maxFuzzyDistance`; exceeding that maximum returns HTTP 400. With a maximum of `0`, set `fuzzyDistance: 0` explicitly. Distance `2` remains preview and requires workload qualification. Every comparator also has a negated form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index afa5aa0fc..c9f378d6e 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -115,7 +115,7 @@ Replacement workers keep the captured budgets after a configuration change. See Full-text indexes require audit logging. New schema tables must explicitly declare `@table(audit: true)` when adding `@fullText`, even if global audit logging is enabled; an existing audited table can add an index without restating `audit`. Setting `@table(audit: false)` while an index is declared is rejected during schema registration. -Harper v5.4 uses `@harperfast/fulltext` 0.9.0 with Tantivy 0.27.0. Compatible text-only 0.7.0 and 0.8.0 checkpoints are reused without another rebuild. Existing 0.5.0 and prerelease 0.6 indexes using current field-based declarations need an automatic local rebuild after this engine upgrade. Follow [upgrading existing indexes](../full-text-search/overview.md#upgrading-existing-indexes) for affected versions, draining traffic, readiness checks, matching changes and current-engine recovery. Earlier beta declarations and indexes have no supported migration path. See [write backpressure](../full-text-search/overview.md#write-backpressure) for local write refusal and retry guidance. +Harper v5.4 uses `@harperfast/fulltext` 0.9.0 with Tantivy 0.27.0. Nodes that already ran v5.4 prereleases with fulltext 0.7.0 or 0.8.0 can reuse compatible text-only checkpoints without another rebuild. Existing 0.5.0 and prerelease 0.6 indexes using current field-based declarations need an automatic local rebuild after this engine upgrade. Follow [upgrading existing indexes](../full-text-search/overview.md#upgrading-existing-indexes) for affected versions, draining traffic, readiness checks, matching changes and current-engine recovery. Earlier beta declarations and indexes have no supported migration path. See [write backpressure](../full-text-search/overview.md#write-backpressure) for local write refusal and retry guidance. | Failure | HTTP status | Code and retry guidance | | ------------------------------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 39aa29582..5ef72ae12 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -186,13 +186,13 @@ Optional `analytics.fullText` collection reports full-text operation counters, q ### Tantivy 0.27 and Existing Index Rebuilds -Harper now uses `@harperfast/fulltext` 0.9.0 with Tantivy 0.27.0. Compatible text-only 0.7.0 and 0.8.0 checkpoints are reused without an additional rebuild. Harper 5.3.1–5.3.4 ship fulltext 0.5.0. Existing 0.5.0 indexes using current field-based declarations, and prerelease 0.6 indexes, rebuild automatically from authoritative records. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; validate that migration separately. Earlier beta declarations and indexes have no supported migration path. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported. +Harper now uses `@harperfast/fulltext` 0.9.0 with Tantivy 0.27.0. Nodes that already ran v5.4 prereleases with fulltext 0.7.0 or 0.8.0 can reuse compatible text-only checkpoints without an additional rebuild. Harper 5.3.1–5.3.4 ship fulltext 0.5.0. Existing 0.5.0 indexes using current field-based declarations, and prerelease 0.6 indexes, rebuild automatically from authoritative records. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; validate that migration separately. Earlier beta declarations and indexes have no supported migration path. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported. English stemming changes despite the unchanged `english@2` analyzer name: `added` now stems to `add` rather than `ad`. Matching and ranking can change. Query modes and schema options are unchanged; benchmark results from the wrapper do not establish application performance parity. Review representative searches and see [upgrading existing indexes](/reference/v5/full-text-search/overview#upgrading-existing-indexes) before restoring traffic. ### Paging and Search Work Limits -Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. The `storage.fullText.runtime` section also provides process-wide native-capacity budgets; changes to those budgets require a full Harper process restart. Separate page requests run independently and can see changes to records or index contents. +Full-text and autocomplete queries use the existing `limit` and `offset` paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new `storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. The `storage.fullText.runtime` section also provides process-wide native-capacity budgets, while `storage.fullText.query` controls analyzed terms, compiled clauses and maximum fuzzy distance. The default maximum fuzzy distance is `1`; requests with `fuzzyDistance: 2` need an explicit maximum of `2`, which remains preview and requires workload qualification. Both sections require a full Harper process restart after changes; `restart_service` alone does not apply them. Separate page requests run independently and can see changes to records or index contents. See [Full-text query execution](/reference/v5/resources/query-optimization#full-text-search-work-and-paging) and [storage configuration](/reference/v5/configuration/options#storage) for limits, restart requirements and recovery guidance.