Document 26.10.0 namespace safety and vector-search evaluation

README.md

@@ -10,7 +10,9 @@
 
 <p align="center">
   <a href="https://github.com/fireflyframework/fireflyframework-agentic/releases/latest"><img src="https://img.shields.io/github/v/release/fireflyframework/fireflyframework-agentic" alt="Latest release"></a>
-  <a href="https://github.com/fireflyframework/fireflyframework-agentic/actions/workflows/pr-gate.yml"><img src="https://github.com/fireflyframework/fireflyframework-agentic/actions/workflows/pr-gate.yml/badge.svg?branch=main" alt="PR gate"></a>
+  <a href="https://github.com/fireflyframework/fireflyframework-agentic/actions/workflows/pr-gate.yml"><img src="https://github.com/fireflyframework/fireflyframework-agentic/actions/workflows/pr-gate.yml/badge.svg?event=pull_request" alt="Latest PR checks"></a>
+  <a href="https://github.com/fireflyframework/fireflyframework-agentic/actions/workflows/release.yml"><img src="https://github.com/fireflyframework/fireflyframework-agentic/actions/workflows/release.yml/badge.svg" alt="Release publication"></a>
+  <a href="https://github.com/fireflyframework/fireflyframework-agentic/actions/workflows/docs.yml"><img src="https://github.com/fireflyframework/fireflyframework-agentic/actions/workflows/docs.yml/badge.svg?branch=main" alt="Documentation deployment"></a>
   <a href="https://github.com/fireflyframework/fireflyframework-agentic/actions/workflows/nightly.yml"><img src="https://github.com/fireflyframework/fireflyframework-agentic/actions/workflows/nightly.yml/badge.svg?branch=main" alt="Nightly"></a>
   <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.13%2B-blue.svg" alt="Python 3.13+"></a>
   <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-green.svg" alt="License: Apache 2.0"></a>
@@ -19,6 +21,12 @@
   <a href="https://microsoft.github.io/pyright/"><img src="https://img.shields.io/badge/type--checked-pyright-2c8a1c.svg" alt="Type checked: pyright"></a>
 </p>
 
+Current release: [26.10.0](https://github.com/fireflyframework/fireflyframework-agentic/releases/tag/v26.10.0).
+PostgreSQL vector writes now reject cross-namespace ID conflicts and roll back the
+whole batch. Read the [upgrade notes](docs/migration.md#postgresql-vector-store-safety-in-26100).
+Pgvector/HNSW remains the supported PostgreSQL backend; the
+[pg_vectorsearch evaluation](docs/pg-vectorsearch-evaluation.md) did not qualify PRISM for adoption.
+
 <p align="center">
   <em>Build with Firefly agents, tools, memory and typed model options — gain lifecycle hooks, delegation, reasoning patterns, validation loops, RAG, and DAG pipelines, all protocol-driven and swappable.</em>
 </p>
@@ -416,7 +424,9 @@ The diagram distinguishes structural protocols from inheritance-based extension
   `parse_scope_namespace` helpers) namespaces documents per tenant or scope.
   Auto-embedding upserts documents without pre-computed vectors. `search_text`
   embeds a query string and searches in one call, and `SearchFilter` narrows results
-  by metadata. Namespace scoping isolates document collections. `VectorStoreRegistry`
+  by metadata. PostgreSQL document IDs remain unique across the table; a conflicting
+  namespace rejects the whole upsert batch. Applications bind scopes to authenticated
+  callers and manage database authorization. `VectorStoreRegistry`
   manages named instances. `EmbeddingStep` and `RetrievalStep` integrate directly
   into DAG pipelines for retrieval-augmented workflows.
 

docs/getting-started/installation.md

@@ -40,6 +40,12 @@ Releases use **YY.MM.Patch**, starting with patch `0` each month. Python normali
 assets; it does not publish to PyPI. Pin the framework and lock application
 dependencies for reproducible deployments.
 
+The release also provides [SHA256SUMS](https://github.com/fireflyframework/fireflyframework-agentic/releases/download/v26.10.0/SHA256SUMS)
+for the wheel and source archive. Verify downloaded files against those hashes
+before installation. Existing PostgreSQL vector-store users should read the
+[26.10.0 namespace conflict changes](../migration.md#postgresql-vector-store-safety-in-26100).
+The fix requires no database migration or PostgreSQL version upgrade.
+
 ## One-line installer
 
 The interactive installer detects your platform, checks Python and uv, lets you

docs/migration.md

@@ -1,8 +1,42 @@
 # Migration Guide
 
-This guide covers the Pydantic AI 2.x upgrade and the earlier tool and Dynamic
-Workflow changes. It distinguishes dependency changes from changes to Firefly's
-public API and model routing.
+Copyright 2026 Firefly Software Foundation. SPDX-License-Identifier: Apache-2.0.
+
+This guide covers PostgreSQL vector-store safety in 26.10.0, the Pydantic AI 2.x
+upgrade, and earlier tool and Dynamic Workflow changes.
+
+## PostgreSQL vector-store safety in 26.10.0
+
+Previously, upserting an existing document ID into another namespace could
+overwrite and move the original PostgreSQL row. Version 26.10.0 raises
+`VectorStoreError` for that conflict and rolls back the whole batch, including
+earlier inserts or updates. The check also applies to concurrent writes and
+`TenantScopedVectorStore`.
+
+The table still uses a global `PRIMARY KEY (id)`. Same-namespace updates and
+repeated IDs within one same-namespace batch keep their existing behavior. No
+database migration, index rebuild, or re-embedding is required for this fix.
+
+Before upgrading a shared-table deployment, check how the application assigns
+document IDs. IDs must be unique across namespaces. New documents in different
+scopes need distinct IDs; do not bulk-rename existing IDs without accounting for
+their references. If the application used cross-namespace upsert to move a
+document, replace that behavior with an explicitly authorized transfer workflow.
+Retrying the same conflicting batch will not succeed.
+
+Install the [26.10.0 release](getting-started/installation.md#install-a-release),
+then verify same-namespace updates, rejection of a foreign-namespace ID, and
+whole-batch rollback in an isolated application test database. This fix does not
+restore content overwritten by earlier versions; any suspected historical loss
+requires review of application audit records or backups.
+
+The wrapper requires scope arguments but does not authenticate them or install
+RLS policies. Keep authorization and transaction-local database scope setup in
+the application. See [vector-store scoping](vectorstores.md#tenantworkspace-scoping)
+and [database authorization](security.md#vector-store-authorization).
+
+Pgvector/HNSW remains unchanged. PostgreSQL 18 and pg_vectorsearch are not
+requirements of this release; [PRISM remains outside the supported backend](pg-vectorsearch-evaluation.md).
 
 ## Pydantic AI 2 and OpenAI API selection
 

docs/pg-vectorsearch-evaluation.md

@@ -0,0 +1,103 @@
+# pg_vectorsearch adoption evaluation
+
+Copyright 2026 Firefly Software Foundation. SPDX-License-Identifier: Apache-2.0.
+
+The October 14, 2026 evaluation did not qualify pg_vectorsearch for adoption in
+Agentic, flycanon, or flyquery. Pgvector/HNSW remains the supported PostgreSQL
+backend. Agentic 26.10.0 fixes namespace writes without enabling PRISM or requiring
+PostgreSQL 18. No PRISM configuration option, automatic extension installation,
+index replacement, or schema migration was added.
+
+## Release and compatibility
+
+The evaluated [pg_vectorsearch v0.1.0 release](https://github.com/timescale/pg_vectorsearch/releases/tag/v0.1.0)
+is commit `36b75d1cb56c12a380d600ef27217d88c9f419ec`. Its
+[changelog](https://github.com/timescale/pg_vectorsearch/blob/36b75d1cb56c12a380d600ef27217d88c9f419ec/CHANGELOG.md)
+describes its quality as alpha despite the final release label. It requires
+PostgreSQL 18 or later and uses the PostgreSQL License. It can index existing
+pgvector columns with its own operator classes, but the native probes rejected
+expression indexes, unlogged tables, and `halfvec(3072)` indexes. The reported
+maximum was 1,968 dimensions.
+
+| Product | Existing storage contract | Decision |
+| --- | --- | --- |
+| Agentic | Fixed-width pgvector column, HNSW bootstrap, namespace and metadata filters | Storage is compatible at supported widths; filtered recall and index lifecycle need further qualification. |
+| flycanon | Mixed widths, per-set partial expression indexes, wide halfvec embeddings | Expression indexes and widths above 1,968 are incompatible. A materialized projection would require a separate migration and synchronization design. |
+| flyquery | Plain `vector(1536)` columns, dataset joins and RLS; computed ranking for examples | Column shape fits, but filtered retrieval and actual query plans are unqualified. An index change alone cannot convert computed ranking into ANN traversal. |
+
+The audited source revisions were Agentic `8e56361`, flycanon `2d7dca1`, and
+flyquery `be68d344`. The inspected flycanon/flyquery development and integration
+stacks used PG16. Their production database upgrades were outside this evaluation.
+
+## Measured results
+
+The isolated runtime used PostgreSQL 18.6, pgvector 0.8.7, and pg_vectorsearch
+0.1.0 on macOS arm64. PRISM was compiled in release mode with native NEON and
+Apple Accelerate. PostgreSQL used 128 MiB shared buffers, 64 MiB work memory,
+256 MiB maintenance memory, and no parallel query or maintenance workers for
+the measured operations.
+
+Two synthetic corpora used seed `20261014`: 30,000 vectors at width 384 and
+12,000 at width 1,536. Normalized vectors formed 64 clusters across 16 tenants and
+four projects. Exact cosine top-10 results came from an unindexed copy of each
+table. Each cohort contained 48 distinct queries repeated three times. Opposite
+tenant/project scopes deliberately selected rows far from the query's nearest
+global clusters; their failure rates are not estimates of production traffic.
+
+| Corpus | Setting | Global recall@10 | Global p50 latency | Incomplete opposite tenant/project queries |
+| --- | --- | ---: | ---: | ---: |
+| 30k × 384 | HNSW ef 40 | 99.17% | 0.347 ms | 144/144 |
+| 30k × 384 | HNSW strict iterative ef 160 | 100% | 0.395 ms | 0/144 |
+| 30k × 384 | PRISM auto | 93.75% | 0.168 ms | 111/144 |
+| 30k × 384 | PRISM nprobe 256 | 94.79% | 0.660 ms | 102/144 |
+| 12k × 1536 | HNSW ef 40 | 99.79% | 0.252 ms | 144/144 |
+| 12k × 1536 | HNSW strict iterative ef 160 | 100% | 0.426 ms | 0/144 |
+| 12k × 1536 | PRISM auto | 87.50% | 0.372 ms | 114/144 |
+| 12k × 1536 | PRISM nprobe 256 | 95.42% | 1.669 ms | 48/144 |
+
+Timings forced each ANN index and include the local client round trip. In the
+1,536-dimensional run, 40 of 48 sampled natural plans chose a sequential scan
+and sort instead. These timings do not describe default application performance.
+The lower PRISM latency at width 384 also has lower recall; no global speedup at
+matched recall was established. HNSW iterative scans filled the filtered result
+slots, but opposite tenant/project recall was only 38.33% and 58.75% in the two
+loaded corpora. Result count alone does not establish retrieval quality.
+
+PRISM index sizes were 5.64 MB versus HNSW's 61.45 MB at width 384, and 5.19 MB
+versus 98.31 MB at width 1,536 (decimal MB). Single build times were 0.663 versus
+3.971 seconds and 0.518 versus 3.564 seconds. Full vectors remain in the heap.
+Build order and background system activity were not controlled across repeated
+independent builds. Post-query RSS includes shared mappings and does not establish
+peak private memory savings.
+
+Each corpus then received 5% inserts, 5% vector updates, and 5% deletes, followed
+by VACUUM ANALYZE. Under simplified FORCE RLS policies and a restricted reader,
+PRISM auto underfilled 37/48 and 38/48 opposite tenant/project queries, with
+16.67% and 13.96% recall. No wrong-scope rows were observed. The test did not
+reproduce actual service middleware, pooling, grants, or dataset joins.
+
+One immediate shutdown and WAL recovery per corpus preserved committed ID/vector
+fingerprints and six sampled query vectors across four cohorts per configuration.
+Uncommitted inserts, deletes, and embedding updates rolled back. This does not
+qualify backup/restore, failover, extension upgrades, or interrupted maintenance.
+Independent review checked dataset parity, unique churn vectors, recorded plans,
+scope results, and recovery comparisons.
+
+## Conditions for a future pilot
+
+Reconsider PRISM only after it preserves supported schemas, embedding widths,
+IDs, scores, authorization, and source/citation contracts. Installation and index
+construction must be operator-owned; PostgreSQL can choose either index when two
+exist, so a configuration flag alone cannot guarantee routing.
+
+A future pilot should achieve at least 95% mean recall@10 in every representative
+filtered cohort, return `min(10, eligible rows)` on every query, and expose zero
+rows outside the authorized scope. Compare latency at recall within one percentage
+point. Require either 20% lower p95 latency or 30% lower measured peak private
+memory without more than 20% p95 latency regression, using representative scale
+and concurrency. These are proposed acceptance thresholds, not achieved results.
+
+Qualify concurrent changes, pooled scope reuse, maintenance, recovery, upgrades,
+and reversal separately. Keep existing indexes and data intact during evaluation.
+For the supported backend's current settings and namespace behavior, see
+[vector stores](vectorstores.md) and the [26.10.0 upgrade notes](migration.md#postgresql-vector-store-safety-in-26100).

docs/security.md

@@ -12,6 +12,25 @@ patterns. Blocking or sanitization policy is selected by the application.
 
 ---
 
+## Vector-store authorization
+
+`TenantScopedVectorStore` requires tenant and workspace arguments and maps them
+to a namespace. Applications must derive or validate those arguments against the
+authenticated caller. Metadata stamps and namespace strings do not grant access.
+
+Since 26.10.0, `PgVectorVectorStore` rejects an ID conflict with another namespace
+and rolls back the entire upsert batch. IDs remain globally unique in its table.
+This closes the cross-namespace overwrite path; it does not install PostgreSQL
+roles or RLS policies, protect access through other database clients, or recover
+previously overwritten records.
+
+For shared databases, manage privileges and RLS through deployment migrations.
+The `_prepare_session` hook runs inside each operation's transaction and can
+apply transaction-local scope settings. Test using the actual application role,
+including pooled connection reuse and missing or foreign scope. See the
+[upgrade notes](migration.md#postgresql-vector-store-safety-in-26100) and
+[vector-store guide](vectorstores.md#tenantworkspace-scoping).
+
 ## Architecture
 
 This shows an explicitly configured input/output scanning path. Add

docs/vectorstores.md

@@ -164,6 +164,17 @@ must bind scopes to authenticated callers and manage database permissions.
 
 Install: `pip install fireflyframework-agentic[vectorstores-pgvector]`
 
+For an existing application, follow the [26.10.0 upgrade notes](migration.md#postgresql-vector-store-safety-in-26100)
+and [pinned release installation](getting-started/installation.md#install-a-release).
+`create_schema=False` leaves schema creation to migrations and checks that the
+`vector` extension and table exist. Operators must separately verify the table's
+dimensions, indexes, permissions, and any RLS policies.
+
+There is no supported PRISM selector or `prism.nprobe` setting in this adapter.
+The [pg_vectorsearch evaluation](pg-vectorsearch-evaluation.md) records the
+compatibility and filtered-recall reasons for retaining pgvector/HNSW. Adding a
+second index manually does not guarantee which index PostgreSQL will choose.
+
 ### sqlite-vec
 
 A sqlite-vec (`vec0` virtual table) backend that co-resides inside an existing
@@ -243,17 +254,22 @@ Note: Filter operator support varies by backend. `eq` is universally supported.
 
 ## Namespaces
 
-All operations accept a `namespace` parameter to isolate documents:
+All operations accept a `namespace` parameter to scope documents. Use distinct
+IDs when writing to different namespaces in a shared PostgreSQL table:
 
 ```python
-await store.upsert(docs, namespace="project-a")
-await store.upsert(docs, namespace="project-b")
+from fireflyframework_agentic.vectorstores import VectorDocument
+
+project_a_docs = [VectorDocument(id="project-a:1", text="Project A notes")]
+project_b_docs = [VectorDocument(id="project-b:1", text="Project B notes")]
+await store.upsert(project_a_docs, namespace="project-a")
+await store.upsert(project_b_docs, namespace="project-b")
 
 # Search only within a namespace
 results = await store.search_text("query", namespace="project-a")
 
 # Delete only within a namespace
-await store.delete(["id-1"], namespace="project-a")
+await store.delete(["project-a:1"], namespace="project-a")
 ```
 
 ---
@@ -305,8 +321,9 @@ every read and write must be confined to a `(tenant_id, workspace_id)` scope, an
 forgetting the scope must fail loudly rather than silently leak across tenants.
 
 `TenantScopedVectorStore` wraps **any** `VectorStoreProtocol` backend and folds the
-scope into the canonical `"t/<tenant_id>/w/<workspace_id>"` namespace, so a single
-wrapper makes every backend multi-tenant.
+scope into the canonical `"t/<tenant_id>/w/<workspace_id>"` namespace. The wrapper
+requires explicit scope on each operation; the application must authorize that
+scope and choose a backend whose storage behavior meets its isolation needs.
 
 ```python
 from fireflyframework_agentic.vectorstores import (
@@ -328,9 +345,10 @@ await store.close()
 
 On `upsert`, each document is copied (never mutating the caller's objects), its
 namespace is set to the scope, and -- when `stamp_metadata=True` (default) -- the
-scope is also stamped onto `metadata["tenant_id"]` / `metadata["workspace_id"]` as
-defense-in-depth. A caller that forgets `tenant_id`/`workspace_id` fails with
-`TypeError`, so isolation can never be lost silently.
+scope is also stamped onto `metadata["tenant_id"]` / `metadata["workspace_id"]`.
+These stamps describe the scope; they are not an authorization boundary. A caller
+that forgets `tenant_id`/`workspace_id` fails with `TypeError`. Document IDs are
+preserved, so PostgreSQL's table-wide uniqueness rule still applies.
 
 The namespace codec is exported directly:
 

mkdocs.yml

@@ -139,6 +139,7 @@ nav:
   - Embeddings & Vector Stores:
       - Embeddings: embeddings.md
       - Vector Stores: vectorstores.md
+      - pg_vectorsearch Evaluation: pg-vectorsearch-evaluation.md
   - Intelligence:
       - Reasoning Patterns: reasoning.md
       - Validation & QoS: validation.md