From e69515805407ad22a5e99151de9dbc3c12850659 Mon Sep 17 00:00:00 2001 From: Helia Mohammadi Date: Tue, 27 Oct 2026 17:51:51 -0400 Subject: [PATCH] retire obsolete spec plans and clarify historical records --- specs/001-retriever-rewrite/plan.md | 156 -------- specs/001-retriever-rewrite/spec.md | 221 +++++------ .../checklists/requirements.md | 50 --- specs/002-default-llm-choice/spec.md | 250 +++++------- .../checklists/requirements.md | 53 --- specs/003-model-configuration/plan.md | 176 --------- specs/003-model-configuration/quickstart.md | 29 +- specs/003-model-configuration/spec.md | 289 +++++++------- specs/003-model-configuration/tasks.md | 119 ------ .../checklists/requirements.md | 64 ---- specs/004-answer-in-user-language/plan.md | 99 ----- specs/004-answer-in-user-language/spec.md | 237 +++++------- specs/004-answer-in-user-language/tasks.md | 52 --- .../checklists/requirements.md | 43 +-- specs/005-mcp-live-data/spec.md | 194 ++++++---- .../checklists/requirements.md | 35 +- specs/006-mcp-hosting/spec.md | 174 +++++---- .../checklists/requirements.md | 30 +- specs/007-answer-cascade/spec.md | 265 ++++++------- specs/008-disease-variant-embeddings/spec.md | 107 ++++-- .../contracts/intent_classifier.md | 16 +- specs/009-collection-routing/data-model.md | 48 ++- specs/009-collection-routing/plan.md | 116 ------ specs/009-collection-routing/quickstart.md | 65 ---- specs/009-collection-routing/research.md | 83 ++-- specs/009-collection-routing/spec.md | 98 ++--- specs/009-collection-routing/tasks.md | 119 ++---- .../contracts/answer_endpoint.md | 344 +++++++++++------ specs/010-search-page-answers/plan.md | 108 ------ specs/010-search-page-answers/quickstart.md | 46 --- specs/010-search-page-answers/research.md | 137 ++++--- specs/010-search-page-answers/spec.md | 357 ++++++++++-------- specs/010-search-page-answers/tasks.md | 127 ++----- .../checklists/requirements.md | 51 --- .../contracts/summary_endpoint.md | 267 +++++++++---- .../data-model.md | 130 ++++--- specs/011-summarise-analysis-results/plan.md | 115 ------ .../quickstart.md | 67 ++-- .../research.md | 139 +++++-- specs/011-summarise-analysis-results/spec.md | 198 +++++----- specs/011-summarise-analysis-results/tasks.md | 182 +-------- specs/012-run-gene-analysis/spec.md | 246 ++++++------ specs/013-continue-chat-hand/spec.md | 194 +++++----- 43 files changed, 2368 insertions(+), 3528 deletions(-) delete mode 100644 specs/001-retriever-rewrite/plan.md delete mode 100644 specs/002-default-llm-choice/checklists/requirements.md delete mode 100644 specs/003-model-configuration/checklists/requirements.md delete mode 100644 specs/003-model-configuration/plan.md delete mode 100644 specs/003-model-configuration/tasks.md delete mode 100644 specs/004-answer-in-user-language/checklists/requirements.md delete mode 100644 specs/004-answer-in-user-language/plan.md delete mode 100644 specs/004-answer-in-user-language/tasks.md delete mode 100644 specs/009-collection-routing/plan.md delete mode 100644 specs/009-collection-routing/quickstart.md delete mode 100644 specs/010-search-page-answers/plan.md delete mode 100644 specs/010-search-page-answers/quickstart.md delete mode 100644 specs/011-summarise-analysis-results/checklists/requirements.md delete mode 100644 specs/011-summarise-analysis-results/plan.md diff --git a/specs/001-retriever-rewrite/plan.md b/specs/001-retriever-rewrite/plan.md deleted file mode 100644 index c695ca3d..00000000 --- a/specs/001-retriever-rewrite/plan.md +++ /dev/null @@ -1,156 +0,0 @@ -# Implementation Plan: Retriever Rewrite - -**Branch**: `plan/retriever-rewrite` | **Date**: 2026-09-08 | **Spec**: [spec.md](./spec.md) - -**Status**: Executed. See [Outcome](./spec.md#outcome-2026-09-09) in the spec — -two of the exit criteria below turned out to be unfalsifiable or wrong, and -which ones is recorded there. - -**Input**: Feature specification from `/specs/001-retriever-rewrite/spec.md` - -## Summary - -Replace `HybridRetriever`, which subclasses `MultiQueryRetriever` and reaches into -LangChain internals in five places, with a plain `BaseRetriever` that owns its own -fusion. Swap `SelfQueryRetriever` for plain semantic search (D1), keep BM25, and -make the context budget an argument rather than a module constant (D2, D3). - -The work is deliberately staged so that a behaviour change and a structural change -are never in the same commit. Stage 1 is a pure refactor that must produce -byte-identical retrieval; only Stage 2 changes what the model sees. - -## Technical Context - -**Language/Version**: Python 3.12 - -**Primary Dependencies**: `langchain-core` (BaseRetriever, Document), `langchain-chroma`, -`langchain-community` (BM25Retriever), `rank-bm25`, `nltk`. The point of the rewrite -is to depend on `langchain-core` contracts only. - -**Storage**: Chroma collections on disk, from an installed embeddings bundle; -sibling `csv_files/*.csv` for BM25. - -**Testing**: pytest. `tests/retrievers/test_hybrid_retriever.py` already pins the -fusion behaviour, including the tie-break and the `[1/n]*n` weighting that cannot -affect ordering. `bin/retrieval_baseline` for before/after measurement. - -**Target Platform**: Linux server (container), and developer machines. - -**Project Type**: Library within a single application repository. - -**Performance Goals**: Retrieval makes exactly one LLM call per message, down -from 21 (FR-009). No wall-clock target; the LLM answer dominates. - -**Constraints**: Sync and async must agree (FR-007). Behaviour must be measurable -with the committed question set before and after (FR-008). - -**Scale/Scope**: 4 Reactome collections, ~121k documents total; `userguide` has its -own retriever and is out of scope. - -## Constitution Check - -*GATE: must pass before implementation, re-checked after.* - -| Article | How this plan satisfies it | -|---|---| -| I — verify the user path | Every stage ends with the full RAG chain answering a real question through **both** `invoke` and `ainvoke`, not just the retriever in isolation. This is the failure mode that recurred most. | -| II — measure, don't argue | `bin/retrieval_baseline capture` before Stage 1 and after each stage; Stage 1 must show **zero** difference. | -| III — characterization tests pin behaviour | The existing retriever tests must pass unchanged through Stage 1. Any test that must change in Stage 2 changes together with the code and says why. | -| IV — fail loudly | Missing bundle raises with an actionable message rather than passing `None` into a `Path` (FR-006), removing the `B008` suppression and four mypy baseline entries. | -| V — derive from the source of truth | Budget and over-fetch stay single named parameters; the harness keeps importing them from the module rather than re-declaring. | -| VI — bias to doing over filing | Stages are small enough to land individually; nothing here is deferred to an issue that could be done now. | - -**No violations.** The one judgement call is Stage 3 (budget as an argument), which -could be deferred — it is kept because the interface should be settled while the -file is open rather than bolted on later, per spec User Story 2. - -## Project Structure - -### Documentation (this feature) - -```text -specs/001-retriever-rewrite/ -├── spec.md # what and why, with D1–D4 settled -└── plan.md # this file -``` - -No `research.md`, `data-model.md` or `contracts/` are generated. There are no -unresolved unknowns — D1–D4 are decided and the evidence is in the spec — and the -feature introduces no new data model or external contract. Generating empty -scaffolding would be ceremony. - -### Source Code - -```text -src/retrievers/ -├── csv_chroma.py # HybridRetriever -> plain BaseRetriever; owns RRF -├── reactome/rag.py # bundle becomes an explicit argument -├── uniprot/rag.py # same -├── plantreactome/rag.py # same -└── userguide/ # already a plain BaseRetriever; the model to follow - -tests/retrievers/ -└── test_hybrid_retriever.py # extended, not rewritten - -bin/retrieval_baseline # already mirrors the pipeline -``` - -## Implementation Stages - -### Stage 1 — Structural only, zero behaviour change - -Replace the `MultiQueryRetriever` subclass with a `BaseRetriever` that implements -`_get_relevant_documents` / `_aget_relevant_documents`, and vendor Reciprocal Rank -Fusion into this repository (FR-001, FR-002). - -Query expansion stays (D4) and `SelfQueryRetriever` stays for now — this stage -changes *how the code is arranged*, not what it returns. - -RRF is copied from `EnsembleRetriever.weighted_reciprocal_rank` as it behaves -today: `weight / (rank + 60)` with rank counted from 1, de-duplicated on -`page_content`, ties resolved by first appearance. Those properties are already -pinned by tests. - -**Exit criteria**: existing tests pass unchanged; `retrieval_baseline compare` -shows **zero** differences; the full chain answers a real question through both -`invoke` and `ainvoke`. - -### Stage 2 — D1: plain semantic search replaces SelfQuery - -Swap `SelfQueryRetriever` for `vectordb.as_retriever(...)` (FR-009). Retrieval -drops to one LLM call per message. - -`metadata_info.py` stays: `evaluator.py` and `bin/retrieval_baseline` still -construct SelfQuery for comparison, so deleting it would break the tool that -measures this change. - -**Exit criteria**: `retrieval_baseline compare` shows differences confined to the -vector side; a test asserts retrieval makes exactly one LLM call; the chain -answers real questions in both paths. - -### Stage 3 — D2/D3: the caller supplies the budget - -`MAX_DOCUMENTS_PER_COLLECTION` becomes a constructor argument with today's value -as the default (FR-005). The bundle becomes an explicit argument, removing the -`B008` suppression and the four `retrievers.*.rag` mypy baseline entries (FR-006, -SC-006). - -**Exit criteria**: two callers with different budgets in one process get different -amounts of context; the four baseline entries are deleted; gates green. - -## Complexity Tracking - -| Decision | Simpler alternative rejected | Why | -|---|---|---| -| Vendor RRF (~15 lines) | Keep calling `EnsembleRetriever.weighted_reciprocal_rank` | The method is public, but reaching it requires `EnsembleRetriever(retrievers=[])` — an empty retriever built solely to borrow a method. Ranking is the product's core quality; a library upgrade should not be able to reorder results silently. | -| Three stages | One commit | A behaviour change and a structural change in the same diff cannot be attributed when results move. Same reason the rewrite precedes the LangChain upgrade. | -| Keep `metadata_info.py` | Delete 339 dead lines | It is not dead: the evaluator and the baseline harness still construct SelfQuery. Deleting it would break the tool that measures Stage 2. | -| Keep multi-query expansion | Remove it too | D4. Removing two things at once makes any quality change unattributable. Measured separately afterwards. | - -## Out of Scope - -- The LangChain upgrade itself. This unblocks it; it does not perform it. -- Answer quality evaluation (ragas). This plan changes what reaches the model; - whether answers improve is measured separately. -- `userguide` retriever — already a plain `BaseRetriever`. -- Token-based budgeting (D3 caveat) and removing expansion (D4 follow-up). diff --git a/specs/001-retriever-rewrite/spec.md b/specs/001-retriever-rewrite/spec.md index 2dc222b7..22769fef 100644 --- a/specs/001-retriever-rewrite/spec.md +++ b/specs/001-retriever-rewrite/spec.md @@ -4,11 +4,11 @@ **Created**: 2026-09-08 -**Status**: **Complete.** All three stages shipped 2026-09-09 (#182, #183, #185). Outcome recorded at the end of this file, including two predictions that were wrong. +**Status**: Historical record of the retriever rewrite, 2026-09-08 to 2026-09-09 (#182, #183, #185). The decisions, requirements and scope below were set for that work and are not requirements for later changes. Where the code has changed since, the section says so. Current behaviour is in `src/retrievers/csv_chroma.py`. The outcome, including two predictions that were wrong, is recorded at the end of this file. **Input**: Rewrite the Reactome retriever: replace the `HybridRetriever` that subclasses `MultiQueryRetriever` with a plain `BaseRetriever`, decide whether `SelfQueryRetriever` is replaced by plain semantic search, and make the context budget caller-supplied. -## Why now +## Why the rewrite was done (2026-09-08) Three things converge on this component: @@ -16,18 +16,18 @@ Three things converge on this component: 2. **Three open PRs are queued behind it** — #116 (FlashRank reranker), #133 (EmbeddingsFilter), and the config work in #112/#151 — because none should be built on a class about to change shape. 3. **The upcoming surfaces need different context budgets.** Chat, chat-alongside-search-results, and analysis summarisation each want a different amount of retrieved material, and the budget is currently a module constant. -## What exists today +## The retriever before the rewrite (2026-09-08) -Per user message, `HybridRetriever`: +Per user message, the pre-rewrite `HybridRetriever`: 1. **expands** the question into 4 alternates via one LLM call (5 queries with `include_original`) 2. for each of **4 Chroma collections** (`reactions`, `summations`, `complexes`, `ewas`), for each query: BM25 over the collection CSV, and a `SelfQueryRetriever` over the Chroma collection — the latter being one LLM call each, so **20 LLM calls per message** for query construction 3. **fuses** BM25 and vector results as separate ranked lists via LangChain's RRF, per collection 4. **caps** at 10 documents per collection, giving 40 documents (~6,300 tokens) -Steps 3 and 4 were fixed in the week of 2026-09-04 (issues #169, #170); steps 1 and 2 are unchanged. +Steps 3 and 4 were fixed in the week of 2026-09-04 (issues #169, #170); steps 1 and 2 were unchanged at the time. -### The five reaches into LangChain internals +### The five reaches into LangChain internals (line numbers refer to the file before #182) | location | what it does | |---|---| @@ -39,91 +39,42 @@ Steps 3 and 4 were fixed in the week of 2026-09-04 (issues #169, #170); steps 1 `retrieve_documents` / `aretrieve_documents` are also overrides of internal methods, not documented extension points. -## User Scenarios & Testing *(mandatory)* +## What the rewrite set out to do -### User Story 1 — Retrieval survives a LangChain major version (Priority: P1) +Three goals, in priority order: retrieval that depends only on `BaseRetriever`, +`Document` and the Runnable interface, so a LangChain major upgrade does not break +it; a context budget the caller sets; and a per-message LLM-call cost that is +understood and reducible. The Outcome section records how each landed. -A developer upgrades LangChain. The retriever continues to work, because it depends only on `BaseRetriever`, `Document` and the Runnable interface — the contracts LangChain has held stable across every version — rather than on internals. +## Requirements of the rewrite, and where they stand (checked 2026-10-21) -**Why this priority**: This is the blocker. The lockfile carries 43 advisories, concentrated in the LangChain line, and they cannot be addressed while the retriever depends on internals. - -**Independent Test**: Upgrade LangChain in a branch. The suite passes and `bin/retrieval_baseline compare` shows no change attributable to the retriever's own logic. - -**Acceptance Scenarios**: - -1. **Given** the rewritten retriever, **When** `mypy` and `pytest` run, **Then** no code references `MultiQueryRetriever`, `EnsembleRetriever` internals, or `SkipJsonSchema` field overrides. -2. **Given** a question, **When** retrieval runs through both `invoke` and `ainvoke`, **Then** both return the same documents in the same order for the same query set. -3. **Given** the RRF fusion, **When** documents tie on score, **Then** the tie-break is defined by the implementation rather than inherited from a library's iteration order. - ---- - -### User Story 2 — The caller states how much context it wants (Priority: P2) - -A surface asks for retrieval and specifies its own budget. The chat surface asks for more; a chat answer displayed beside full search results asks for less, because the search results are the comprehensive part; analysis summarisation asks for something different again. - -**Why this priority**: Independently valuable and independently testable, but the interface should be settled during the rewrite rather than bolted on. Recorded in #172. - -**Independent Test**: Call the retriever twice with different budgets and assert the returned document counts differ accordingly, with no change to module-level state. - -**Acceptance Scenarios**: - -1. **Given** a caller-supplied budget of N documents per collection, **When** retrieval runs, **Then** at most N documents per collection are returned. -2. **Given** no budget supplied, **When** retrieval runs, **Then** a documented default applies and the behaviour matches today's. -3. **Given** two callers with different budgets in the same process, **When** both retrieve, **Then** neither affects the other. - ---- - -### User Story 3 — Retrieval cost per message is understood and reducible (Priority: P3) - -An operator can see, and change, how many LLM calls a single message costs in retrieval. - -**Why this priority**: Real but not blocking. Worth settling while the code is open, since the answer determines whether `SelfQueryRetriever` stays. - -**Independent Test**: Count LLM calls for one message before and after; assert the number matches what the configuration implies. - -**Acceptance Scenarios**: - -1. **Given** a question, **When** retrieval runs, **Then** the number of LLM calls is derivable from the number of collections, query variants, and whether self-querying is enabled. -2. **Given** self-querying is disabled, **When** retrieval runs, **Then** no LLM call is made for query construction. - -### Edge Cases - -- No embeddings bundle installed — must fail with an actionable message, not `None` flowing into a `Path` parameter (current `B008` / mypy baseline). -- A collection whose documents carry no `st_id` — de-duplication must fall back rather than raise. -- A bundle whose collections were built with different embedding models — already reported by `resolve_embedding_model`; retrieval must not silently proceed. -- Zero results from one retriever for a query — fusion must not divide by zero or drop the other retriever's results. -- A budget of 0 or larger than the corpus. - -## Requirements *(mandatory)* - -### Functional Requirements - -- **FR-001**: The retriever MUST implement `langchain_core.retrievers.BaseRetriever` via `_get_relevant_documents` / `_aget_relevant_documents` and MUST NOT subclass `MultiQueryRetriever`. -- **FR-002**: Reciprocal Rank Fusion MUST be owned by this repository rather than borrowed from `EnsembleRetriever`, and its constant, tie-break and de-duplication key MUST be explicit and tested. -- **FR-003**: BM25 and plain semantic results MUST be fused as separate ranked lists - (preserving the #170 fix), and the BM25-vs-vector weighting MUST be adjustable -- - today it is `[1/n]*n`, which cannot affect ordering at all. -- **FR-004**: Results MUST be de-duplicated per Reactome entity, falling back to page content where no `st_id` exists (preserving the #169 fix). -- **FR-005**: The context budget MUST be supplied per call, counted in documents per - collection (D2, D3), with a documented default matching today's behaviour. -- **FR-006**: The embeddings bundle MUST be an explicit argument, not resolved at import time in a default argument. -- **FR-007**: Synchronous and asynchronous paths MUST return identical results for identical inputs, and MUST be tested together. -- **FR-008**: Behaviour MUST be measured with `bin/retrieval_baseline` before and after, - on the committed question set. -- **FR-009**: The vector retriever MUST NOT make an LLM call (D1). Retrieval MUST make - exactly one LLM call per message, for query expansion (D4). +| ID | Required by the rewrite | Today | +|---|---|---| +| FR-001 | Plain `BaseRetriever`, not a `MultiQueryRetriever` subclass | Holds (`csv_chroma.py:387`) | +| FR-002 | RRF owned here; constant, tie-break and de-dup key explicit and tested | Holds: `reciprocal_rank_fusion`, `RRF_K = 60`, ties by first appearance, keyed on `page_content` (`csv_chroma.py:218-250`) | +| FR-003 | BM25 and vector fused as separate lists; BM25-vs-vector weighting adjustable | Separate lists hold (`csv_chroma.py:610-611`). Adjustable weighting was not built: `weighted_reciprocal_rank` always uses equal weights (`csv_chroma.py:570-574`) | +| FR-004 | De-duplicate per entity, falling back to page content | Holds: `dedupe_by_entity` (`csv_chroma.py:199`) | +| FR-005 | Budget supplied per call, in documents per collection | Built as a constructor argument, `max_documents_per_collection` (default 10), not per call. No current caller passes one, so every retriever uses the default (`csv_chroma.py:366-379`) | +| FR-006 | Bundle passed explicitly | Holds (`embeddings_directory` parameter) | +| FR-007 | Sync and async identical, tested together | Holds (`tests/retrievers/test_sync_async_equivalence.py`) | +| FR-008 | Measured with `bin/retrieval_baseline` before and after | A process requirement for the rewrite itself | +| FR-009 | No LLM call on the vector side; exactly one per message, for expansion | The vector side holds. One call per message by default; none when `QUERY_EXPANSION_ALTERNATES=0`, added 2026-09-20 (see D4) | ### Key Entities -- **Collection** — one Chroma directory plus its BM25 source CSV (`reactions`, `summations`, `complexes`, `ewas`, and `userguide` on its own retriever). +- **Collection** — one Chroma directory plus its BM25 source CSV. The retriever + searches whatever collections the installed bundle has. At the time of the + rewrite, the Reactome bundle had `reactions`, `summations`, `complexes` and + `ewas`; `disease_variants` was added later (spec 008). `userguide` has its own + retriever. - **Document** — a `langchain_core.documents.Document`, identified by `st_id` where present. - **Budget** — how many documents a caller wants, per collection. ## Decisions -### D1 — `SelfQueryRetriever` is replaced by plain semantic search — DECIDED (Helia) +### D1 — `SelfQueryRetriever` replaced by plain semantic search (decision for the rewrite, 2026-09-08) -BM25 stays; the vector side becomes `vectordb.as_retriever(search_kwargs={"k": ...})` +BM25 was kept; the vector side became `vectordb.as_retriever(search_kwargs={"k": ...})`, with no LLM in the loop. **Consequences, stated plainly because this is not a free simplification:** @@ -150,10 +101,15 @@ with no LLM in the loop. real change in what reaches the model, not the removal of noise. - **Metadata filtering is lost as a capability**, not merely as code. SelfQuery translated a question into a Chroma metadata filter; nothing replaces that. -- **339 lines across the `metadata_info.py` files become dead in the retrieval - path**, still referenced only by `src/evaluation/evaluator.py` and - `bin/retrieval_baseline`, which construct SelfQuery for comparison. They should - not be deleted while those still need them. + Still the case: the vector retriever is configured with `k` only + (`csv_chroma.py:484`). +- **`metadata_info.py` left the retrieval path.** At the time, `src/evaluation/evaluator.py` + and `bin/retrieval_baseline` still used it to build SelfQuery for comparison. + Since then, the evaluator calls the shipping pipeline (cf9ad6a), and + `reactome_descriptions_info` has become a serving input: the intent classifier + builds its collection list from it (`src/agent/tasks/intent_classifier.py:55-67`), + and so does `bin/routing-probe`. `bin/retrieval_baseline` still builds SelfQuery + from it. **Verification**: done. The figures above supersede the 0.48/0.19 in #171, which predated the #169 and #170 fixes and described a retriever that no longer exists. @@ -162,7 +118,7 @@ real configuration; rank agreement was unchanged at ~0.18. The conclusion is unmoved: this is a substantial change to retrieval, and it should be judged on answer quality rather than on overlap, which cannot say which set is better. -### D2 — The budget is per collection — DECIDED +### D2 — Budget counted per collection (decision for the rewrite) Not global. The decisive reason is technical rather than editorial: each collection is fused independently, so an RRF score of 0.03 in `reactions` and @@ -171,30 +127,35 @@ a comparison that has no meaning. The secondary reason is coverage: `reactions` holds mechanism, `summations` holds prose description, `complexes` composition, `ewas` protein identity. A global cap -could return 40 reactions and no descriptions. Measured today, the per-collection -cap yields an even 10/10/10/10 split. +could return 40 reactions and no descriptions. Measured 2026-09-08, the +per-collection cap yielded an even 10/10/10/10 split. -Revisit only if the collections are ever fused into one ranked list, at which -point the scores would be comparable and this reasoning would no longer hold. +**Later change (38f8885, 2026-09-18).** The per-collection lists are now also fused +with each other by RRF, so the final list is ranked across collections instead of +grouped by collection. The per-collection cap still applies first: each searched +collection contributes at most `max_documents_per_collection` documents, and the +fused list is cut to that cap times the number of collections searched. The change +therefore reorders documents without dropping any (`csv_chroma.py:612-633`). A +global top-N, which the reasoning above argued against, has not been introduced. -### D3 — The budget is counted in documents — DECIDED, with a caveat +### D3 — Budget counted in documents (decision for the rewrite) Tokens are the better unit in principle, and #139 was right to say so. But the -measurement does not currently justify the complexity: 40 documents span -5,796-9,531 tokens, a 1.6x spread, at roughly 7% of the context window. A token -ceiling would tighten a bound that is not binding. +measurement did not justify the complexity: 40 documents spanned 5,796-9,531 +tokens (measured 2026-09-08, four collections), a 1.6x spread, at roughly 7% of +the context window. A token ceiling would have tightened a bound that was not +binding. Documents are also the unit a caller can reason about when asking for "less" alongside search results. -**Caveat**: keep it a single named parameter so a token ceiling can be added as a -second bound without changing the interface. `tiktoken` is already in the -lockfile via `langchain-openai`, so that costs no new dependency when it is -wanted. +The rewrite kept the budget as one named parameter so that a token ceiling could +be added later as a second bound without changing the interface. None has been +added. `tiktoken` is in the lockfile. -### D4 — Multi-query expansion stays, for now — DECIDED +### D4 — Multi-query expansion kept during the rewrite -Keep it, and measure it separately afterwards. +The rewrite left expansion unchanged and left measuring it to a separate change. The reason is not that it is known to be valuable — its value has never been measured — but that D1 already changes retrieval materially. Removing expansion @@ -202,34 +163,29 @@ in the same change would make any quality difference impossible to attribute. This is the same principle as rewriting before upgrading LangChain: change one variable at a time. -It is also cheap in relative terms once D1 lands: with self-querying gone, -expansion is the *only* LLM call in retrieval. Its real cost is multiplying -retrieval work by five, which is local Chroma and BM25 work rather than API -calls. - -**Follow-up**: measure expansion on its own after D1, using -`bin/retrieval_baseline` with and without it. +With self-querying gone, expansion became the *only* LLM call in retrieval. Its +other cost is multiplying retrieval work by five, which is local Chroma and BM25 +work rather than API calls. -## Success Criteria *(mandatory)* +**Follow-up, done 2026-09-20 (spec 010, T020).** Over six tracked questions, the +expansion call cost about 1.27s whatever it returned, so fewer alternates save +only fan-out (table in `csv_chroma.py:89-112`). The count became configurable +through `QUERY_EXPANSION_ALTERNATES` (default 4; 0 skips the call). The default +stayed at 4. That measurement covered the tracked questions only and does not +establish that recall is unaffected in general. -### Measurable Outcomes +**How expansion works now:** one LLM call asks for exactly four alternates +(`multi_query_prompt`, `csv_chroma.py:131`). The retriever uses the first N, where +N is the configured count, and the Reactome retriever appends the original +question last (`include_original=True`, `csv_chroma.py:377`; `_queries`, `:536`). +A count above four does not add queries. -- **SC-001**: No reference to LangChain internals remains in the retriever; the LangChain upgrade proceeds without further retriever changes. -- **SC-002**: `bin/retrieval_baseline compare` before and after shows only differences attributable to a decision recorded above, not to incidental refactoring. -- **SC-003**: Sync and async return identical documents for the same query set, asserted by a test. -- **SC-004**: Two callers with different budgets get different amounts of context in the same process. -- **SC-005**: Retrieval makes exactly one LLM call per message, down from 21, asserted - by a test that counts them. -- **SC-006**: The four mypy baseline entries for `retrievers.*.rag` -- reactome, uniprot, - plantreactome and userguide -- are deleted, not carried forward. All four share the - same cause: `EmbeddingEnvironment.get_dir()` resolved in a default argument (FR-006). - -## Assumptions +## Assumptions at the time (2026-09-08) - The bundle format is unchanged: `///` containing per-collection Chroma directories and a sibling `csv_files/`. -- BM25 stays. Helia's proposal keeps it, it is deterministic, and it costs no LLM call. +- BM25 was kept: it is deterministic and costs no LLM call. - Answer quality is judged by the ragas work, not by this spec. This spec changes *what reaches the model*; whether that improves answers is measured separately. -- The rewrite happens before the LangChain upgrade, so that a behaviour change and an upgrade change cannot be confused for one another. +- The rewrite was to precede the LangChain upgrade, so a behaviour change and an upgrade change could not be confused. --- @@ -271,11 +227,20 @@ them. It changed what reaches the model four times over — over-fetch, de-duplication, genuine BM25/vector fusion, and plain semantic search — and **none of those changes -has been evaluated for answer quality**. The plan said so under Out of Scope, which -made it a known gap rather than an oversight, but it is still a gap. - -`src/evaluation/evaluator.py` is the tool for it and cannot be used as it stands: it -builds its own `SelfQueryRetriever` + `EnsembleRetriever` + `MergerRetriever` rather -than the shipping pipeline, so after Stage 2 it measures a configuration that no -longer exists. Carried into [spec 002](../002-default-llm-choice/spec.md) as its P1, -where the same fix serves both. +has been evaluated for answer quality**. The rewrite's plan said so under Out of +Scope, which made it a known gap rather than an oversight, but it is still a gap. + +At the time, `src/evaluation/evaluator.py` could not measure the shipping pipeline: +it built its own `SelfQueryRetriever` + `EnsembleRetriever` + `MergerRetriever`. It +was changed on 2026-09-09 to call `create_reactome_rag` (cf9ad6a; see +`src/evaluation/README.md`). This repository does not record whether an +answer-quality evaluation of the four retrieval changes has been run since. + +### Alternatives rejected during the rewrite + +| Choice | Simpler alternative rejected | Why | +|---|---|---| +| Vendor RRF (~15 lines) | Keep calling `EnsembleRetriever.weighted_reciprocal_rank` | The method is public, but reaching it required `EnsembleRetriever(retrievers=[])` -- an empty retriever built solely to borrow a method. Ranking is the product's core quality; a library upgrade should not be able to reorder results silently. | +| Three stages | One commit | A behaviour change and a structural change in the same diff cannot be attributed when results move. Same reason the rewrite preceded the LangChain upgrade. | +| Keep `metadata_info.py` | Delete it | The evaluator and the baseline harness still built SelfQuery from it, so deleting it would have broken the tool that measured Stage 2. D1 records how its use has changed since. | +| Keep multi-query expansion | Remove it too | D4. Removing two things at once makes any quality change unattributable. | diff --git a/specs/002-default-llm-choice/checklists/requirements.md b/specs/002-default-llm-choice/checklists/requirements.md deleted file mode 100644 index af972e58..00000000 --- a/specs/002-default-llm-choice/checklists/requirements.md +++ /dev/null @@ -1,50 +0,0 @@ -# Specification Quality Checklist: Choosing the Default Answering Model - -**Purpose**: Validate specification completeness and quality before proceeding to planning -**Created**: 2026-09-09 -**Feature**: [spec.md](../spec.md) - -## Content Quality - -- [x] No implementation details (languages, frameworks, APIs) -- [x] Focused on user value and business needs -- [x] Written for non-technical stakeholders -- [x] All mandatory sections completed - -## Requirement Completeness - -- [x] No [NEEDS CLARIFICATION] markers remain -- [x] Requirements are testable and unambiguous -- [x] Success criteria are measurable -- [x] Success criteria are technology-agnostic (no implementation details) -- [x] All acceptance scenarios are defined -- [x] Edge cases are identified -- [x] Scope is clearly bounded -- [x] Dependencies and assumptions identified - -## Feature Readiness - -- [x] All functional requirements have clear acceptance criteria -- [x] User scenarios cover primary flows -- [x] Feature meets measurable outcomes defined in Success Criteria -- [x] No implementation details leak into specification - -## Notes - -Three deviations from the template, each deliberate: - -1. **Model and file names appear in the spec.** `gpt-5.6-luna`, `evaluator.py`, - `resolve_temperature`. The subject of this decision *is* a named third-party - model and a named drifted file; writing around them would make the document - unusable for the three people who have to act on it. - -2. **The evidence section leads.** The template puts user stories first. Here the - measurements are the substance and the stories follow from them — and two of - the measurements are explicitly labelled as too weak to decide on, which is the - most important thing on the page. - -3. **No [NEEDS CLARIFICATION] markers, but three open questions.** They are not - gaps in the specification — the spec is complete and actionable as written. They - are decisions reserved to the team, recorded in "Decisions for the team" so they - are answered once and in the open rather than assumed. The P1 work (FR-001..004) - proceeds regardless of how they are answered. diff --git a/specs/002-default-llm-choice/spec.md b/specs/002-default-llm-choice/spec.md index c2dbcbce..77664018 100644 --- a/specs/002-default-llm-choice/spec.md +++ b/specs/002-default-llm-choice/spec.md @@ -4,7 +4,7 @@ **Created**: 2026-09-09 -**Status**: Evidence gathered, decision open. Three questions for the team below. +**Status**: Historical record of the default-model evaluation, 2026-09-09: the measurements taken then, and three questions (D1-D3) put to the team. No answer to D1-D3 and no choice of default is recorded in this repository; the default is still `gpt-4o-mini` (`src/agent/graph.py:172`). The evaluator work this spec asked for was done the same day (cf9ad6a). The requirements below were set for that work, not for later changes. How the model is chosen today is in [spec 003](../003-model-configuration/spec.md) and `resolve_llm_model` (`src/agent/graph.py:175-197`). **Input**: `gpt-4o-mini` is today's default. `gpt-5.6-luna` is now usable (#186). Does it become the default, on which surfaces, and what evidence settles it? @@ -23,22 +23,24 @@ answering model looks like a bump and is not one: None of that is recoverable from the diff. That is what this file is for. -## What has already been settled +## What was settled before this spec (2026-09-09) -`gpt-5.6-luna` **works end to end**. #186 removed the two things that stopped it: +`gpt-5.6-luna` worked end to end after #186, which removed the two things that stopped it: -| blocker | resolution | -|---|---| -| `temperature=0.0` hardcoded; the gpt-5.5/5.6/6 families accept only their default of 1 | `resolve_temperature()` per model family, `LLM_TEMPERATURE` to override | -| the three graders used `function_calling`, which gpt-5.6 refuses on `/v1/chat/completions` | `method="json_schema"`, verified on **both** models so the path cannot rot | +| blocker | resolution in #186 | today | +|---|---|---| +| `temperature=0.0` hardcoded; the gpt-5.5/5.6/6 families accept only their default of 1 | `resolve_temperature()` matching those name prefixes, `LLM_TEMPERATURE` to override | The prefix rule was wrong for eleven models and was replaced (#189) by an exact-match set of measured models, with dated snapshot suffixes stripped (`src/agent/graph.py:103-138`) | +| the three graders used `function_calling`, which gpt-5.6 refuses on `/v1/chat/completions` | `method="json_schema"`, verified on both models | Unchanged (`src/agent/tasks/safety_checker.py:97-99`, `intent_classifier.py:206`, `completeness_grader.py:38-40`) | -Switching is now one environment variable: `LLM_MODEL=gpt-5.6-luna`. **This -specification is about whether to set it, not how.** +Switching was one environment variable, `LLM_MODEL=gpt-5.6-luna`; since spec 003 it can +also be `llm.model` in `config.yml`. This spec was about whether to switch, not how. -## Evidence gathered +## Evidence gathered (2026-09-09) -Measured through `AgentGraph.ainvoke` on the React-to-Me profile — the entry point -`bin/chat-chainlit.py` uses — against the Release95 reactome bundle. +Measured on 2026-09-09 through `AgentGraph.ainvoke` on the React-to-Me profile — the +entry point `bin/chat-chainlit.py` used — against the Release95 reactome bundle. The +pipeline has changed since (a fifth collection, preprocessing in two rounds, collection +routing), so these figures describe that date, not current behaviour. ### Latency and shape (3 questions, averaged) @@ -81,153 +83,83 @@ the weakest evidence behind it. **A read of three answers is an anecdote.** - **Price per token.** No endpoint reports it. The premise "just as cheap" is unverified here and has to come from the pricing page or from a billing period. -- **Answer quality at any scale.** See below — the harness that would do it does - not currently measure the pipeline that ships. +- **Answer quality at any scale.** At the time, the harness that would do it did + not measure the pipeline that ships (next section). +- **Time to first token.** Not measured for either model here. Spec 010 later + measured it for `gpt-4o-mini` only (p50 9.6s, 2026-09-18; + `specs/010-search-page-answers/contracts/answer_endpoint.md`, "Budget"). -## The blocker behind the blocker +## The evaluator gap, and how it was closed `src/evaluation/evaluator.py` runs ragas with `faithfulness`, `answer_relevancy`, -`context_recall` and `context_utilization` over a golden set — exactly the metrics -that would settle the answer-quality question, and `faithfulness` is exactly the -axis on which luna appeared to differ. - -It cannot be used as it stands. It builds its own retriever — `SelfQueryRetriever` -+ `EnsembleRetriever` + `MergerRetriever` — rather than calling -`create_bm25_chroma_ensemble_retriever`. Stage 2 removed `SelfQueryRetriever` from -the shipping pipeline, so the evaluator now measures a configuration that no longer -exists. Running it today would produce numbers that look like an answer and are not. - -This is the same drift `bin/retrieval_baseline` had and had fixed: a measurement -tool that has quietly stopped measuring the thing. - -It also blocks work already owed: the four retrieval changes from the week of -2026-09-04 (#169, #170, over-fetch, D1) went in **unevaluated for answer quality**. -Pointing the evaluator at the real pipeline pays for both. - -## User Scenarios & Testing *(mandatory)* - -### User Story 1 — The evaluator measures what ships (Priority: P1) - -A developer changes the model, or the retriever, and runs one command that reports -faithfulness, answer relevancy, context recall and context utilization for the -pipeline a user actually talks to. - -**Why this priority**: Nothing else here can be decided without it, and it is owed -regardless of which model wins. It is the only item on this page whose value does -not depend on the outcome of the decision. - -**Independent Test**: Run the evaluator against `main` twice with no change in -between; the metrics agree within noise. Then swap the retriever's document budget -and watch context utilization move. - -**Acceptance Scenarios**: - -1. **Given** the evaluator, **When** it constructs the chain, **Then** it calls the - same factory `bin/chat-chainlit.py` reaches, so a change to retrieval cannot - alter the product without altering the measurement. -2. **Given** a golden question set, **When** the evaluator runs on two models, - **Then** it emits a per-metric comparison rather than two unrelated reports. -3. **Given** the evaluator, **When** `SelfQueryRetriever` no longer exists in the - pipeline, **Then** the evaluator does not construct one. - ---- - -### User Story 2 — The default model is chosen on evidence (Priority: P2) - -The team picks the default from a table of measured differences, not from a -recollection of three answers. - -**Why this priority**: This is the actual question. It is P2 only because it is -gated on P1. - -**Independent Test**: Both models are run over the golden set; the report shows -faithfulness, relevancy, recall, utilization, latency and token counts side by -side. - -**Acceptance Scenarios**: - -1. **Given** both models evaluated, **When** faithfulness differs by less than the - run-to-run noise, **Then** the difference is reported as "not measurable", not - as a win. -2. **Given** a chosen default, **When** it is committed, **Then** the reason and - the numbers are recorded here, so the next person does not re-derive them. - ---- - -### User Story 3 — Different surfaces may choose differently (Priority: P3) - -*Superseded: this is now [spec 003](../003-model-configuration/spec.md), which -covers how a model is chosen. This specification keeps only the question of which -model wins.* - -Chat, chat-alongside-search-results, and analysis summarisation each pick a model -suited to their tolerance for latency. - -**Why this priority**: Only chat exists today. Recording it now costs a paragraph; -discovering later that the model is a global constant costs a refactor — which is -the same mistake the context budget made, fixed in Stage 3. - -**Acceptance Scenarios**: - -1. **Given** a surface that must answer inside a search-results page, **When** it - builds its chain, **Then** it can request a faster model without changing what - the chat surface uses. - -### Edge Cases - -- **A model is added to a fixed-temperature family that the prefix table has not - met.** It returns a 400 on the first user question, not at startup. `LLM_TEMPERATURE` - is the escape hatch; the table is matched on prefix so dated snapshots are covered. -- **The evaluator's own judge model.** ragas uses an LLM to score. If the judge is - the same model being evaluated, the comparison is biased. The judge must be pinned - and stated, and must not change between the two runs being compared. -- **Latency on a streamed interface.** 41s to a complete answer is not 41s of - silence if tokens stream; the perceived cost depends on time-to-first-token, which - has not been measured and is the number a user actually experiences. - -## Requirements *(mandatory)* - -### Functional Requirements - -- **FR-001**: The evaluator MUST build its chain from the same factory the - application uses, so that retrieval changes cannot alter the product without - altering the measurement. -- **FR-002**: The evaluator MUST accept the model under test as an argument and - report metrics per model, so two models can be compared in one run. -- **FR-003**: The evaluator MUST pin and report the judge model separately from the - model under test. -- **FR-004**: The evaluator MUST report run-to-run variance, so a difference smaller - than noise is not read as a result. -- **FR-005**: The default model MUST remain selectable per deployment without a code - change. -- **FR-006**: A model whose temperature requirement is unknown MUST fail with a - message naming the environment variable that fixes it. -- **FR-007**: The chosen default and the numbers behind it MUST be recorded in this - specification when the decision is made. - -### Key Entities +`context_recall` and `context_utilization` over a question set — the metrics that +would settle the answer-quality question, and `faithfulness` is the axis on which +luna appeared to differ. + +When this spec was written it could not be used: it built its own retriever +(`SelfQueryRetriever` + `EnsembleRetriever` + `MergerRetriever`) rather than the +shipping one, and the retriever rewrite (spec 001) had removed `SelfQueryRetriever` +from the product, so it measured a configuration that no longer existed. The same +day it was changed to call `create_reactome_rag` (cf9ad6a), with a repeatable +`--model`, `--repeat`, and a pinned judge; see `src/evaluation/README.md`. + +The four retrieval changes from the week of 2026-09-04 (#169, #170, over-fetch, and +spec 001's D1) went in unevaluated for answer quality. This repository records no +evaluator run since, for them or for the model comparison. + +## What this work set out to do (2026-09-09) + +In priority order: make the evaluator measure the pipeline a user talks to (P1, User +Story 1), then choose the default model from a table of measured differences rather +than from three answers (P2, User Story 2). A third story, letting each surface choose +its own model, moved to [spec 003](../003-model-configuration/spec.md) the same day and +is unbuilt there. + +User Story 2 set two acceptance conditions: a faithfulness difference smaller than +run-to-run noise is reported as "not measurable" rather than as a win, and the chosen +default is recorded here with its numbers. Neither has been met, because no comparison +run is recorded. + +### Edge cases noted at the time + +- **A fixed-temperature model the table has not met.** It returns a 400 on the first + user question, not at startup; `LLM_TEMPERATURE` is the escape hatch. The table is + now an exact-match set, and dated snapshots (`-YYYY-MM-DD`) are covered by + stripping the suffix, not by prefix matching (`src/agent/graph.py:131-138`). +- **The evaluator's own judge model.** If the judge is the model being evaluated, the + comparison is biased. The evaluator pins the judge (`--judge-model`, default + `gpt-4o`) and refuses one that is also under test (`src/evaluation/evaluator.py:67`, + `:433-437`). Keeping it the same between two compared runs is left to whoever runs + it (`src/evaluation/README.md`, "The judge"). +- **Latency on a streamed interface.** 41s to a complete answer is not 41s of silence + if tokens stream; what a user feels is time to first token, which was not measured + for luna (see "What could not be measured"). + +## Requirements of this work, and where they stand (checked 2026-10-21) + +| ID | Required by this work | Today | +|---|---|---| +| FR-001 | Evaluator builds its chain from the application's factory | Holds with a gap: `build_chain` calls `create_reactome_rag` (`src/evaluation/evaluator.py:260-275`, cf9ad6a), but it does not apply collection routing (added 2026-09-19, spec 009), so it searches every collection. It also skips the safety check and intent classification (`src/evaluation/README.md:96-106`) | +| FR-002 | Model under test as an argument; per-model metrics, several models in one run | Holds: `--model` is repeatable, and the report puts models side by side (`evaluator.py:382-386`, `:578-610`) | +| FR-003 | Judge pinned and reported separately | Holds: `--judge-model`, default `gpt-4o`, named in the report; a judge that is also under test is refused (`evaluator.py:67`, `:433-437`, `:582-585`) | +| FR-004 | Report run-to-run variance | Holds as a spread: `--repeat N` reports max−min per metric (`evaluator.py:399-405`, `:592-605`) | +| FR-005 | Default selectable per deployment without a code change | Holds: `LLM_MODEL`, or `llm.model` in `config.yml` since spec 003 (`src/agent/graph.py:175-197`) | +| FR-006 | A model with an unknown temperature requirement fails, naming the variable | Not adopted. Spec 003's FR-007 (2026-09-10) reversed it: a model the table has not met is not validated and does not block startup (`graph.py:150-151`; `tests/agent/test_model_temperature.py:186-190`) | +| FR-007 | Record the chosen default and its numbers here | Not met: no decision or comparison is recorded | +| SC-001 | One command reports answer quality for the shipping pipeline, for a named model | `./bin/evaluate --model `, with FR-001's gap | +| SC-002 | Two runs on unchanged code differ by less than the reported noise | No recorded run | +| SC-003 | Default-model decision recorded with a per-metric comparison | Not met | +| SC-004 | Switching the default is one environment variable, no rebuild | Holds (`LLM_MODEL`) | +| SC-005 | Answer-quality measurement for the four retrieval changes of 2026-09-04 | No recorded measurement | + +### Key terms - **Model under test**: the model that answers, and that the graders use. - **Judge model**: the model ragas uses to score. Independent of the above, pinned. -- **Golden set**: `tests/golden/questions.txt`, 28 questions, already committed and - already used by `bin/retrieval_baseline`. - -## Success Criteria *(mandatory)* - -### Measurable Outcomes - -- **SC-001**: One command reports answer quality for the pipeline a user talks to, - for a named model, against the golden set. -- **SC-002**: Two consecutive runs of that command on unchanged code differ by less - than the threshold the command itself reports as noise. -- **SC-003**: The default-model decision is recorded with a per-metric comparison - covering both candidates. -- **SC-004**: Switching the default requires changing one environment variable, and - reverting it requires changing it back — no rebuild, no code edit. -- **SC-005**: The four retrieval changes from 2026-09-04 have an answer-quality - measurement attached, closing the gap left open by spec 001. +- **Golden set**: `tests/golden/questions.txt`, 20 questions (the file's other lines are comments), + used by `bin/retrieval_baseline` and, by default, by the evaluator. -## Assumptions +## Assumptions at the time (2026-09-09) - The golden set is representative enough to compare two models. It was assembled for retrieval measurement, not answer grading; if it proves too narrow that is a @@ -238,10 +170,12 @@ the same mistake the context budget made, fixed in Stage 3. - Cost is a real constraint but not the binding one at current volume; latency and answer quality are what the team will notice first. -## Decisions for the team +## Open questions put to the team (2026-09-09) -Not gaps in the specification — the P1 work proceeds however these are answered. -They are recorded so they are answered once, in the open. +These were proposals for a decision, not decisions. No answer to any of them is +recorded in this repository; that does not show whether they were discussed +elsewhere. Each "Recommendation" is the one made at the time. The evaluator work, +which none of them blocked, was done on 2026-09-09. ### D1 — Does the default flip before or after the quality evaluation? @@ -258,7 +192,7 @@ grounding difference matters more than the latency; C is not, on this evidence. ### D2 — Is ~2x latency acceptable on the chat surface? 22.5s → 41.2s per question, measured to the complete answer. Chainlit streams, so -what a user feels is time-to-first-token, which has not been measured. If the answer +what a user feels is time-to-first-token, which was not measured for luna. If the answer is "no", that alone settles the default for chat and D1 becomes moot for that surface — though luna may still suit analysis summarisation, where nobody is watching a cursor. @@ -270,9 +204,9 @@ this could not be checked here. If luna is materially more expensive per token, larger answers it produces multiply that, and cost becomes a first-order input rather than the third one. -## Out of Scope +## Out of scope for this work -- The LangChain upgrade. Unblocked by spec 001, unrelated to this. +- The LangChain upgrade (done separately the same day, 4e607c8). - Fine-tuning, or any change to the prompts. Changing prompt and model together would make the comparison unattributable — the same reason spec 001 was staged. - Streaming behaviour and time-to-first-token, beyond noting above that it is the diff --git a/specs/003-model-configuration/checklists/requirements.md b/specs/003-model-configuration/checklists/requirements.md deleted file mode 100644 index c3574bd7..00000000 --- a/specs/003-model-configuration/checklists/requirements.md +++ /dev/null @@ -1,53 +0,0 @@ -# Specification Quality Checklist: Model Configuration - -**Purpose**: Validate specification completeness and quality before proceeding to planning -**Created**: 2026-09-09 -**Feature**: [spec.md](../spec.md) - -## Content Quality - -- [x] No implementation details (languages, frameworks, APIs) -- [x] Focused on user value and business needs -- [x] Written for non-technical stakeholders -- [x] All mandatory sections completed - -## Requirement Completeness - -- [x] No [NEEDS CLARIFICATION] markers remain -- [x] Requirements are testable and unambiguous -- [x] Success criteria are measurable -- [x] Success criteria are technology-agnostic (no implementation details) -- [x] All acceptance scenarios are defined -- [x] Edge cases are identified -- [x] Scope is clearly bounded -- [x] Dependencies and assumptions identified - -## Feature Readiness - -- [x] All functional requirements have clear acceptance criteria -- [x] User scenarios cover primary flows -- [x] Feature meets measurable outcomes defined in Success Criteria -- [x] No implementation details leak into specification - -## Notes - -Deviations from the template, each deliberate: - -1. **Named files, functions and model ids appear throughout.** `resolve_embedding_model`, - `FIXED_TEMPERATURE_MODELS`, `gpt-5.6-luna`. The central finding — that two - contributed PRs would silently break Plant Reactome retrieval — cannot be stated - without naming the function they bypass. Writing around it would leave the three - people who must act on this unable to. - -2. **A defect review of two open PRs sits in the spec body.** They are the starting - material for the work, and the reason to read them is inseparable from the - reason not to merge them as they stand. - -3. **FR-004 and FR-005 are prohibitions, not capabilities.** Both say what must - *not* be configurable. They are requirements because the two obvious - implementations — and both contributed PRs — get them wrong, so leaving them - unstated would mean they were not decided. - -4. **One decision (D1) is left to the team.** Not a gap: the P1 work proceeds under - the recommendation either way, and only the strength of the startup check - changes. diff --git a/specs/003-model-configuration/plan.md b/specs/003-model-configuration/plan.md deleted file mode 100644 index 2c0482ca..00000000 --- a/specs/003-model-configuration/plan.md +++ /dev/null @@ -1,176 +0,0 @@ -# Implementation Plan: Model Configuration - -**Branch**: `feat/model-configuration` | **Date**: 2026-09-10 | **Spec**: [spec.md](./spec.md) - -**Input**: Feature specification from `/specs/003-model-configuration/spec.md` - -## Summary - -Move the answering model into `config.yml`, where a deployment already declares its -profiles, quotas, features and messages. Keep the model's *call requirements* — -temperature, structured-output method — derived in code, and refuse at startup a -configuration that contradicts them. - -Harvests the LLM half of #112 (@AaryanCode69) and #151 (@bhavyakeerthi3), and -rejects the embedding half of both. - -## Technical Context - -**Language/Version**: Python 3.12 - -**Primary Dependencies**: pydantic v2 (`Config` is already a `BaseModel`), -`langchain-openai` 1.6 via `agent.models.get_llm`. - -**Storage**: `config.yml` at the repo root, bind-mounted read-only into the -container; `config_default.yml` shipped in the image as the fallback. - -**Testing**: pytest. `tests/util/test_config.py` already pins the loader's failure -behaviour, including that an invalid file is fatal rather than silently defaulted. -`tests/agent/test_model_temperature.py` pins the derived table. - -**Target Platform**: Linux container, and developer machines. - -**Project Type**: Library within a single application repository. - -**Performance Goals**: None. This is a startup-time concern; it runs once. - -**Constraints**: A deployment with no model configured must behave exactly as it -does today (FR-002). `resolve_embedding_model()` must remain the only source of the -embedding model (FR-004). - -**Scale/Scope**: One new config section, one validation, four call sites at most. - -## Constitution Check - -*GATE: must pass before implementation, re-checked after.* - -| Article | How this plan satisfies it | -|---|---| -| I — verify the user path | The exit criterion is the server starting with a model set in `config.yml` and answering a question with it — not a unit test asserting the field parses. | -| II — measure, don't argue | Nothing here changes retrieval, so no baseline is owed. The model *choice* is spec 002's measurement, and this plan deliberately does not make it. | -| III — characterization tests pin behaviour | `test_config.py`'s existing tests must pass untouched: adding a section must not change what an invalid config does. | -| IV — fail loudly | The whole point of Stage 2. A contradictory model/temperature pair stops startup; it does not get silently corrected. | -| V — derive from the source of truth | FR-004 and FR-005 are this article restated: the embedding model comes from the bundle, the temperature from the measured table. Neither becomes configurable. | -| VI — bias to doing over filing | Two contributed PRs are harvested here rather than left open another six months. | - -**No violations.** The one judgement call is D1 (validate against the table, not the -API), taken as recommended in the spec: no network call at startup. - -## Project Structure - -### Documentation (this feature) - -```text -specs/003-model-configuration/ -├── spec.md # what and why, with D1 recommended -├── plan.md # this file -└── checklists/ - └── requirements.md -``` - -No `research.md`: there are no unresolved unknowns. The two contributed PRs are the -research, the temperature behaviour was measured empirically in #189, and D1 is -decided. No `data-model.md` or `contracts/` — the "model" is three optional fields -on an existing pydantic class, described below in full, and the only external -contract is `config.yml`, whose schema file is edited in Stage 1. - -### Source Code - -```text -src/util/config_yml/ -├── __init__.py # Config gains an optional `llm` section -└── models.py # new: LLMConfig, after #112's shape - -src/agent/ -├── graph.py # AgentGraph reads the config; resolve_temperature validates -└── models.py # unchanged -- get_llm already takes model and temperature - -.config.schema.yaml # the editor-facing schema gains the same section -config_default.yml # documents the section without setting it -``` - -## Implementation Stages - -### Stage 1 — The model becomes configurable, with today's behaviour as the default - -Add `LLMConfig` (`provider`, `model`, `base_url`) and an optional `llm` field on -`Config`. `AgentGraph` prefers it, falling back to `LLM_MODEL`, then to the current -default. - -Precedence is **environment over file** (FR-003), the opposite of Docker secrets in -`util/secrets.py`, and the difference is worth stating: a secret is mounted *by* the -deployment and should beat a checked-in file, whereas `LLM_MODEL` is how an operator -overrides a committed `config.yml` for one container. Both rules are "the more -specific thing wins"; they only look contradictory. - -Nothing is required. A `config.yml` with no `llm:` section behaves exactly as today, -which is what keeps every existing deployment working (FR-002). - -**Exit criteria**: `test_config.py` passes untouched; a `config.yml` naming -`gpt-5.6-luna` starts the server and answers a question with it; the effective model -appears in the startup log (FR-008). - -### Stage 2 — A contradictory configuration stops startup - -`resolve_temperature` already knows which models refuse `0.0`. If `config.yml` (or -`LLM_TEMPERATURE`) sets a temperature a model will reject, startup fails naming the -model, the value, and the fix. - -This is the reason the feature is a specification rather than a bump. Today that -mistake is a 400 on a user's first question — visible to a user, attributed to the -chatbot, and diagnosable only from logs. - -Per D1 the check is against the table alone. No API call, so it is instant, works -offline, and cannot make startup depend on OpenAI being reachable. A model the table -has not met does not block startup (FR-007). - -**Exit criteria**: `gpt-5.6-luna` with `temperature: 0` refuses to start and names -all three of model, value and fix; an unknown model starts normally; the existing -`LLM_TEMPERATURE` escape hatch still wins. - -### Stage 3 — Close #112 and #151 - -Both are superseded by Stages 1–2. Close them courteously, crediting each -contributor in the commit that lands their idea, and say plainly why the embedding -half was rejected — it bypasses `resolve_embedding_model()` and would silently break -Plant Reactome retrieval. - -**Exit criteria**: both PRs closed with credit; `.config.schema.yaml` and -`config_default.yml` document the new section; no configuration file can name an -embedding model (SC-004). - -## Complexity Tracking - -| Decision | Simpler alternative rejected | Why | -|---|---|---| -| A nested `llm:` section | A flat `llm: "openai/gpt-4o-mini"` string, as #151 does | Re-parsing a string to recover provider and model is work the loader can do once. `base_url` has nowhere to live in a flat string, and Plant Reactome needs it. | -| Validate at startup | Validate on first use | First use is a user's first question. The whole point is that the operator learns before a user does. | -| Table only, no probe | One API call at startup | D1. It catches the mistake actually being made, for free and offline. A mistyped model name fails immediately and unmistakably anyway. | -| Leave the embedding model alone | Make it configurable "for symmetry", as both PRs do | It is derived from the bundle that built the vectors. Configuring it is how you get silent nonsense instead of an error. | - -## Out of Scope - -- **Which model becomes the default.** [Spec 002](../002-default-llm-choice/spec.md), - gated on an answer-quality run. This plan makes the choice expressible, not made. -- Per-surface model selection (spec 003 User Story 3). Only chat exists today; the - nesting introduced in Stage 1 is what makes it cheap later. - - Confirmed nestable without a schema break (T020): a second surface adds a key - beside `llm:` holding the same `LLMConfig` shape -- - - ```yaml - llm: # the deployment default - model: gpt-4o-mini - surfaces: - analysis_summary: # slower is fine; nobody is watching a cursor - model: gpt-5.6-luna - ``` - - `LLMConfig` needs no change for that, and `resolve_llm_model` takes the config - object rather than reading globals, so a caller can pass a different one. - - Why surfaces will want to differ (T021): spec 002 measured 22.5s per question - for gpt-4o-mini against 41.2s for gpt-5.6-luna. A search-results panel and a - background summarisation have opposite tolerances for that, so forcing them to - agree is a choice with a real cost. -- The embedding model, permanently. diff --git a/specs/003-model-configuration/quickstart.md b/specs/003-model-configuration/quickstart.md index 46b8756f..8f30f574 100644 --- a/specs/003-model-configuration/quickstart.md +++ b/specs/003-model-configuration/quickstart.md @@ -1,7 +1,9 @@ # Quickstart: verifying model configuration -How to prove this feature works, end to end, the way a deployment uses it. Every -step here is a thing that can be observed failing, not a unit test. +Operator checks for how the answering model is configured, end to end, the way a +deployment uses it. Written for the 2026-09-10 implementation and re-checked against +the code on 2026-10-21. Every step is a thing that can be observed failing, not a +unit test. Requirement IDs refer to [spec.md](./spec.md). ## Prerequisites @@ -17,8 +19,9 @@ grep -c '^llm:' config.yml || echo "no llm section — good" poetry run chainlit run bin/chat-chainlit.py ``` -**Expect**: the server starts and the startup log names `gpt-4o-mini`. Ask a -question; it answers. +**Expect**: the server starts and the startup log reads `Answering with +openai/gpt-4o-mini at temperature 0.0` (`src/agent/graph.py:315-317`), unless +`LLM_MODEL` is set in the environment. Ask a question; it answers. ## 2. A model set in config.yml is the one that answers (FR-001, FR-008) @@ -29,8 +32,8 @@ llm: model: gpt-5.6-luna ``` -**Expect**: the startup log names `gpt-5.6-luna`, and `resolve_temperature` has -silently sent `1.0` — that model refuses `0.0` and nobody had to know. +**Expect**: the startup log names `gpt-5.6-luna` at temperature `1.0`: that model +refuses `0.0`, and `resolve_temperature` derives `1.0` without being told. ## 3. The environment overrides the file (FR-003) @@ -50,7 +53,9 @@ llm: ``` **Expect**: the server **refuses to start**, naming the model, the value, and the -fix. It must not start and fail later — that is the entire point of Stage 2. +fix (`src/agent/graph.py:160-166`). Starting and failing later is the bug this guards +against (spec User Story 2). `LLM_TEMPERATURE=0` is not checked this way: it is the +escape hatch and is sent as given. Compare with what happens without this feature: the server starts, and the first user to ask a question gets an error that reads as the chatbot being broken. @@ -72,7 +77,9 @@ empirical and always behind. It will fail on the first request with OpenAI's own grep -rn "embedding" .config.schema.yaml config_default.yml ``` -**Expect**: no `embedding` model field anywhere. The embedding model is read from +**Expect**: matches only in comments, which explain why there is no such field; no +`embedding` model field. (`tests/agent/test_model_configuration.py:93-109` makes the +same check, skipping comment lines.) The embedding model is read from the bundle path by `resolve_embedding_model()`, because a query embedded with a different model than built the vectors returns nonsense rather than an error. @@ -86,5 +93,7 @@ poetry run pytest tests/agent/test_embedding_model_resolution.py -q poetry run ruff check . && poetry run mypy . && poetry run pytest ``` -`tests/util/test_config.py` must pass **untouched**: adding a section must not -change what the loader does with an invalid file. +When this shipped, the existing tests in `tests/util/test_config.py` passed +unmodified (6f016dc), showing that adding the section did not change what the loader +does with an invalid file. The same PR then added tests for unknown top-level keys +(ce7cfef). diff --git a/specs/003-model-configuration/spec.md b/specs/003-model-configuration/spec.md index e4076676..e2c23a6b 100644 --- a/specs/003-model-configuration/spec.md +++ b/specs/003-model-configuration/spec.md @@ -4,9 +4,13 @@ **Created**: 2026-09-09 -**Status**: **Implemented** 2026-09-10 (#200 plan, #201 implementation). D1 taken as -recommended — validation against the table, no API call at startup. #112 and #151 -closed with credit. Outcome recorded at the end of this file. +**Status**: Historical record of the model-configuration work, 2026-09-09 to +2026-09-10. User Stories 1 and 2 were implemented 2026-09-10 (#200 plan, #201 +implementation). User Story 3, a model per surface, was not built. D1 was implemented +as the spec recommended (table only, no API call). The requirements below were set for +that work, not for later changes. Current behaviour is in `resolve_temperature` and +`resolve_llm_model` (`src/agent/graph.py:141-197`) and `src/util/config_yml/`. The +outcome is recorded at the end of this file. **Input**: Make it possible to switch between models — `gpt-4o-mini` and `gpt-5.6-luna` in particular — when per-model settings such as temperature differ. Does this need something to configure it with? @@ -37,21 +41,25 @@ it. An operator editing YAML has no better information and less context. So: **configure the choice, derive the consequences, and validate the pair before the server accepts traffic.** -## What already works +## What worked before this change (2026-09-09) -`LLM_MODEL` switches the answering model today, and `resolve_temperature` and the -`json_schema` grader method (#186, #189) make `gpt-5.6-luna` work through it. The -gap is not "can we switch" — it is: +`LLM_MODEL` switched the answering model, and `resolve_temperature` and the +`json_schema` grader method (#186, #189) made `gpt-5.6-luna` work through it. The gap +was not "can we switch" but: -1. the model is **one global value**, so every surface must use the same one -2. it lives in an environment variable rather than beside the other settings that - shape a deployment -3. nothing checks the model is usable **until a user asks a question** +1. the model was **one global value**, so every surface had to use the same one — + still true (see Outcome) +2. it lived in an environment variable rather than beside the other settings that + shape a deployment — closed by the `llm:` section of `config.yml` +3. nothing checked the model was usable **until a user asked a question** — closed + for a configured temperature the table knows the model refuses; still open, by + D1, for an unknown or mistyped model name -## Two contributed pull requests to harvest +## Two contributed pull requests (reviewed 2026-09-09) -Both from GSoC applicants who will not update them. Two independent people hitting -the same gap is strong evidence it is real. +Both came from GSoC applicants who were not expected to update them, and both were +closed when this work landed (11c3116). Two independent people hitting the same gap +was strong evidence it was real. | | #112 — @AaryanCode69 | #151 — @bhavyakeerthi3 | |---|---|---| @@ -85,135 +93,80 @@ exists as an override for the one case that needs it, and it is documented as "leave unset". This specification harvests the LLM half of both PRs and rejects the embedding half, with credit to both contributors. -## User Scenarios & Testing *(mandatory)* - -### User Story 1 — A deployment names its model beside its other settings (Priority: P1) - -An operator sets the answering model in `config.yml`, next to profiles, quotas and -features, rather than in a separate environment variable. - -**Why this priority**: It is the smallest thing that closes the gap, and both -contributed PRs already do most of it. - -**Independent Test**: Set the model in `config.yml`, start the server, ask a -question, and confirm from the response which model answered. - -**Acceptance Scenarios**: - -1. **Given** a `config.yml` naming a model, **When** the server starts, **Then** - that model answers. -2. **Given** no model in `config.yml`, **When** the server starts, **Then** the - current default answers and nothing breaks — every existing deployment keeps - working untouched. -3. **Given** a model in `config.yml` and `LLM_MODEL` set, **Then** the precedence - between them is defined and documented, not accidental. - ---- - -### User Story 2 — An unusable model stops the server, not the conversation (Priority: P1) - -An operator names a model that cannot work as configured. The server refuses to -start and says why. - -**Why this priority**: Equal to Story 1 and the reason this is a specification. The -failure today is a 400 on a user's first question — visible to a user, attributed -to the chatbot, and traceable only by reading logs. Both contributed PRs make this -*more* likely by adding two more ways to get the pair wrong. - -**Independent Test**: Configure `gpt-5.6-luna` with `temperature: 0`. The server -must refuse to start and name the conflict. - -**Acceptance Scenarios**: +## What the work set out to do (2026-09-09) + +- **User Story 1 (P1)** — a deployment names its answering model in `config.yml`, + beside profiles, quotas and features. With no model named, nothing changes, and the + precedence against `LLM_MODEL` is defined rather than accidental. +- **User Story 2 (P1)** — an unusable model stops the server, not the conversation. + A configured temperature that contradicts a known model requirement refuses to + start, naming the model, the value and the fix (independent test: `gpt-5.6-luna` + with `temperature: 0`). A model the table has not met still starts, and no API call + is made at startup (D1). +- **User Story 3 (P2)** — chat, chat-alongside-search-results and analysis + summarisation each name their own model, falling back to the deployment default. + The reason given: `gpt-4o-mini` answered in 22.5s and `gpt-5.6-luna` in 41.2s + (spec 002, 2026-09-09). Not built; see Outcome. + +US2 also said that for an unknown model "the first failure names `LLM_TEMPERATURE`". +That was not built. The first failure is the provider's own error on the first +request; only the startup refusal for a known model names `LLM_TEMPERATURE` +(`src/agent/graph.py:160-166`). + +### Edge cases considered at the time, and how they stand + +- **`config.yml` names a model the API does not have.** Still a 404 on the first + question, not at startup: D1 chose not to spend an API call to check. +- **`LLM_MODEL` and `config.yml` disagree.** The environment wins. The spec asked for + the losing value to be reported; it is not. `resolve_llm_model` returns `LLM_MODEL` + without logging the `config.yml` value it overrides (`src/agent/graph.py:197`), and + only the effective model is logged (`:315-317`). +- **A legal temperature the repository does not want** (`0.7` on `gpt-4o-mini` + makes the graders non-deterministic). Allowed; the effective temperature is in the + same startup log line. +- **Provider `ollama` with a local model.** The table matches exact model names only + (`src/agent/graph.py:114-138`), so a local model gets 0.0 unless its name is in the + set or a temperature is configured. + +## Requirements of this work, and where they stand (checked 2026-10-21) + +Code and tests cite these IDs. + +| ID | Required by this work | Today | +|---|---|---| +| FR-001 | Answering model settable in `config.yml` | Holds: optional `llm:` section, `LLMConfig` (`src/util/config_yml/models.py:18-45`, `src/util/config_yml/__init__.py:37`) | +| FR-002 | No model configured behaves as before | Holds: `llm` defaults to `None` and the built-in default is `gpt-4o-mini` (`src/agent/graph.py:172`; `tests/agent/test_model_configuration.py:27-38`) | +| FR-003 | Precedence between `config.yml` and `LLM_MODEL` defined, documented, tested | Holds: environment, then `config.yml`, then the default; `LLM_BASE_URL` works the same way (`graph.py:175-197`; `test_model_configuration.py:46-58`) | +| FR-004 | Embedding model not settable in `config.yml` | Holds: there is no such field, and `LLMConfig` and `Config` reject unknown keys (`models.py:27`, `__init__.py:30`), which `Config.from_yaml` turns into a refusal to start. The model comes from `resolve_embedding_model()`; `EMBEDDING_MODEL` remains an environment override | +| FR-005 | Per-model call requirements derived in code | Holds: temperature from `FIXED_TEMPERATURE_MODELS` (`graph.py:114-128`); the graders fix `json_schema` | +| FR-006 | A configured temperature that contradicts a known requirement stops startup, naming model, value and fix | Holds for `llm.temperature` (`graph.py:160-166`; `tests/agent/test_model_temperature.py:162-174`). `LLM_TEMPERATURE` is deliberately not checked: it is the escape hatch and wins over the table (`graph.py:153-158`; `test_model_temperature.py:119-124`) | +| FR-007 | A model unknown to the table does not block startup | Holds (`graph.py:150-151`; `test_model_temperature.py:186-190`) | +| FR-008 | Effective model logged at startup | Holds, with the temperature (`graph.py:315-317`) | +| FR-009 | Each surface can name its own model | Not built. Every surface uses the one deployment model (see Outcome) | +| FR-010 | Configuration that cannot be honoured stops the process | Holds for an invalid `config.yml`, including unknown keys (`__init__.py:113-128`), a refused configured temperature, and an unparseable `LLM_TEMPERATURE` (`graph.py:155-158`) | +| SC-001 | Switching model needs one file edit and a restart | Holds (`config.yml`, or `LLM_MODEL`) | +| SC-002 | Existing `config.yml` files keep working | Checked when shipped: neither the deployment `config.yml` nor `config_default.yml` carried an unknown key (`__init__.py:24-29`); `tests/util/test_config.py:154-156` covers the shipped default | +| SC-003 | A contradictory model/temperature pair is refused at startup | Holds for `config.yml`; not for `LLM_TEMPERATURE` (see FR-006) | +| SC-004 | No configuration file can name an embedding model | Holds (`tests/agent/test_model_configuration.py:93-109`) | +| SC-005 | Two surfaces run different models in one process | Not built (FR-009) | +| SC-006 | #112 and #151 credited and closed | Done 2026-09-10 (11c3116) | + +### Key terms -1. **Given** a model whose temperature requirement is known, **When** configuration - contradicts it, **Then** startup fails naming the model, the value and the fix. -2. **Given** a model the table has not met, **When** the server starts, **Then** it - starts — an unknown model is not an error — and the first failure names - `LLM_TEMPERATURE`. -3. **Given** a valid configuration, **When** the server starts, **Then** no - additional API call is made on the happy path unless D1 says otherwise. +- **Deployment default model**: what answers when a surface names nothing. +- **Surface**: chat (Chainlit), search-page answers (spec 010) and analysis + summarisation (spec 011). Only chat existed when this was written. All three now + use the same model: the answer endpoint shares the chat's graph + (`src/agent/registry.py:1-16`), and analysis summaries resolve the model the same + way (`src/api/analysis_summary.py:333-353`). +- **Model requirements**: temperature and structured-output method, derived in code; + see `FIXED_TEMPERATURE_MODELS` and `bin/probe_model_temperature`. Temperature can + be overridden (`llm.temperature`, `LLM_TEMPERATURE`) for a model the table has not met. ---- +## Decision D1 (put to the team 2026-09-09) -### User Story 3 — Surfaces choose their own model (Priority: P2) - -Chat, chat-alongside-search-results and analysis summarisation each name a model -suited to their tolerance for latency. Measured today: `gpt-4o-mini` answers in -22.5s and `gpt-5.6-luna` in 41.2s, so a search-results panel and a background -summarisation should not be forced to agree. - -**Why this priority**: Only chat exists today, so this is not yet blocking. It is -P2 rather than P3 because the context budget made exactly this mistake — a global -constant that had to be unpicked in spec 001 Stage 3 — and the cost of designing -for it now is one nesting level. - -**Acceptance Scenarios**: - -1. **Given** two surfaces configured with different models, **When** both answer, - **Then** each uses its own, in one process. -2. **Given** a surface with no model of its own, **Then** it uses the deployment - default rather than failing. - -### Edge Cases - -- **`config.yml` names a model the API does not have.** A 404 on the first - question. Same class as the temperature mismatch and wanted at startup for the - same reason — but only D1 decides whether that costs a network call. -- **`LLM_MODEL` and `config.yml` disagree.** One must win, and the loser must be - reported, not silently discarded. Rate limiting was broken once by a config that - quietly failed open; the constitution's Article IV is that scar. -- **An operator sets a temperature the model accepts but the repository does not - want.** `0.7` on `gpt-4o-mini` is legal and makes the graders non-deterministic. - That is the operator's call, and it must be visible in the logs at startup. -- **Provider is `ollama` with a local model.** The temperature table is - OpenAI-derived and says nothing about it; local models must not be forced to 1.0. - -## Requirements *(mandatory)* - -### Functional Requirements - -- **FR-001**: The answering model MUST be settable in `config.yml`. -- **FR-002**: A deployment with no model configured MUST behave exactly as it does - today. -- **FR-003**: Precedence between `config.yml` and `LLM_MODEL` MUST be defined, - documented and tested. -- **FR-004**: The embedding model MUST NOT be settable in `config.yml`. It is - derived from the installed bundle by `resolve_embedding_model()`. -- **FR-005**: Per-model call requirements — temperature, structured-output method — - MUST remain derived in code, not configured. -- **FR-006**: A configured temperature that contradicts a known model requirement - MUST stop startup, naming the model, the value and the fix. -- **FR-007**: A model unknown to the temperature table MUST NOT block startup. -- **FR-008**: The effective model MUST be logged at startup, so which one answered - is answerable from the logs alone. -- **FR-009**: Each surface MUST be able to name its own model, falling back to the - deployment default. -- **FR-010**: Configuration that cannot be honoured MUST stop the process rather - than fall back to a default (Article IV, and `Config.from_yaml`'s existing - behaviour). - -### Key Entities - -- **Deployment default model**: what answers when a surface names nothing. -- **Surface**: chat, chat-alongside-search, analysis summarisation. Only the first - exists today. -- **Model requirements**: temperature and structured-output method. Derived, never - configured; see `FIXED_TEMPERATURE_MODELS` and `bin/probe_model_temperature`. - -## Success Criteria *(mandatory)* - -- **SC-001**: Switching the answering model requires editing one file and - restarting — no code change, no rebuild. -- **SC-002**: Every existing deployment keeps working with its `config.yml` - unchanged. -- **SC-003**: A contradictory model/temperature pair is refused at startup, and no - user ever sees the resulting error. -- **SC-004**: No configuration file can name an embedding model. -- **SC-005**: Two surfaces run different models in one process. -- **SC-006**: The credit for #112 and #151 appears in the commit that harvests - them, and both PRs are closed courteously. - -## Decisions for the team +Implemented as option A in #201. This repository does not record whether the team +chose it explicitly; the commit and the Outcome say it was "taken as recommended". ### D1 — Does startup validate against the API, or only against the table? @@ -228,7 +181,7 @@ plausible-but-wrong pairing, which the table catches for free. A mistyped model name is a different and more obvious failure — it fails immediately, for everyone, on the first request, and no one is confused about the cause. -## Assumptions +## Assumptions at the time (2026-09-09) - `config.yml` is the right home. It already carries profiles, quotas, features and messages; it has a schema, and a loader that refuses to start on an invalid file. @@ -238,11 +191,11 @@ on the first request, and no one is confused about the cause. - Provider stays OpenAI-compatible. `base_url` covers self-hosting, as Plant Reactome already does. -## Out of Scope +## Out of scope for this work -- **Which model becomes the default.** That is [spec 002](../002-default-llm-choice/spec.md), - and it is gated on an answer-quality measurement that does not run yet. This - specification makes the choice expressible; it does not make it. +- **Which model becomes the default.** Left to [spec 002](../002-default-llm-choice/spec.md), + which made it wait on an answer-quality measurement. No decision is recorded there. + This work made the choice expressible; it did not make it. - Per-request model selection by an end user. - The embedding model, permanently. FR-004. @@ -274,12 +227,44 @@ works today is refused. The pattern in both: the guard was placed on the thing being built rather than on the seam beside it. -### Still open +### Not built: a model per surface (User Story 3) + +When this shipped only chat existed. Search-page answers (spec 010) and analysis +summaries (spec 011) have been added since, and all three use the one deployment +model. There is no per-surface setting. Because `Config` rejects unknown keys +(`src/util/config_yml/__init__.py:30`), adding one needs a code change. Whether it is +still wanted is not recorded. + +The plan for this work checked (2026-09-10) that a per-surface section could reuse +`LLMConfig` unchanged, for example: -**User Story 3, per-surface model selection**, ships nothing. Only chat exists. The -nesting chosen in Stage 1 is what makes it cheap when a second surface appears — -which is the mistake spec 001 made with the context budget, made deliberately this -time rather than discovered. +```yaml +llm: # the deployment default + model: gpt-4o-mini +surfaces: # proposed, not implemented + analysis_summary: + model: gpt-5.6-luna +``` -Spec 002's decision — *whether* luna becomes the default — remains open. This -specification made the choice expressible, not made. +It also noted that `resolve_llm_model` takes the config object rather than reading +globals, so a caller can pass a different one. + +Spec 002 records no decision on whether luna becomes the default; the default is +still `gpt-4o-mini` (`src/agent/graph.py:172`). + +### Why the environment beats the file + +`LLM_MODEL` overrides `config.yml`, the reverse of `util/secrets.py`, where a mounted +Docker secret beats the environment. Both follow "the more specific source wins": a +secret is mounted by a deployment and should beat a committed file, while `LLM_MODEL` +is how an operator overrides a committed `config.yml` for one container +(`src/agent/graph.py:180-186`). + +### Alternatives rejected during this work + +| Choice | Simpler alternative rejected | Why | +|---|---|---| +| A nested `llm:` section | A flat `llm: "openai/gpt-4o-mini"` string, as #151 did | `base_url` has nowhere to live in a flat string, and Plant Reactome needs it. | +| Validate at startup | Validate on first use | First use is a user's first question. | +| Table only, no probe | One API call at startup | D1. | +| Leave the embedding model alone | Make it configurable, as both PRs did | It is derived from the bundle that built the vectors; configuring it gives silent nonsense instead of an error. | diff --git a/specs/003-model-configuration/tasks.md b/specs/003-model-configuration/tasks.md deleted file mode 100644 index 4ec42884..00000000 --- a/specs/003-model-configuration/tasks.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -description: "Task list for model configuration" ---- - -# Tasks: Model Configuration - -**Input**: Design documents from `/specs/003-model-configuration/` - -**Prerequisites**: [plan.md](./plan.md), [spec.md](./spec.md), [quickstart.md](./quickstart.md) - -**Tests**: Included. This repository's constitution (Article III) makes tests the -tripwire for behaviour, and Article IV's "fail loudly" requirement is only real if a -test asserts the failure. They are written with the code they cover, not after. - -**Branch**: `feat/model-configuration` - -## Phase 1: Setup - -- [x] T001 Create branch `feat/model-configuration` from `origin/main` -- [x] T002 Re-read #112 and #151 with `gh pr diff`, to credit them accurately in the commits that land their idea - -## Phase 2: Foundational - -**Blocking: every user story below depends on the config field existing.** - -- [x] T003 Create `LLMConfig` (`provider: str = "openai"`, `model: str | None = None`, `base_url: str | None = None`, `temperature: float | None = None`) in `src/util/config_yml/models.py`, after the shape in #112 and crediting @AaryanCode69 -- [x] T004 Add `llm: LLMConfig | None = None` to `Config` in `src/util/config_yml/__init__.py` — optional, so a config without it is unchanged (FR-002) -- [x] T005 [P] Add the matching `llm` object to `.config.schema.yaml`, with **no** embedding field (FR-004) -- [x] T006 [P] Document the section, commented out, in `config_default.yml` - -## Phase 3: User Story 1 — A deployment names its model beside its other settings (P1) - -**Goal**: the model comes from `config.yml`. - -**Independent test**: set a model in `config.yml`, start the server, ask a question, -confirm from the log which model answered. Quickstart steps 1–3. - -- [x] T007 [US1] Add `resolve_llm_model(config)` to `src/agent/graph.py`: `LLM_MODEL` beats `config.llm.model` beats the current default, and document why the precedence is the reverse of `util/secrets.py` (both are "the more specific wins") -- [x] T008 [US1] Wire `AgentGraph.__init__` to it, passing `base_url` and `provider` from the config when present, in `src/agent/graph.py` -- [x] T009 [US1] Log the effective model at startup in `src/agent/graph.py` (FR-008) — the name only, never a key -- [x] T010 [P] [US1] Test in `tests/agent/test_model_configuration.py`: no `llm` section behaves exactly as today (FR-002) -- [x] T011 [P] [US1] Test in `tests/agent/test_model_configuration.py`: a configured model is the one selected -- [x] T012 [P] [US1] Test in `tests/agent/test_model_configuration.py`: `LLM_MODEL` overrides `config.yml` (FR-003) -- [x] T013 [US1] Run quickstart steps 1–3 against a real bundle and confirm the log names the expected model each time (constitution Article I) - -## Phase 4: User Story 2 — An unusable model stops the server, not the conversation (P1) - -**Goal**: a contradictory model/temperature pair is refused at startup. - -**Independent test**: `gpt-5.6-luna` with `temperature: 0` must refuse to start. -Quickstart steps 4–5. - -- [x] T014 [US2] Extend `resolve_temperature` in `src/agent/graph.py` to accept a configured temperature and raise `SystemExit` naming model, value and fix when the model refuses it (FR-006) -- [x] T015 [P] [US2] Test in `tests/agent/test_model_temperature.py`: luna + `temperature: 0` exits, and the message contains all three of model, value and remedy -- [x] T016 [P] [US2] Test in `tests/agent/test_model_temperature.py`: a model absent from the table starts normally (FR-007) -- [x] T017 [P] [US2] Test in `tests/agent/test_model_temperature.py`: `LLM_TEMPERATURE` still wins over the configured value -- [x] T018 [US2] Perturbation check: delete the guard and confirm T015 fails — a test that cannot fail is not a tripwire (Article III) -- [x] T019 [US2] Run quickstart steps 4–5 and confirm the server refuses to start rather than failing on the first question - -## Phase 5: User Story 3 — Surfaces choose their own model (P2) - -**Goal**: not built now; make sure Stage 1's shape does not preclude it. - -**Independent test**: none — this phase ships no behaviour. - -- [x] T020 [US3] Confirm `LLMConfig` is nestable per surface without a schema break, and record in `specs/003-model-configuration/plan.md` what a second surface would add -- [x] T021 [US3] Cross-reference spec 002's latency table (22.5s vs 41.2s per question) in the spec as the reason surfaces will want to differ - -## Phase 6: Polish & Cross-Cutting - -- [x] T022 [P] Verify `grep -rn embedding .config.schema.yaml config_default.yml` finds no embedding model field (SC-004), and add a test asserting it -- [x] T023 [P] Confirm `tests/util/test_config.py` passes **untouched** — adding a section must not change what an invalid config does (Article III) -- [x] T024 Run `ruff check`, `ruff format --check`, `mypy`, `pytest` -- [x] T025 Close #112 with credit to @AaryanCode69, stating plainly that the LLM half is harvested and the embedding half rejected because it bypasses `resolve_embedding_model()` and would silently break Plant Reactome -- [x] T026 Close #151 with credit to @bhavyakeerthi3, noting the flat-string shape was reasonable but `base_url` has nowhere to live in it -- [x] T027 Update `specs/003-model-configuration/spec.md` with the outcome, and record D1 as taken-as-recommended - -## Dependencies - -```text -Phase 1 (T001-T002) - | -Phase 2 (T003-T006) <- blocking; nothing below works without the field - | - +-- Phase 3 US1 (T007-T013) P1 - | | - +-- Phase 4 US2 (T014-T019) P1, needs US1's resolution to validate - | - +-- Phase 5 US3 (T020-T021) P2, documentation only - | -Phase 6 (T022-T027) -``` - -US2 depends on US1: there is nothing to validate until the model is configurable. -US3 depends on neither and ships no code. - -## Parallel Opportunities - -- **T005, T006** — different files, no shared state -- **T010, T011, T012** — three tests in one new file; write together, they share fixtures -- **T015, T016, T017** — same, in the existing temperature test file -- **T022, T023** — independent checks - -`T025` and `T026` are deliberately **not** parallel with the rest: close a -contributor's PR only once the thing that replaces it has actually landed and its -gates are green. - -## Implementation Strategy - -**MVP is Phase 2 + Phase 3 (US1).** That alone delivers FR-001 through FR-003 and -FR-008 — a deployment can name its model — and is independently shippable. - -**Phase 4 (US2) is the reason this is a specification.** It is P1 alongside US1 -rather than after it, because shipping US1 alone adds two new ways to configure a -model wrongly without adding any way to find out before a user does. - -Phase 5 ships nothing and could be dropped without loss; it exists so the nesting -decision in T003 is made deliberately rather than discovered later, which is the -mistake the context budget made in spec 001. diff --git a/specs/004-answer-in-user-language/checklists/requirements.md b/specs/004-answer-in-user-language/checklists/requirements.md deleted file mode 100644 index 42420b1d..00000000 --- a/specs/004-answer-in-user-language/checklists/requirements.md +++ /dev/null @@ -1,64 +0,0 @@ -# Specification Quality Checklist: Answer in the User's Language - -**Purpose**: Validate specification completeness and quality before proceeding to planning -**Created**: 2026-09-10 -**Feature**: [spec.md](../spec.md) - -## Content Quality - -- [x] No implementation details (languages, frameworks, APIs) -- [x] Focused on user value and business needs -- [x] Written for non-technical stakeholders -- [x] All mandatory sections completed - -## Requirement Completeness - -- [x] No [NEEDS CLARIFICATION] markers remain -- [x] Requirements are testable and unambiguous -- [x] Success criteria are measurable -- [x] Success criteria are technology-agnostic -- [x] All acceptance scenarios are defined -- [x] Edge cases are identified -- [x] Scope is clearly bounded -- [x] Dependencies and assumptions identified - -## Feature Readiness - -- [x] All functional requirements have clear acceptance criteria -- [x] User scenarios cover primary flows -- [x] Feature meets measurable outcomes defined in Success Criteria -- [x] No implementation details leak into specification - -## Adversarial review of this specification - -Run before trusting it. Two load-bearing claims were checked against the corpus -rather than reasoned about, and **one of them was wrong**. - -| claim as first written | verdict | -|---|---| -| "BM25, which is lexical, retrieves close to nothing for a French query" | **wrong.** It returns a full ten documents; named entities survive translation. The real cost is that 7 of 10 differ. Corrected, with the measurement. | -| "#140's appended instruction pollutes retrieval" | **right, but first measured wrongly.** BM25 alone gives 0/10 — dramatic and irrelevant, because the query expander sits in front of it and four of five queries reach BM25 clean. Through the real retriever: **20/40 and 21/40 survive**, so about half the context changes. Corrected after Adam pushed back on the number looking too low. | -| "#140 targets React-to-Me" | right, but it edits `call_model`, which main renamed to `generate_answer`. Added. | -| "Plant Reactome's answer does not receive the language" | verified in `plantreactome.py`. | -| "Cross-Database's summary does" | verified in `cross_database.py`. | -| "#125 keeps the translate-to-English step" | verified in its diff. The obvious mistake, not made. | - -The wrong claim mattered: it would have justified FR-004 with a reason a reader could -disprove in five minutes, which is worse than justifying it with the smaller, true -one. - -And the review itself needed a second pass. It checked whether each claim was true -but not whether the *test* measured the product — so it confirmed a component -measurement and reported it as a pipeline one. That is the same failure -`evaluator.py` had, four days apart. **An adversarial review has to attack the -measurement as well as the claim.** - -## Notes - -Deviations from the template, deliberate: - -1. **Two contributed PRs are reviewed in the spec body.** They are the starting - material and the reason to reject each is inseparable from the reason to read it. -2. **Measurements appear in a specification.** Constitution Article II says retrieval - claims come with numbers. The numbers are what make the choice between the two - approaches obvious rather than aesthetic. diff --git a/specs/004-answer-in-user-language/plan.md b/specs/004-answer-in-user-language/plan.md deleted file mode 100644 index 93c67cbf..00000000 --- a/specs/004-answer-in-user-language/plan.md +++ /dev/null @@ -1,99 +0,0 @@ -# Implementation Plan: Answer in the User's Language - -**Branch**: `feat/answer-in-user-language` | **Date**: 2026-09-10 | **Spec**: [spec.md](./spec.md) - -## Summary - -Pass `detected_language` to the answer prompt as its own variable, in the two -profiles that ignore it. Nothing else moves. - -The constraint that decides the design: **`input` must stay exactly as it is**, -because `create_retrieval_chain` hands it to the retriever, and `HybridRetriever` -hands it to the query expander. Both must keep seeing the English rephrasing and -nothing else. That is what rules out #140's mechanism and what makes this change -provably free of retrieval risk — the retrieval query is byte-identical before and -after, which a test can assert directly rather than measure statistically. - -## Technical Context - -**Language/Version**: Python 3.12 - -**Primary Dependencies**: `langchain-core` prompts; no new dependency. - -**Testing**: pytest. `bin/retrieval_baseline` is *not* needed — see below. - -**Constraints**: English questions must take exactly today's path (FR-007), with no -added prompt content and no extra model call. - -**Scale/Scope**: Two prompt templates, two call sites, one shared instruction. - -## Constitution Check - -| Article | How this plan satisfies it | -|---|---| -| I — verify the user path | The exit criterion is asking the assembled chain a French question and reading the answer, which is how the bug was found. | -| II — measure, don't argue | Retrieval is unchanged **by construction**: `input` is not touched. A test asserts the retrieval query is byte-identical, which is stronger than a baseline diff and does not spend an hour of API time. | -| III — characterization tests pin behaviour | A test pins that an English question produces no language instruction at all. | -| IV — fail loudly | Nothing to fail: an absent language falls back to today's behaviour, which is correct English output. | -| V — derive from the source of truth | The language comes from `BaseState`, where the detector already put it. Nothing re-detects. | -| VI — bias to doing over filing | Two contributed PRs are resolved rather than left open. | - -**No violations.** - -## Implementation Stages - -### Stage 1 — React-to-Me answers in the detected language - -Add a language instruction to the reactome answer prompt as a template variable, and -pass `state["detected_language"]` at the call site in `generate_answer`. - -The instruction carries #140's nomenclature rule, which is the part of that PR worth -keeping: gene symbols, protein names, pathway names, `R-HSA-*` identifiers and URLs -stay in English. - -**What an English question pays.** No extra model call, and a byte-identical -retrieval query — but it does gain the instruction in its answer prompt, saying to -answer in English. - -An earlier draft of this plan claimed English "costs nothing", which contradicted the -spec's own FR-007 and was simply untrue: a sentence added to every prompt is a -change, however small. FR-007 has been corrected to say what is actually guaranteed. -The cost is one sentence against roughly 3,200 tokens of retrieved context; the -alternative is a branch whose common path only non-English users exercise. - -**Exit criteria**: a French question is answered in French with nomenclature intact; -the retrieval query is byte-identical to today's; an English question still gets a -well-formed English answer. - -### Stage 2 — Plant Reactome, identically - -The same change to `plantreactome/prompt.py` and its call site. It is a separate -stage only because it is a separate deployment and can be verified separately. - -**Exit criteria**: same three, against the plantreactome profile. - -### Stage 3 — Close #125 and #140 - -Close both with credit, saying specifically what each contributed: #140's target and -nomenclature rule, #125's mechanism. Say plainly why #140's mechanism was not taken, -with the measured number — half the retrieved context changes. - -#125's hallucination-grading and web-search work is **not** resolved by this and must -not be described as such; it belongs to #123. - -**Exit criteria**: both closed with credit; the spec records the outcome. - -## Complexity Tracking - -| Decision | Simpler alternative rejected | Why | -|---|---|---| -| A prompt variable | Append to `input`, as #140 does | Measured: about half the fused documents change. The instruction reaches the retriever and the query expander, neither of which should see it. | -| Instruct the generator | Translate the finished answer | D1. An extra model call and its latency on every non-English message, plus translation errors over a scientific answer, to gain a nomenclature protection the prompt already achieves. | -| Always pass the language | Branch on "is it English" | A branch means two paths, one of which is rarely exercised. Passing "English" is the same code doing the same thing. | -| No baseline run | `bin/retrieval_baseline` before and after | The retrieval query is unchanged by construction; a test asserting that is exact, where a baseline diff would only show noise and cost an hour. | - -## Out of Scope - -- Verifying the answer really is in the requested language. -- The corpus, embeddings, or the detector. -- #125's hallucination grader and web search — #123's decision. diff --git a/specs/004-answer-in-user-language/spec.md b/specs/004-answer-in-user-language/spec.md index 664d8965..b3b351e9 100644 --- a/specs/004-answer-in-user-language/spec.md +++ b/specs/004-answer-in-user-language/spec.md @@ -4,18 +4,26 @@ **Created**: 2026-09-10 -**Status**: **Implemented** 2026-09-10 (#204 spec, #205 implementation). D1 taken as -recommended — instruct the answer prompt, no translation pass. #140 and #125 closed -with credit. Outcome at the end of this file. +**Status**: Historical record of the answer-language work, 2026-09-10. Implemented +2026-09-10 (#204 spec, #205 implementation) for React-to-Me and Plant Reactome, with +D1 implemented as recommended: the answer prompt is instructed and there is no +translation pass. The requirements below were set for that work, not for later +changes. Current behaviour is in `src/agent/tasks/language_instruction.py`, the answer +prompts that include it (`src/retrievers/{reactome,plantreactome,userguide}/prompt.py`) +and the call sites in `src/agent/profiles/`. The outcome is at the end of this file. **Input**: Issue #104, "RAG only responds in English". Two competing pull requests, #125 and #140. -## The gap, precisely +## The gap before this work (2026-09-10) -Language detection already works. `detect_language` runs on every message, concurrently -with the safety check, and `detected_language` is on `BaseState` for every profile. +Language detection already worked. `detect_language` ran on every message, +concurrently with the safety check, and `detected_language` was on `BaseState` for +every profile. (React-to-Me has since moved detection to run alongside the rephrase +instead, 9677ecb, 2026-09-18: `src/agent/profiles/react_to_me.py:155-164`. The shared +`preprocess` used by Plant Reactome and Cross-Database still pairs it with the safety +check: `src/agent/profiles/base.py:69-72`.) -It reaches some places and not others: +It reached some places and not others: | where | uses the detected language? | |---|---| @@ -24,10 +32,11 @@ It reaches some places and not others: | **React-to-Me's answer** | **no** | | **Plant Reactome's answer** | **no** | -So a question in French is detected as French, rephrased into English for retrieval, +So a question in French was detected as French, rephrased into English for retrieval, answered in English — and if it had instead been *refused*, the refusal would have -come back in French. The two profiles that serve real users are the two that answer -in the wrong language; the one that gets it right is the prototype. +come back in French. The two profiles that serve real users were the two that answered +in the wrong language; the one that got it right was the prototype. Since this work +they answer in the detected language (see the requirements table). That asymmetry is the whole feature. Nothing needs detecting, routing or storing that is not already detected, routed and stored. @@ -54,7 +63,7 @@ tell what language was used — nothing asks it to reply in that language. **Retrieval must stay in English.** The Reactome corpus is English: the CSV rows BM25 tokenises, and the vectors built from them. -Measured on the `summations` collection rather than assumed, because the obvious +Measured on the `summations` collection (2026-09-10, Release95) rather than assumed, because the obvious version of this claim is wrong. BM25 does **not** collapse on a non-English query — it still returns a full ten documents, because named entities survive translation: @@ -95,7 +104,7 @@ will be used to pass into the retriever."* So that 61-word block of English pros about response languages becomes part of the BM25 query and the embedded vector, for every non-English question. -Measured through the **whole retriever**, appending it to an English question: +Measured through the **whole retriever** (2026-09-10, Release95, four collections), appending it to an English question: | query | fused documents surviving the appended instruction | |---|---| @@ -151,126 +160,64 @@ taking a second feature that needs its own judgement. Neither is mergeable as it stands. Between them they contain the whole answer: **#140's target and its nomenclature rule, #125's mechanism.** -## User Scenarios & Testing *(mandatory)* - -### User Story 1 — A question in French is answered in French (Priority: P1) - -A researcher asks React-to-Me a question in French. Retrieval happens in English, and -the answer comes back in French, with gene names and Reactome links untouched. - -**Why this priority**: It is the issue. Everything else here is a refinement of it. - -**Independent Test**: Ask the same question in English and in French. The retrieved -documents should be substantially the same; the answers should differ only in -language. - -**Acceptance Scenarios**: - -1. **Given** a question in French, **When** it is answered, **Then** the answer is in - French and retrieval used the English rephrasing, so the documents are the set the - English rendering retrieves. -2. **Given** a question in English, **When** it is answered, **Then** nothing about - the behaviour differs from today — no extra instruction, no extra call. -3. **Given** an answer in any language, **When** it names a gene, protein, pathway or - `R-HSA-*` identifier, **Then** that term appears in its original English form. -4. **Given** an answer in any language, **When** it cites Reactome, **Then** the URL - is unchanged and still resolves. - ---- - -### User Story 2 — Retrieval quality does not move (Priority: P1) - -A developer changes the language handling and can show that what reaches the model is -unchanged for English questions. - -**Why this priority**: Equal to Story 1, because the plausible way to implement Story -1 is the way #140 does it, and that quietly degrades retrieval for exactly the users -the feature is meant to help. A feature that answers in French by retrieving worse -has not helped anyone. - -**Independent Test**: `bin/retrieval_baseline` before and after. English questions -must be unchanged beyond the known ANN noise floor. - -**Acceptance Scenarios**: - -1. **Given** the change, **When** the baseline is compared, **Then** English - retrieval differs only within the documented run-to-run variance. -2. **Given** a non-English question, **When** retrieval runs, **Then** the text sent - to BM25 and the vector store contains no instruction prose — only the rephrased - question. - ---- - -### User Story 3 — Plant Reactome behaves the same way (Priority: P2) - -The Plant Reactome deployment answers in the user's language too. - -**Why this priority**: The same gap, the same fix, a second profile. P2 only because -the Reactome deployment is the larger audience. - -**Acceptance Scenarios**: - -1. **Given** a non-English question to Plant Reactome, **Then** the answer is in that - language, with nomenclature preserved. - -### Edge Cases - -- **The detector is wrong.** It returns a single label from one LLM call on the raw - input; a short question, or one mixing languages, can be misread. The cost is an - answer in the wrong language, which the user can see and rephrase — acceptable, but - it argues against building anything expensive on top of the label. -- **A language the model answers poorly.** The instruction is best-effort; there is - no verification step and this specification does not add one. -- **Nomenclature that is also an ordinary word.** Gene symbols like `SET`, `MAX` or - `CAT` are English words. A translator may render them as words rather than symbols. -- **A question already in English.** Must take exactly today's path, with no added - instruction — otherwise every existing user pays for a feature they do not use. - -## Requirements *(mandatory)* - -### Functional Requirements - -- **FR-001**: React-to-Me MUST answer in the language its question was asked in. -- **FR-002**: Plant Reactome MUST do the same. -- **FR-003**: The text used for retrieval MUST contain only the rephrased question — - no language instruction, no prompt scaffolding. -- **FR-004**: Retrieval MUST continue to happen in English. -- **FR-005**: Scientific nomenclature — gene symbols, protein names, pathway names, - `R-HSA-*` identifiers — MUST NOT be translated. -- **FR-006**: URLs and citation links MUST NOT be translated or altered. -- **FR-007**: An English question MUST cost no additional model call, and MUST reach - the retriever with a byte-identical query. - - It does gain a short instruction in the *answer* prompt, saying to answer in - English. That is a real change for every current user and is stated rather than - hidden. The alternative — branching on "is this English" — creates two paths where - the common one is untested by anyone who only ever asks in English, and the - instruction is a sentence against roughly 3,200 tokens of retrieved context. -- **FR-008**: The language MUST reach the answer prompt as its own input, not - concatenated into another field. - -### Key Entities - -- **Detected language**: one label per message, already on `BaseState`, produced - concurrently with the safety check. -- **Rephrased question**: the English rendering, already produced, already used for - retrieval. -- **Answer prompt**: per profile; the place the language belongs. - -## Success Criteria *(mandatory)* - -- **SC-001**: A question asked in French, German, Spanish or Japanese is answered in - that language by React-to-Me. -- **SC-002**: `bin/retrieval_baseline` shows English retrieval unchanged beyond the - documented noise floor. -- **SC-003**: For a non-English question, the retrieval query is byte-identical to - what the same question produces today. -- **SC-004**: In a non-English answer, every gene symbol, pathway name and `R-HSA-*` - identifier appears in English, and every Reactome URL resolves. -- **SC-005**: An English question costs the same number of model calls as today, and - its answer remains in English and well-formed. - -## Decisions for the team +## What the work set out to do (2026-09-10) + +- **User Story 1 (P1)** — a question to React-to-Me in French is answered in French; + retrieval still uses the English rephrasing; gene names, `R-HSA-*` identifiers and + Reactome links are left untouched. +- **User Story 2 (P1)** — retrieval does not move: the text sent to BM25 and the + vector store holds only the rephrased question. The planned test was a + `bin/retrieval_baseline` diff. It was replaced by a test that the retrieval query is + byte-identical across languages (`tests/agent/test_answer_language.py:69-79`), and + no baseline was run. +- **User Story 3 (P2)** — Plant Reactome does the same. + +### Edge cases noted at the time + +- **The detector is wrong.** One label from one LLM call on the raw input; a short or + mixed-language question can be misread, and the answer comes back in the wrong + language, which the user can see and rephrase. Unknown or ambiguous input is + labelled English (`src/agent/tasks/detect_language.py:12`). +- **A language the model answers poorly.** The instruction is best-effort; nothing + verifies the answer's language. +- **Nomenclature that is also an ordinary word** (`SET`, `MAX`, `CAT`). The + instruction tells the model not to translate symbols even when they are ordinary + words (`src/agent/tasks/language_instruction.py:24-26`). +- **A question already in English.** A first draft required English questions to take + exactly the old path, with no added instruction. That was dropped before + implementation: every question, English included, gets the instruction. An English + question costs no extra model call and reaches the retriever unchanged (FR-007). + +## Requirements of this work, and where they stand (checked 2026-10-21) + +Tests cite these IDs. + +| ID | Required by this work | Today | +|---|---|---| +| FR-001 | React-to-Me answers in the question's language | Holds for all three React-to-Me sources. The reactome and userguide answer prompts include `LANGUAGE_INSTRUCTION` (`src/retrievers/reactome/prompt.py:49`; `src/retrievers/userguide/prompt.py:50`, since f0e8581, 2026-10-04). Live answers carry their own "Answer in {language}" (`src/reactome_mcp/answer.py:81`, passed from `src/agent/profiles/react_to_me.py:252`) | +| FR-002 | Plant Reactome does the same | Holds (`src/retrievers/plantreactome/prompt.py:46`; `src/agent/profiles/plantreactome.py:87`) | +| FR-003 | Retrieval text holds only the rephrased question | Holds: the language travels in a separate `detected_language` key (`react_to_me.py:307-314`, `plantreactome.py:81-95`); `tests/agent/test_answer_language.py:60-66` | +| FR-004 | Retrieval stays in English | Holds: the rephrase step translates to English (`src/agent/tasks/rephrase.py:8`, `:16`) | +| FR-005 | Nomenclature not translated | An instruction to the model (`language_instruction.py:24-26`), not enforced. Checked on one French answer, 2026-09-10 | +| FR-006 | URLs and citation links not altered | Same: instructed, not enforced | +| FR-007 | An English question costs no extra model call and reaches the retriever with a byte-identical query | Holds. Byte identity is tested (`test_answer_language.py:69-79`); no extra call because there is no translation step. English questions do get the instruction ("Answer in English", about 60 words, `language_instruction.py:19-26`), stated rather than hidden. Branching on "is this English" was rejected because the common path would then go untested by anyone who only asks in English | +| FR-008 | Language reaches the answer prompt as its own input | Holds (`test_answer_language.py:82-89`) | +| SC-001 | French, German, Spanish or Japanese answered in that language | Verified for French only (2026-09-10). German, Spanish and Japanese are not recorded as checked | +| SC-002 | `bin/retrieval_baseline` shows English retrieval unchanged | Not run; replaced by the byte-identity test (User Story 2) | +| SC-003 | Non-English retrieval query byte-identical to before | Holds by the same test | +| SC-004 | Nomenclature in English, Reactome URLs resolve | Checked on one French answer, 2026-09-10 | +| SC-005 | English costs the same model calls and stays English | Calls: holds (FR-007). The English answer was checked once, 2026-09-10 | + +### Key terms + +- **Detected language**: one label per message, on `BaseState`. +- **Rephrased question**: the English rendering, used for retrieval. +- **Answer prompt**: per source; where the language instruction goes. + +## Decision D1 (put to the team 2026-09-10) + +Implemented as option A in #205. This repository does not record whether the team +chose it explicitly beyond "taken as recommended". ### D1 — Instruct the answer prompt, or translate afterwards? @@ -284,7 +231,7 @@ the Reactome deployment is the larger audience. extra, and B's advantage on nomenclature is reachable from A with prompt instructions — which is what both contributors did, independently. -## Assumptions +## Assumptions at the time (2026-09-10) - The existing detector is good enough. It is already trusted for refusals, which are the more sensitive case; nothing here makes it more load-bearing. @@ -292,7 +239,7 @@ which is what both contributors did, independently. English one. If that is wrong, the feature is wrong, not the implementation. - The corpus stays English. Translating Reactome content is a different project. -## Out of Scope +## Out of scope for this work - Translating the corpus, or multilingual embeddings. - Verifying that the answer really is in the requested language. @@ -326,15 +273,15 @@ differ. **"#140's appended instruction leaves 0 of 10 documents"** — measured on BM25 directly, which the pipeline never does: the query expander rewrites the question first, so four of five queries reach BM25 clean. Through the whole retriever it is -**20 of 40**. Half the context, not all of it. Adam pushed back that the number -looked too low and was right. +**20 of 40**. Half the context, not all of it. A review questioned the number as +looking too low, and was right. The second is the more useful lesson. The first review checked whether each claim was *true* without checking whether the *test measured the product* — the same failure `evaluator.py` had four days earlier. An adversarial review has to attack the measurement as well as the claim. -**And the plan contradicted the spec.** FR-007 promised English questions "no +**And the plan (removed; in git history at 75ca531) contradicted the spec.** FR-007 promised English questions "no additional prompt content" while the plan always passes the language, which adds a sentence to every English prompt. FR-007 was the wrong half and was corrected to promise what is actually true: no extra model call, byte-identical retrieval query, @@ -344,12 +291,22 @@ and one added sentence stated rather than hidden. The French answer carried 2 `R-HSA` citations against the English answer's 9. One sample, so not a finding — but if non-English answers systematically cite less, this -feature would be creating a quality gap while closing a language one. It is the kind -of thing the ragas run in [spec 002](../002-default-llm-choice/spec.md) should look -at once that harness is worth running. +feature would be creating a quality gap while closing a language one. Not followed up +in this repository. The evaluator (`./bin/evaluate`, measuring the shipping chain since +2026-09-09; see [spec 002](../002-default-llm-choice/spec.md)) could look at it, but its +default question set is English only (`tests/golden/questions.txt`). ### Not resolved #125 also carried a hallucination grader and web-search wiring for Cross-Database. Neither is touched here, and closing that PR is not a decision about them; they belong to #123. + +### Alternatives rejected during this work + +| Decision | Simpler alternative rejected | Why | +|---|---|---| +| A prompt variable | Append to `input`, as #140 did | Measured 2026-09-10: about half the fused documents change. The instruction would reach the retriever and the query expander. | +| Instruct the generator | Translate the finished answer | D1. An extra model call and its latency on every non-English message, plus translation errors over a scientific answer, for a nomenclature protection the prompt already gives. | +| Always pass the language | Branch on "is it English" | Two paths, one rarely exercised. Passing "English" is the same code doing the same thing. | +| No baseline run | `bin/retrieval_baseline` before and after | The retrieval query is unchanged by construction; a test asserting that is exact, where a baseline diff would only show noise. | diff --git a/specs/004-answer-in-user-language/tasks.md b/specs/004-answer-in-user-language/tasks.md deleted file mode 100644 index a5734a04..00000000 --- a/specs/004-answer-in-user-language/tasks.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -description: "Task list for answering in the user's language" ---- - -# Tasks: Answer in the User's Language - -**Input**: [plan.md](./plan.md), [spec.md](./spec.md) - -**Tests**: Included. FR-003 and FR-007 are both "nothing changed" claims, and an -unasserted claim of that kind is worth nothing. - -## Phase 1: Foundational - -- [x] T001 Add a shared language instruction constant carrying the nomenclature rule (gene symbols, protein names, pathway names, `R-HSA-*`, URLs stay English) in `src/agent/tasks/language_instruction.py`, crediting @bleedblack1 for the wording - -## Phase 2: User Story 1 — React-to-Me answers in the detected language (P1) - -**Independent test**: ask the assembled chain a French question; read the answer. - -- [x] T002 [US1] Add a `{detected_language}` variable and the instruction to `src/retrievers/reactome/prompt.py` -- [x] T003 [US1] Pass `state["detected_language"]` in `generate_answer` in `src/agent/profiles/react_to_me.py`, leaving `input` untouched -- [x] T004 [P] [US1] Test in `tests/agent/test_answer_language.py`: the retrieval query is byte-identical with and without a language (FR-003) -- [x] T005 [P] [US1] Test in `tests/agent/test_answer_language.py`: the prompt receives the language as its own variable, never concatenated into `input` (FR-008) -- [x] T006 [P] [US1] Test in `tests/agent/test_answer_language.py`: an English question adds no extra model call and reaches the retriever with a byte-identical query (FR-007) -- [x] T006a [US1] Ask the real chain an English question and confirm the answer is still English and well-formed — the instruction is new prompt content for every existing user (SC-005) -- [x] T007 [US1] Ask the real chain a French question against the Release95 bundle; confirm the answer is French and gene symbols, `R-HSA-*` IDs and URLs are unchanged (Article I) - -## Phase 3: User Story 3 — Plant Reactome, identically (P2) - -- [x] T008 [US3] Same change to `src/retrievers/plantreactome/prompt.py` and its call site in `src/agent/profiles/plantreactome.py` -- [x] T009 [P] [US3] Test that both profiles use the same shared instruction, so they cannot drift - -## Phase 4: Polish & Cross-Cutting - -- [x] T010 Perturbation check: revert the call site and confirm the French test fails -- [x] T011 Run `ruff check`, `ruff format --check`, `mypy`, `pytest` -- [x] T012 Close #140 with credit to @bleedblack1 — the target and the nomenclature rule are kept; the mechanism is not, with the measured number -- [x] T013 Close #125 with credit to @bhavyakeerthi3 for the mechanism, noting its hallucination-grading work belongs to #123 and is untouched -- [x] T014 Record the outcome in `specs/004-answer-in-user-language/spec.md` - -## Dependencies - -```text -T001 -> Phase 2 (T002-T007) -> Phase 3 (T008-T009) -> Phase 4 -``` - -Phase 3 depends on Phase 2 only for the shared constant's final shape. - -## Implementation Strategy - -**MVP is Phase 2.** React-to-Me is the deployment with users. Phase 3 is the same -change to a second profile and could ship separately. diff --git a/specs/005-mcp-live-data/checklists/requirements.md b/specs/005-mcp-live-data/checklists/requirements.md index b68c49ab..70325a9b 100644 --- a/specs/005-mcp-live-data/checklists/requirements.md +++ b/specs/005-mcp-live-data/checklists/requirements.md @@ -1,27 +1,12 @@ -# Specification Quality Checklist: Live Reactome Data and Analysis +# Claim Checks: Live Reactome Data and Analysis -**Purpose**: Validate specification completeness and quality before planning **Created**: 2026-09-10 **Feature**: [spec.md](../spec.md) -## Content Quality +Historical. How the specification's claims were checked when it was written. The +template's completion checkboxes are in git history up to 75ca531. -- [x] No implementation details beyond what the contributed PRs already fix -- [x] Focused on user value and business needs -- [x] Written for non-technical stakeholders -- [x] All mandatory sections completed - -## Requirement Completeness - -- [x] No [NEEDS CLARIFICATION] markers remain -- [x] Requirements are testable and unambiguous -- [x] Success criteria are measurable -- [x] All acceptance scenarios are defined -- [x] Edge cases are identified -- [x] Scope is clearly bounded -- [x] Dependencies and assumptions identified - -## Adversarial review of this specification +## Adversarial review of this specification (2026-09-10) Every load-bearing claim was checked rather than reasoned about, after spec 004 shipped a wrong one. @@ -30,7 +15,7 @@ shipped a wrong one. |---|---|---| | "the bundle is two releases behind" | `cat embeddings/current` → Release95; `reactome.org/ContentService/data/database/version` → 97 | **verified** | | "it was already stale when built" | bundle mtime 2026-09-02, eight days ago | **verified** — so this is not drift from age | -| "`reactome-mcp` is a prototype" | its own repository description: *"This is just a prototype for now"*, last push 2026-07-01 | **verified** | +| "`reactome-mcp` is a prototype" | its own repository description: *"This is just a prototype for now"*, last push 2026-07-01 | the description said so; **wrong about the code**, as spec 006 found on 2026-09-14 | | "#127 spawns a subprocess" | `asyncio.create_subprocess_exec` with stdio pipes, in its diff | **verified** | | "#142 duplicates the existing router" | it adds `create_query_router`; `create_intent_classifier` already exists and routes reactome/userguide | **verified** | @@ -41,17 +26,9 @@ rebuild needs no dependency, no subprocess and no router. Stating it that way is whole contribution of this document: the PRs' own justification is mostly a cheaper problem wearing the expensive problem's clothes. -### Not verified, and worth knowing - -Nobody has run `reactome-mcp` from this repository. Its 53 tools, their latency and -their behaviour under failure are taken from #127's description, not observed. Any -plan built on this should start by running it once. - -## Notes - -Deviations from the template, deliberate: +### Not verified at the time (2026-09-10) -1. **Three contributed PRs are reviewed in the spec body.** They are the starting - material and the reason to take or reject each is inseparable from reading them. -2. **The spec argues against most of its own feature's justification.** That is the - finding, not a digression. +Nobody had run `reactome-mcp` from this repository; its 53 tools, their latency and +their behaviour under failure were taken from #127's description. It was run from +here on 2026-09-14: `bin/mcp-probe` (dbd02cc) connected to reactome 1.4.0, which +exposed 53 tools (output in `src/reactome_mcp/README.md:34-43`). diff --git a/specs/005-mcp-live-data/spec.md b/specs/005-mcp-live-data/spec.md index b2eefba9..51bbb29d 100644 --- a/specs/005-mcp-live-data/spec.md +++ b/specs/005-mcp-live-data/spec.md @@ -4,9 +4,18 @@ **Created**: 2026-09-10 -**Status**: Draft. Three contributed PRs; two decisions (D1, D2) for the team. - -**Input**: #127, #137 and #142 from @GovindhKishore, integrating the `reactome-mcp` +**Status**: Historical record of the review of three contributed PRs, 2026-09-10 to +2026-09-14. Drafted as a proposal; this file records no approval of its requirements. +Afterwards, the client and tools from #127 and #137 were taken in, reworked (dbd02cc, +PR #212, 2026-09-14); #142 was not merged; a `live` destination was added to the +existing intent classifier (4c4e8a7, PR #216, 2026-09-14), off unless an MCP server is +configured. Enrichment analysis in the chat was later built without MCP. The +requirements and decisions below were set for that review, not for later changes; +their status is in "Requirements, and where they stand". Current behaviour is in +`src/reactome_mcp/session.py`, `src/agent/tasks/intent_classifier.py` and +`src/agent/profiles/react_to_me.py`. + +**Input**: #127, #137 and #142, contributed PRs integrating the `reactome-mcp` server so the chatbot can query live Reactome APIs and run enrichment analysis. ## Two problems, wrongly bundled as one @@ -15,9 +24,9 @@ The PRs are motivated by a single sentence: MCP tools *"query Reactome APIs dire meaning they always return current data regardless of when embeddings were built."* That is true, and it conflates two problems with very different costs. -### Problem 1 — the answers are two releases out of date +### Problem 1 — the answers were two releases out of date (measured 2026-09-10) -Measured today: +Measured 2026-09-10: | | | |---|---| @@ -25,27 +34,29 @@ Measured today: | bundle built | 2026-09-02, eight days ago | | `reactome.org/ContentService/data/database/version` | **97** | -The bundle is two releases behind, and it was already two behind on the day it was -built. So this is not drift from age — nothing rebuilt it against a current release. +The bundle was two releases behind, and it was already two behind on the day it was +built. So this was not drift from age — nothing had rebuilt it against a current release. -**This does not need MCP.** It needs the bundle rebuilt, and rebuilt on a schedule. -That is `bin/embeddings_manager` and a cron entry, against a capability the -repository already has. +**This did not need MCP.** It needed the bundle rebuilt, and rebuilt on a schedule: +`bin/embeddings_manager` and a cron entry, against a capability the repository +already had. What happened next is under D1. -### Problem 2 — the chatbot cannot analyse anything +### Problem 2 — the chatbot could not analyse anything (2026-09-10) `reactome-mcp` exposes enrichment analysis, pathway traversal and entity lookup. None of that is expressible as similarity search over stored text. No amount of rebuilding fixes it, because it is not a retrieval problem: the user gives a gene list and wants a computation. -**This is the part only MCP can do**, and it is the honest reason to take the work. +**At the time this looked like the part only MCP could do**, and it was the honest +reason to take the work. -Separating them matters because Problem 1 is most of the stated benefit and the -cheapest fix, while Problem 2 is the smaller-sounding benefit that actually requires -the architecture. +Separating them mattered because Problem 1 was most of the stated benefit and the +cheapest fix, while Problem 2 was the smaller-sounding benefit that seemed to require +the architecture. In the event, the chat's analysis was built against Reactome's +Analysis Service and ReactomeGSA directly, without MCP (see D2). -## What the three PRs do +## What the three PRs did (as submitted) | PR | adds | lines | |---|---|---| @@ -53,30 +64,42 @@ the architecture. | #137 | five MCP tools wrapped as LangChain `StructuredTool`s, wired to React-to-Me | +317/−20 | | #142 | an LLM router choosing between RAG, MCP search and MCP analysis | +450/−20 | -They build on each other in order and are the work of one contributor. Taken -together they are a coherent design, and the sequencing is right: client, then +They built on each other in order and were the work of one contributor. Taken +together they were a coherent design, and the sequencing was right: client, then tools, then routing. -## What taking them costs +Outcome: the client and tools from #127 and #137 were taken in, reworked, as +`src/reactome_mcp/` (dbd02cc, PR #212, 2026-09-14). #142 was not merged. + +## What taking them would cost (assessment, 2026-09-10) -**A prototype dependency.** `reactome/reactome-mcp` is Reactome's own repository, -which is the good case — but its description reads *"This is just a prototype for -now"* and it was last pushed 2026-07-01, over two months ago. The chatbot would take -a runtime dependency on it. +**A dependency described as a prototype.** `reactome/reactome-mcp` is Reactome's own +repository, which is the good case. Its repository description read *"This is just a +prototype for now"*, and it was last pushed 2026-07-01. Spec 006 checked the code on +2026-09-14 and found the description stale (see its "A correction to spec 005"). +The chatbot would take a runtime dependency on it. -**A subprocess in the container.** #127 uses `asyncio.create_subprocess_exec` with -stdio pipes, so the image must carry the MCP server and its runtime, and each chat -process spawns and supervises a child. The container now runs as a non-root user -(#198), which that must work under. Nothing today spawns a process; this would be -the first. +**A subprocess in the container.** #127 used `asyncio.create_subprocess_exec` with +stdio pipes, so the image would have to carry the MCP server and its Node runtime, +and each chat process would spawn and supervise a child, under the non-root user the +container runs as since #198. Nothing spawned a process at the time. Outcome: the +stdio subprocess is kept for developer machines only (`REACTOME_MCP_SERVER`, +`src/reactome_mcp/process.py:39`); deployments connect over Streamable HTTP +(`REACTOME_MCP_URL`, c02219a, 2026-09-15; `src/reactome_mcp/session.py:91-104`), and +the image has no Node (`Dockerfile:3`). -**A second router.** #142 adds `create_query_router` alongside the existing -`create_intent_classifier`, which already routes between `reactome` and `userguide`. +**A second router.** #142 added `create_query_router` alongside the existing +`create_intent_classifier`, which then routed between `reactome` and `userguide`. Two LLM classification calls on the same question, in a pipeline where removing LLM calls was the point of spec 001 — retrieval went from 21 calls to one. The -existing classifier should gain the new destinations rather than acquire a sibling. +review's position was that the existing classifier should gain the new destinations +instead. Outcome: #142 was not merged, and `live` was added to the existing +classifier (`src/agent/tasks/intent_classifier.py:10`, 4c4e8a7). + +## User stories as proposed (2026-09-10) -## User Scenarios & Testing *(mandatory)* +Proposed, not recorded as approved. What became of each is in "Requirements, and +where they stand" below. ### User Story 1 — Answers reflect the current release (Priority: P1) @@ -102,9 +125,10 @@ since 95, compare against today. A user pastes a gene list and asks which pathways are over-represented. The chatbot runs the analysis and explains the result. -**Why this priority**: Genuinely new capability, and impossible without something -like MCP. P2 below staleness only because staleness affects every question and this -affects a class of question we do not serve at all today. +**Why this priority**: Genuinely new capability, and at the time thought impossible +without something like MCP (it was later built against Reactome's services directly; +see D2). P2 below staleness only because staleness affected every question and this +affected a class of question not served at all at the time. **Acceptance Scenarios**: @@ -132,60 +156,59 @@ A question is classified once, into one of the destinations available. - **The MCP server dies mid-conversation.** A supervised subprocess needs a defined answer: restart, degrade to RAG, or fail. Silently degrading to RAG is the worst option, because an analysis question would get a retrieval answer. -- **The prototype changes its tool surface.** Five tools are wrapped by name; a +- **The server changes its tool surface.** Five tools are wrapped by name; a rename upstream breaks them at call time, not at start-up. - **Live and stored data disagree.** The vector store says Release95, the API says 97. An answer that mixes both without saying so is a new class of wrong. - **Analysis latency.** Enrichment is not a sub-second call, and the chat surface - already runs at 22s per question. - -## Requirements *(mandatory)* - -### Functional Requirements + then ran at about 22s per question. -- **FR-001**: The installed bundle MUST be rebuildable against the current release - without code changes. -- **FR-002**: Rebuilding MUST be schedulable, not a remembered manual step. -- **FR-003**: The deployed release version MUST be visible — an operator must be able - to see which release is being answered from. -- **FR-004**: If MCP is adopted, it MUST be optional: with it disabled, behaviour and - LLM call count are exactly as today. -- **FR-005**: If MCP is adopted, questions MUST be classified exactly once, by - extending the existing intent classifier rather than adding a second. -- **FR-006**: An unavailable MCP server MUST produce an explicit failure for - analysis questions, never a silent fall back to retrieval. -- **FR-007**: Where an answer draws on live API data rather than the bundle, that - MUST be distinguishable. +## Requirements, and where they stand (checked 2026-10-21) -## Success Criteria *(mandatory)* +Proposed by this review; no approval is recorded. Where one is not built, whether it +is still wanted is not recorded. -- **SC-001**: The gap between the deployed bundle's release and Reactome's current - release is at most one. -- **SC-002**: An operator can determine the answering release without reading code. -- **SC-003**: With MCP disabled, a question costs the same LLM calls as today. -- **SC-004**: A gene-list question produces a real analysis or an explicit refusal, - never a retrieval answer dressed as one. - -## Decisions for the team +| ID | Proposed | Status | +|---|---|---| +| FR-001 | Bundle rebuildable against the current release without code changes | `bin/embeddings_manager make` exists and needs a Neo4j connection passed explicitly (`deploy/rebuilding-embeddings.md`). Whether a full rebuild against a current release has run without code changes is not recorded | +| FR-002 | Rebuilding schedulable, not a remembered manual step | Not built: no scheduled rebuild exists in this repository | +| FR-003 | Operator can see which release is being answered from | Partly: the answer endpoint reports `release` in its `start` event (`src/api/answer.py:130`), read once from the bundle directory name (`bin/chat-fastapi.py:59`; `src/util/embedding_environment.py:67-79`). The name is the only record and can disagree with the data inside (see D1). The chat UI does not show it | +| FR-004 | MCP optional; disabled means behaviour and LLM call count unchanged | Holds: `live` is offered only when `REACTOME_MCP_URL` or `REACTOME_MCP_SERVER` is set (`src/agent/profiles/react_to_me.py:86-88`; `src/reactome_mcp/session.py:59-60`); otherwise the classifier gets the two-source prompt (`src/agent/tasks/intent_classifier.py:110-141, 189-206`; `tests/agent/test_intent_classifier_sources.py:21-33`) | +| FR-005 | Classified once, by extending the existing classifier | Holds: `live` is a value of the existing classifier's `source` (`intent_classifier.py:10, 151-154`) | +| FR-006 | Unavailable MCP gives an explicit failure for analysis, never a silent fallback | Analysis does not use MCP; the chat calls the Analysis Service directly and says so when that fails (`bin/chat-chainlit.py:455-458`; `src/analysis/gene_list.py:585-589`). A `live`-routed question with the MCP unavailable falls back to retrieval with only a log warning (`react_to_me.py:219-243`) | +| FR-007 | Answers drawing on live data distinguishable | Not built as a label. Live answers carry no citations, because no retriever ran (d3740aa) | +| SC-001 | Deployed bundle at most one release behind | Not established; see D1 | +| SC-002 | Answering release visible without reading code | As FR-003 | +| SC-003 | MCP disabled costs the same LLM calls | As FR-004 | +| SC-004 | A gene list gets a real analysis or an explicit refusal | In the chat: a recognised gene list is offered an over-representation analysis, run on confirmation (`bin/chat-chainlit.py:732`; `src/analysis/gene_list.py:419`), with an explicit message on failure. The search page has no analysis path | + +## Decisions put to the team (2026-09-10) ### D1 — Rebuild the bundle now, independently of MCP? -Recommended: **yes, and first.** Two releases behind is the larger share of the -stated benefit, it needs no new dependency, no subprocess and no router. It also -makes the MCP decision honest by removing staleness from its justification, leaving -analysis — which is the real case. +The review recommended **yes, and first**: two releases behind was the larger share +of the stated benefit, and it needed no new dependency, subprocess or router. It +would also remove staleness from MCP's justification, leaving analysis as the real +case. -**Answered 2026-09-14: Release 98 is nearly done, and the plan is to generate new -embeddings and move them to production then.** So D1 is settled and MCP is to be -judged on analysis alone. +Recorded answer, 2026-09-14: Release 98 was nearly done, and the plan was to generate +new embeddings and move them to production then. On that basis MCP was to be judged +on analysis alone. -A readiness check against this machine found three things that are not ready and -would each fail partway through a long job: disk (3.4 G bundle, 5.2 G free), S3 -(no credentials here, so `push` cannot reach production), and Neo4j (needed by -`make`, configured nowhere). Written up in +A readiness check on 2026-09-14 found three things not ready, each of which would fail +partway through a long job: disk (3.4 G bundle, 5.2 G free), S3 (no credentials on +that machine, so `push` could not reach production), and Neo4j (needed by `make`, +configured nowhere). Written up in [`deploy/rebuilding-embeddings.md`](../../deploy/rebuilding-embeddings.md). -### D2 — Adopt MCP for analysis? +What the repository shows since (checked 2026-10-21): by 2026-09-17 beta served a +bundle directory named `Release97` (spec 010; spec 009's `before.json`). Its four +Neo4j-backed collections were built from Release95 data and its `disease_variants` +from the Release 97 download (`src/data_generation/disease_variant/__init__.py:152-156`). +No Release 98 bundle is recorded here, and whether that rebuild happened is not +recorded. + +### D2 — Adopt MCP for analysis? (options as proposed, 2026-09-10) | option | what it means | |---|---| @@ -193,11 +216,22 @@ would each fail partway through a long job: disk (3.4 G bundle, 5.2 G free), S3 | **B. Adopt behind a flag** | Take #127 and #137, fold routing into the existing classifier rather than #142's second one, ship disabled by default. Real capability, contained blast radius, a prototype dependency that cannot affect anyone who has not enabled it. | | **C. Adopt fully** | Everything the PRs propose, on by default. Fastest to the capability, and puts a self-described prototype and a supervised subprocess in the path of every user. | -**Recommendation: B**, contingent on D1 being done first. It buys the irreplaceable -part while the dependency is still a prototype, and FR-004 means the cost of being -wrong is a flag nobody turned on. +The review recommended B, contingent on D1 being done first, on the grounds that it +bought the irreplaceable part while the dependency was described as a prototype, and +FR-004 meant the cost of being wrong was a flag nobody turned on. + +Outcome: approval of an option is not recorded. What was built resembles B, but for +live database lookups rather than analysis: #127 and #137 reworked (dbd02cc), routing +folded into the existing classifier (4c4e8a7), off unless an MCP server is +configured. The MCP's analysis tool is among the five wrapped +(`src/reactome_mcp/tools.py:74`), so it is reachable from `live`-routed questions, but +the classifier describes `live` as questions about the database itself, not analysis +(`src/agent/tasks/intent_classifier.py:29-36`). Analysis in the chat was built +without MCP: gene lists go to the Analysis Service (`src/analysis/client.py:23, 252`; +3b2c792, 2026-09-25) and expression matrices to ReactomeGSA (`src/gsa/client.py:50`; +f51283a, 2026-09-22). -## Assumptions +## Assumptions at the time (2026-09-10) - `reactome-mcp` staying a Reactome project. If it were third-party the answer would be different; a prototype from the same organisation is a shared risk, not an @@ -205,7 +239,7 @@ wrong is a flag nobody turned on. - Rebuilding bundles is routine. If it is not, that is the finding, and it makes D1 more urgent rather than less. -## Out of Scope +## Out of scope for the review - Which MCP tools to wrap beyond the five #137 chose. - Replacing retrieval with MCP. Retrieval over curated text is what the product is; diff --git a/specs/006-mcp-hosting/checklists/requirements.md b/specs/006-mcp-hosting/checklists/requirements.md index e624702d..26a9c5be 100644 --- a/specs/006-mcp-hosting/checklists/requirements.md +++ b/specs/006-mcp-hosting/checklists/requirements.md @@ -1,29 +1,16 @@ -# Specification Quality Checklist: Hosting reactome-mcp +# Claim Checks: Hosting reactome-mcp -**Purpose**: Validate completeness and quality before planning **Created**: 2026-09-14 **Feature**: [spec.md](../spec.md) -## Content Quality +Historical. How the specification's claims were checked when it was written. The +template's completion checkboxes are in git history up to 75ca531. -- [x] Focused on user value and business needs -- [x] Written for non-technical stakeholders -- [x] All mandatory sections completed - -## Requirement Completeness - -- [x] No [NEEDS CLARIFICATION] markers remain -- [x] Requirements are testable and unambiguous -- [x] Success criteria are measurable -- [x] Edge cases identified -- [x] Scope clearly bounded -- [x] Dependencies and assumptions identified - -## Adversarial review of this specification +## Adversarial review of this specification (2026-09-14) | claim | how it was checked | verdict | |---|---|---| -| "`reactome-mcp` is stdio-only" | `src/index.ts` constructs `StdioServerTransport`; no SSE, express or `listen(` anywhere in `src/` | **verified** | +| "`reactome-mcp` is stdio-only" | `src/index.ts` constructs `StdioServerTransport`; no SSE, express or `listen(` anywhere in `src/` | **verified** on 2026-09-14; reactome-mcp gained Streamable HTTP by 2026-09-15 (c02219a) | | "the SDK ships Streamable HTTP" | unpacked `@modelcontextprotocol/sdk@1.30.0`: `server/streamableHttp.js` present | **verified** | | "available without changing the pin" | `reactome-mcp` pins `^1.12.0`, which resolves to 1.30.0 | **verified** | | "Cypher tools are not registered when Neo4j is unset" | `if (isNeo4jConfigured()) registerCypherTools(server)`, and `isNeo4jConfigured()` is `Boolean(NEO4J_URI)` | **verified** — not registered, so not merely disabled | @@ -40,9 +27,11 @@ top of this specification rather than quietly. The lesson is the same one spec 004 recorded: check the thing, not the description of the thing. -### Not verified +### Not verified at the time (2026-09-14) -**The server has still never been run from here.** Tool latency, Analysis Service -behaviour under load, and what a failure looks like to a client are all unobserved. -Every number in this document is a count of code, not a measurement of behaviour. A -plan should open by running it once. +When this was written the server had not been run from here, so every number in the +specification is a count of code, not a measurement of behaviour. Later the same day +it was run: spec 007 measured an analysis at 0.2s over stdio, and `bin/mcp-probe` +recorded per-tool latencies (`src/reactome_mcp/README.md:34-43`). The HTTP client was +verified end to end against a real server on 2026-09-15 (c02219a). Analysis Service +behaviour under load has not been recorded here. diff --git a/specs/006-mcp-hosting/spec.md b/specs/006-mcp-hosting/spec.md index 7c057e44..c19aa2c9 100644 --- a/specs/006-mcp-hosting/spec.md +++ b/specs/006-mcp-hosting/spec.md @@ -4,26 +4,35 @@ **Created**: 2026-09-14 -**Status**: Draft. Findings settled; two decisions (D1, D2) for the team. +**Status**: Historical record of the review of hosting `reactome-mcp`, 2026-09-14. +Drafted as a proposal; this file records no approval of its requirements. D3 and D4 +were recorded as settled and deferred on 2026-09-14. Since then, reactome-mcp gained a +Streamable HTTP transport and the chatbot connects to it when `REACTOME_MCP_URL` is +set (c02219a, PR #219, 2026-09-15); where the server runs (D2) is not recorded in this +repository. The findings, requirements and decisions below were set for that review, +not for later changes; their status is in "Requirements, and where they stand". +Current client behaviour is in `src/reactome_mcp/session.py` and +`src/reactome_mcp/http_client.py`. **Input**: Is `reactome/reactome-mcp` the right design, does it have the features we -want against the alternatives, and where should it run? Analysis must work in the -app; direct Neo4j access is not wanted for now. +want against the alternatives, and where should it run? At the time, analysis was to +work in the app and direct Neo4j access was not wanted. -## A correction first +## A correction to spec 005 (2026-09-14) [Spec 005](../005-mcp-live-data/spec.md) called `reactome-mcp` "just a prototype". That was its GitHub *description* field, not its state. Reading the code says otherwise, and the difference matters because it was an argument against adopting it. -It registers **over 40 tools and 10 resources** across Reactome's Content Service and +As read on 2026-09-14, it registered **over 40 tools and 10 resources** across Reactome's Content Service and Analysis Service: enrichment analysis, full-text search with faceting and spellcheck, pathway hierarchy traversal, entity and complex lookup, PSICQUIC interactors, diagram -and SBGN/SBML export, species and disease annotation, and external ID mapping. The -most recent release (1.4.0) fetches the graph schema live via APOC and caches it per -session — not prototype behaviour. +and SBGN/SBML export, species and disease annotation, and external ID mapping. Its +then most recent release (1.4.0) fetched the graph schema live via APOC and cached it +per session — not prototype behaviour. (`bin/mcp-probe` later counted 53 tools on +1.4.0; `src/reactome_mcp/README.md:34-37`.) -## Does it have what we want? +## Does it have what we want? (assessment, 2026-09-14) **Yes, and specifically the thing that justifies the whole exercise.** Analysis in the app means submitting a gene or protein list and getting over-representation results @@ -31,10 +40,17 @@ with p-values, FDR and found/not-found identifiers. That is `reactome-mcp`'s fir listed feature, wrapping the Analysis Service directly. It cannot come from the embeddings bundle at any freshness, because it is a -computation over a user's list rather than a lookup of stored text. This is the -capability gap spec 005 identified, and this server closes it. +computation over a user's list rather than a lookup of stored text. This was the +capability gap spec 005 identified, and this server could close it. -### Direct Neo4j access stays off, verifiably +In the event, analysis in the chat was built against Reactome's services directly, +not through this server: gene lists go to the Analysis Service +(`src/analysis/client.py:23, 252`) and expression matrices to ReactomeGSA +(`src/gsa/client.py:50`). The server's analysis tool is among the five the chatbot +wraps (`src/reactome_mcp/tools.py:74`) and is reachable only from `live`-routed +questions. + +### Direct Neo4j access stays off, verifiably (as read in reactome-mcp, 2026-09-14) Not a configuration convention — the tools are never registered: @@ -50,10 +66,10 @@ tools do not exist in the server's tool list, so the model cannot see or call th That satisfies "no direct Neo4j for now" without a fork or a patch, and it is a property to re-check rather than assume if the server is ever upgraded. -## The alternatives +## The alternatives (surveyed 2026-09-14) -Six Reactome MCP servers exist. Ours is the most recently maintained and the only one -Reactome controls. +Six Reactome MCP servers existed. Ours was the most recently maintained and the only +one Reactome controls. | repository | language | last push | notes | |---|---|---|---| @@ -69,29 +85,35 @@ strange choice, and none is better positioned on features. `tc2fh/reactome-mcp` worth reading for its transport work; `tc2fh/reactome-db-mcp` goes direct to MySQL, which is the coupling we are explicitly avoiding. -**Recommendation: keep `reactome/reactome-mcp`.** The question worth asking is not -which to adopt but what the others do better, and the one answer is transport. +**The review recommended keeping `reactome/reactome-mcp`**, and it is the server the +chatbot uses. The question it asked was not which to adopt but what the others did +better, and the one answer was transport. -## Where it should run — the actual blocker +## Where it should run — the blocker at the time (2026-09-14) -`reactome-mcp` is **stdio-only**. `src/index.ts` constructs a -`StdioServerTransport` and nothing else: no SSE listener, no HTTP server. +`reactome-mcp` was **stdio-only**: `src/index.ts` constructed a +`StdioServerTransport` and nothing else — no SSE listener, no HTTP server. MCP over stdio is a local subprocess protocol — one client, one spawned process, JSON-RPC over pipes. It cannot be publicly accessible, and it cannot serve two -clients. So "host it somewhere" is not a deployment question yet; it is a change to +clients. So "host it somewhere" was not yet a deployment question; it was a change to `reactome-mcp`. -### The change is small and supported +**Resolved by 2026-09-15.** reactome-mcp gained Streamable HTTP (run as +`node dist/http-server.js` with `MCP_HTTP_PORT`; `src/reactome_mcp/README.md:77-84`), +and the chatbot connects to it when `REACTOME_MCP_URL` is set (c02219a, PR #219; +`src/reactome_mcp/session.py:54-56, 91-104`). + +### The change was small and supported The MCP TypeScript SDK ships `server/streamableHttp` — Streamable HTTP is the -current standard transport for remote MCP servers. `reactome-mcp` already depends on -`@modelcontextprotocol/sdk` at `^1.12.0`, which resolves to 1.30.0, so the transport -is available today without changing the pin. +current standard transport for remote MCP servers. `reactome-mcp` already depended on +`@modelcontextprotocol/sdk` at `^1.12.0`, which then resolved to 1.30.0, so the +transport was available without changing the pin. ### Why hosting it is better than spawning it -The alternative — what #127 proposes — is for the chatbot to spawn the server as a +The alternative — what #127 proposed — was for the chatbot to spawn the server as a subprocess. Hosting removes three costs at once: | | subprocess | hosted over HTTP | @@ -106,7 +128,10 @@ a hosted `reactome-mcp` is useful to anyone with an MCP client, not only to this chatbot. That is a Reactome service, not a chatbot implementation detail — which is the instinct behind running it alongside the website. -## User Scenarios & Testing *(mandatory)* +## User stories as proposed (2026-09-14) + +Proposed, not recorded as approved. What became of each is in "Requirements, and +where they stand" below. ### User Story 1 — A user gets an enrichment analysis (Priority: P1) @@ -162,40 +187,37 @@ supervises no child process. endpoint needs rate limiting, and that is a hosting decision rather than an MCP one. - **Tool surface drift.** Tools are called by name; a rename upstream fails at call time, not at startup. -- **Analysis latency.** Enrichment is not sub-second, on a surface already at 22s per - question. +- **Analysis latency.** Enrichment is not sub-second, on a surface then at about 22s + per question. -## Requirements *(mandatory)* +## Requirements, and where they stand (checked 2026-10-21) -- **FR-001**: `reactome-mcp` MUST serve MCP over Streamable HTTP. -- **FR-002**: The Cypher/Neo4j tools MUST remain unregistered unless explicitly - configured, and this MUST be re-verified on any upgrade. -- **FR-003**: The chatbot MUST reach the server over HTTP, spawning no subprocess and - adding no Node.js to its image. -- **FR-004**: With MCP disabled, chatbot behaviour and LLM call count MUST be exactly - as today. -- **FR-005**: An unreachable server MUST produce an explicit failure for analysis - questions, never a silent fall back to retrieval. -- **FR-006**: The deployment MUST rate-limit, since the server fronts a shared - Reactome service. +Proposed by this review; no approval is recorded. Rows about reactome-mcp itself or +its deployment cannot be checked from this repository. -## Success Criteria *(mandatory)* - -- **SC-001**: A gene list produces a real analysis with p-values, or an explicit - refusal. -- **SC-002**: A second MCP client can use the same server concurrently. -- **SC-003**: No Cypher tool is listed when Neo4j is unconfigured. -- **SC-004**: With MCP off, a question costs the same LLM calls as today. -- **SC-005**: The chatbot image contains no Node.js. +| ID | Proposed | Status | +|---|---|---| +| FR-001 | reactome-mcp serves MCP over Streamable HTTP | Done in reactome-mcp by 2026-09-15; the chatbot's HTTP client was verified against a real server (c02219a) | +| FR-002 | Cypher/Neo4j tools unregistered unless configured; re-checked on upgrade | A property of reactome-mcp; not checked from this repository since 2026-09-14 | +| FR-003 | Chatbot reaches the server over HTTP, no subprocess, no Node in its image | Holds when `REACTOME_MCP_URL` is set (`src/reactome_mcp/session.py:91-104`, `http_client.py`); the image is `python:*-slim` with no Node (`Dockerfile:3`). The stdio subprocess remains for developer machines via `REACTOME_MCP_SERVER` (`session.py:105-112`; `process.py:39`) | +| FR-004 | MCP disabled leaves behaviour and LLM call count unchanged | Holds; see spec 005, FR-004 | +| FR-005 | Unreachable server gives an explicit failure for analysis, never a silent fallback | Analysis does not go through MCP; the chat reports a failed Analysis Service call (`bin/chat-chainlit.py:455-458`). A `live`-routed question with the MCP unreachable falls back to retrieval with only a log warning (`src/agent/profiles/react_to_me.py:219-243`); a failed start is retried after 60 s (`session.py:44-45`) | +| FR-006 | The deployment rate-limits | Concerns the MCP deployment, which is not in this repository; not established. The chatbot bounds what one question sends — at most 6 tool calls per round, 2 rounds (`src/reactome_mcp/answer.py:31-32`) — not what the server accepts | +| SC-001 | A gene list gets a real analysis with p-values, or an explicit refusal | In the chat, via the Analysis Service directly (spec 005, SC-004) | +| SC-002 | A second MCP client can use the same server concurrently | Not recorded. Within one chatbot process, overlapping calls on one session were fixed in 42c505b (PR #313) | +| SC-003 | No Cypher tool listed when Neo4j is unconfigured | As FR-002 | +| SC-004 | MCP off costs the same LLM calls | As FR-004 | +| SC-005 | Chatbot image contains no Node.js | Holds (`Dockerfile`) | -## Decisions for the team +## Decisions put to the team (2026-09-14) -### D1 — Who adds Streamable HTTP to reactome-mcp? +### D1 — Who adds Streamable HTTP to reactome-mcp? (resolved by 2026-09-15) -It is a change in that repository, not this one, and it blocks everything else. The -SDK already provides the transport. This needs an owner and is the critical path. +It was a change in that repository, not this one, and it blocked everything else; the +SDK already provided the transport. reactome-mcp had gained Streamable HTTP by +2026-09-15, when the chatbot's HTTP client was verified against it (c02219a, PR #219). -### D2 — Where does it run? +### D2 — Where does it run? (no decision recorded) | option | what it means | |---|---| @@ -203,16 +225,25 @@ SDK already provides the transport. This needs an owner and is the critical path | B. Beside the chatbot, private | Smaller blast radius, no public surface, no rate-limit worry. Wastes the main advantage: nobody else can use it. | | C. Both | A public instance and a private one for the chatbot. Honest about differing reliability needs; two deployments to maintain. | -**Recommendation: A**, which is where this started. A hosted MCP endpoint for -Reactome is a product in its own right, and the chatbot is then simply its first +The review recommended A, which is where this started: a hosted MCP endpoint for +Reactome is a product in its own right, and the chatbot would then simply be its first client. -### D3 — Does it stay in its own repository? *(settled 2026-09-14: yes)* +No decision is recorded. What this repository shows (checked 2026-10-21): no +deployment file here configures an MCP server — `compose.yaml`, `env_template` and +`deploy/beta/env.beta.template` have no `REACTOME_MCP_*` variable. Commit messages and +code comments record MCP configured on beta from 2026-09-18 (d3740aa; +`src/agent/graph.py:511`), reached at an internal host name +(`src/reactome_mcp/answer.py:156`), which matches option B there. Whether a public +instance exists, and what production uses, is not recorded. + +### D3 — Does it stay in its own repository? *(recorded as settled 2026-09-14: yes)* -Hosting does not change this, because **HTTP is an additional transport rather than -a replacement**. `main()` picks a transport; branching on an environment variable -gives stdio when a client spawns it and HTTP when it runs as a server. Same codebase, -same tools, both modes — which is why the SDK ships both transports side by side. +It has stayed in its own repository. Hosting did not change this, because **HTTP is +an additional transport rather than a replacement**. The proposal was for `main()` to +pick a transport by environment variable; as built, the HTTP server is its own entry +point, `dist/http-server.js`, beside the stdio `dist/index.js` +(`src/reactome_mcp/README.md:30, 81`). Same codebase, same tools, both modes. That matters because people already clone it and point Claude at it. Nothing about hosting takes that away. @@ -234,25 +265,26 @@ features. belongs to whoever operates it and points at a built image; one does not vendor nginx's source to deploy nginx. -### D4 — npm publishing *(deferred 2026-09-14)* +### D4 — npm publishing *(recorded as deferred 2026-09-14)* -`reactome-mcp` is not on npm — `registry.npmjs.org/reactome-mcp` returns 404. Publishing -it would replace clone-plus-build with `npx reactome-mcp` for the people already using -it locally, which is a real ergonomic gain and independent of hosting. +On 2026-09-14 `reactome-mcp` was not on npm — `registry.npmjs.org/reactome-mcp` +returned 404. Publishing it would replace clone-plus-build with `npx reactome-mcp` for +the people already using it locally, which is a real ergonomic gain and independent of +hosting. -**Deferred deliberately**, to be done once across `reactome-mcp`, the website and the -other repositories together. Publishing one in isolation sets a precedent the others -then have to match, so the batching is the point rather than the delay. +Recorded as deferred, to be done once across `reactome-mcp`, the website and the +other repositories together, because publishing one in isolation sets a precedent the +others then have to match. Whether it has been published since is not recorded here. -## Assumptions +## Assumptions at the time (2026-09-14) - The Analysis Service can absorb the traffic. If not, that is a rate-limiting parameter rather than a design change. - `reactome-mcp` stays maintained. It is Reactome's own, so this is a staffing question rather than a third-party risk. -## Out of Scope +## Out of scope for the review - Which tools to expose to the chatbot's model beyond analysis. - Replacing retrieval. MCP adds computation beside it. -- The Cypher tools, deliberately, for now. +- The Cypher tools, deliberately, at the time. diff --git a/specs/007-answer-cascade/checklists/requirements.md b/specs/007-answer-cascade/checklists/requirements.md index 6e05164f..b7230749 100644 --- a/specs/007-answer-cascade/checklists/requirements.md +++ b/specs/007-answer-cascade/checklists/requirements.md @@ -1,25 +1,12 @@ -# Specification Quality Checklist: The Answer Cascade +# Claim Checks: The Answer Cascade -**Purpose**: Validate completeness and quality before planning **Created**: 2026-09-14 **Feature**: [spec.md](../spec.md) -## Content Quality +Historical. How the specification's claims were checked when it was written. The +template's completion checkboxes are in git history up to 75ca531. -- [x] Focused on user value and business needs -- [x] Written for non-technical stakeholders -- [x] All mandatory sections completed - -## Requirement Completeness - -- [x] No [NEEDS CLARIFICATION] markers remain -- [x] Requirements are testable and unambiguous -- [x] Success criteria are measurable -- [x] Edge cases identified -- [x] Scope clearly bounded -- [x] Dependencies and assumptions identified - -## Adversarial review of this specification +## Adversarial review of this specification (2026-09-14) | claim | how it was checked | verdict | |---|---|---| @@ -40,9 +27,8 @@ no obviously right answer. Writing it as though the signal were obvious would have been the easy mistake, and would have hidden the only hard decision in the document. -### Not verified +### Not verified at the time (2026-09-14) -Nobody has measured how often a question would actually fall through to the Search -API. That number decides whether this cascade is worth building or whether the -bundle is simply stale — and the Release 98 rebuild may change it substantially. -Worth measuring before, not after. +Nobody had measured how often a question would actually fall through to the Search +API. That number would decide whether this cascade was worth building or whether the +bundle was simply stale. It has not been measured since, and the cascade was not built. diff --git a/specs/007-answer-cascade/spec.md b/specs/007-answer-cascade/spec.md index f34e9ea1..fe7cf53a 100644 --- a/specs/007-answer-cascade/spec.md +++ b/specs/007-answer-cascade/spec.md @@ -4,13 +4,23 @@ **Created**: 2026-09-14 -**Status**: Draft. One decision (D1) for the team, and it is the hard one. +**Status**: Historical record of the answer-cascade proposal and the observations +behind it, 2026-09-14. Drafted as a proposal; this file records no approval of its +requirements, and D1 was never recorded as decided. Of what it proposed, only FR-012 +(questions about the database answered from live services) was built as written +(4c4e8a7, PR #216, 2026-09-14), off unless an MCP server is configured. Analysis was +built differently: the chat recognises gene lists and analysis requests before the +graph rather than through the classifier. The Search API step was not built. Whether +the unbuilt parts are still wanted is not recorded. The requirements below were set +for that proposal, not for later changes; their status is in "Requirements, and where +they stand". Current routing is in `src/agent/tasks/intent_classifier.py`, +`src/agent/profiles/react_to_me.py` and `handle_message` in `bin/chat-chainlit.py`. **Input**: Detect when a question wants an analysis. Otherwise check the embeddings; if they have no match, search the Reactome Search API for the term; if that finds nothing, fall back to Tavily. -## The shape +## The proposed shape ``` ┌─ analysis question? ──> MCP analysis agent @@ -23,19 +33,19 @@ Two different mechanisms, and the distinction matters: **analysis is a branch ta up front**, because it is a different kind of question. The rest is a **cascade**, each step tried only when the one before it found nothing. -## Half of it already exists +## What existed on 2026-09-14, and since -| step | today | -|---|---| -| classify the question | `intent_classifier` — routes `reactome` / `userguide` | -| embeddings | the RAG chain | -| "was that good enough?" | `completeness_grader` | -| Tavily | `tavily_wrapper`, run in `postprocess` | -| **Reactome Search API** | **nothing** | -| **analysis routing** | **nothing** | - -So the cascade's skeleton is real: retrieval, a grader that decides whether the -answer was good enough, and a web search when it was not. What is being asked for is +| step | 2026-09-14 | since (checked 2026-10-21) | +|---|---|---| +| classify the question | `intent_classifier` — routes `reactome` / `userguide` | also `live` when an MCP server is configured (4c4e8a7), and picks collections (spec 009) | +| embeddings | the RAG chain | same role | +| "was that good enough?" | `completeness_grader` | still gates only the Tavily step (`src/tools/external_search/workflow.py:16-19, 73-79`) | +| Tavily | `tavily_wrapper`, run in `postprocess` | unchanged; chat only — the answer endpoint turns postprocess off (`src/api/answer.py:164`) | +| **Reactome Search API** | **nothing** | still no cascade step; the MCP's `reactome_search` tool is reachable only from `live`-routed questions | +| **analysis routing** | **nothing** | in the chat, by pattern matching before the graph (`bin/chat-chainlit.py:699-760`), not by the classifier | + +So the cascade's skeleton was real: retrieval, a grader that decides whether the +answer was good enough, and a web search when it was not. What was asked for was one new step inserted in the middle, and one new branch at the top. ## The hard part is the signal, not the order @@ -45,7 +55,7 @@ equally easy. ### Search API — free and unambiguous -Measured against `ContentService/search/query`: +Measured 2026-09-14 against `ContentService/search/query`: | query | entries | |---|---| @@ -78,29 +88,36 @@ This is D1, and it is the only decision here that is not obvious. ### Analysis — a classifier, and there is one already Detecting "this wants an analysis" is classification, and `intent_classifier` already -classifies. It should gain the destination rather than acquire a sibling, which is -the mistake #142 makes: a second LLM classification call on the same question, in a +classified. The proposal was for it to gain the destination rather than acquire a +sibling, which was the mistake #142 made: a second LLM classification call on the same question, in a pipeline where cutting calls from 21 to one was the point of spec 001. Analysis questions look different enough to be tractable — they carry a list of identifiers, and ask what is enriched or over-represented. -## What it costs +**What was built instead (checked 2026-10-21).** The classifier has no analysis +destination (`src/agent/tasks/intent_classifier.py:10`). The chat recognises gene +lists and analysis requests by pattern on the user's own message, before the graph +runs and so before the rephrase (`src/analysis/gene_list.py:41-80, 419`; +`bin/chat-chainlit.py:699-760`). That costs no LLM call; a recognised list is offered +an analysis and run only on confirmation. The search page has no analysis path. + +## What it would cost (estimate, 2026-09-14) Worst case, a question that falls all the way through: classification, retrieval and generation, a completeness judgement, a Search API call, and a Tavily call. On a -surface already at ~22s per question, each step must earn its place. +surface then at ~22s per question, each step had to earn its place. -Two mitigations are inherent to the design. Most questions stop at the embeddings, so -the deep path is rare. And the two new steps are the cheap ones — the Search API -measured at well under a second, and analysis at **0.2s** measured through -`reactome-mcp`. +Two mitigations were expected. Most questions would stop at the embeddings, so the +deep path would be rare (assumed, never measured). And the two new steps were the +cheap ones — the Search API measured at well under a second, and analysis at **0.2s** +measured through `reactome-mcp`, both on 2026-09-14. ## Observed, 2026-09-14 Two questions asked on beta by a Reactome developer. Beta was running `e398a37` — production's commit, which predates both the intent classifier and the user -guide source — so this is not a report on current `main`. It is recorded because +guide source — so this is not a report on `main` as it then was. It is recorded because it is what the cascade is for, asked by a real user in their own words. ### "I want to run a gsea with my list of genes" @@ -127,13 +144,19 @@ vector store did not have this" from "Reactome does not have this". That is the "user cannot tell which source answered" edge case, showing up as an assertion rather than as a gap. -It is also a user guide question. `main` has a user guide source and routes to -it — but the user guide bundle is not installed on the beta host, so `main` +Later change (3ad234f, 2026-09-18): the Reactome answer prompt now tells the model to +say an answer was not found "in the Reactome pathway content searched", never that +Reactome lacks it (`src/retrievers/reactome/prompt.py:17-18`). + +It is also a user guide question. `main` had a user guide source and routed to +it — but the user guide bundle was not installed on the beta host, so `main` would log `User guide embeddings not configured; routing will use reactome only` and fall back the same way. **Routing is necessary and not sufficient**; the -source has to be present, and its absence currently degrades silently. +source has to be present. Its absence still degrades silently: a missing user guide +bundle is reported only by that INFO log line, and the classifier is then simply not +offered `userguide` (`src/agent/profiles/react_to_me.py:94-97, 120-125`). -### Reactome does do GSEA — through a service nothing here talks to +### Reactome does do GSEA — through a service nothing here talked to (2026-09-14) An earlier draft of this spec asserted that Reactome performs over-representation analysis and not GSEA, and that the honest reply was to say @@ -145,20 +168,25 @@ There are two analysis services: | | | | |---|---|---| | `AnalysisService` | over-representation over an identifier list | what `reactome-mcp` wraps | -| **ReactomeGSA** (`gsa.reactome.org`) | **PADOG, Camera, ssGSEA, terapadog** | **nothing here talks to it** | +| **ReactomeGSA** (`gsa.reactome.org`) | **PADOG, Camera, ssGSEA, terapadog** | **nothing here talked to it** on 2026-09-14; `src/gsa/` does since 2026-09-21 | Camera is described by the service itself as *"a gene set analysis algorithm similar to the classical GSEA algorithm"*. Verified 2026-09-14 against `GET https://gsa.reactome.org/0.1/methods`, and `reactome.org/gsa/` is a live page. -So the answer to "I want to run a GSEA" is not "Reactome does ORA instead". It -is ReactomeGSA — a Reactome product — and the reason the beta chatbot sent a -user to `fgsea` and a YouTube tutorial is that **no part of this system knows -ReactomeGSA exists**. +So the answer to "I want to run a GSEA" was not "Reactome does ORA instead". It +was ReactomeGSA — a Reactome product — and the reason the beta chatbot sent a +user to `fgsea` and a YouTube tutorial was that **no part of this system knew +ReactomeGSA existed**. This spec treated that as a gap in `reactome-mcp` first. -That is a gap in `reactome-mcp` before it is a gap in the cascade: routing the -question correctly cannot help while there is no tool behind the route. +**What was built instead (checked 2026-10-21).** The chat talks to ReactomeGSA +itself, without MCP: an attached expression matrix runs a ReactomeGSA analysis +(`src/gsa/client.py:50`; `bin/chat-chainlit.py:699-701`), and a request to run one in +words is answered with how to do it in the chat (`src/gsa/chat.py:242-253`; +`bin/chat-chainlit.py:757`). A pasted gene list — identifiers without per-sample +measurements — is offered an over-representation analysis, never ReactomeGSA +(93827ea, 2026-10-02; `src/analysis/gene_list.py`). ### The rephrase turns "do this" into "how do I do this" @@ -181,7 +209,10 @@ alone while the step in front of it is rewriting requests-to-act into questions-about-how — by the time the classifier sees the question, the signal it needs is gone. **FR-011.** -### What `main`'s classifier actually does with them +In the chat this was resolved by detecting analysis requests on the raw message, +before the graph and its rephrase (`bin/chat-chainlit.py:732`). + +### What `main`'s classifier did with them (2026-09-14) Measured 2026-09-14, `gpt-4o-mini`, **three runs per question, classifying the rephrased form as production does**. Every question routed identically on all @@ -204,23 +235,26 @@ hiding variance: Three things follow. -**The second beta failure is already routed correctly.** "what tools are -available for me to run on here" classifies as `userguide` on `main` today. The -classifier is not the problem for that question; the missing bundle is. Install -it on the beta host and that failure goes away without any cascade work — which -is a cheaper fix than this spec, and should be done first. +**The second beta failure was already routed correctly.** "what tools are +available for me to run on here" classified as `userguide` on `main` at the time. The +classifier was not the problem for that question; the missing bundle was. Installing +it on the beta host was proposed as a cheaper first fix than any cascade work. This +repository does not record whether that was done. -**The analysis questions land wherever the two existing destinations allow.** -Two go to `reactome`, one to `userguide` — not wrong given the choices, simply +**The analysis questions landed wherever the two existing destinations allowed.** +Two went to `reactome`, one to `userguide` — not wrong given the choices, simply unable to express the right answer. That is the case for FR-001, measured rather than argued. **A question that belongs nowhere is forced somewhere.** The Nature subscription -question routes to `userguide`, because with two destinations and a required -choice there is no way to say "neither". Today the completeness grader catches -this downstream and Tavily answers it, so the outcome is acceptable by accident. -Adding destinations makes the guess worse, not better: the more sources, the -more confident the misrouting. FR-010. +question routed to `userguide`, because with two destinations and a required +choice there was no way to say "neither". The completeness grader caught this +downstream and Tavily answered it, so the outcome was acceptable by accident. The +argument was that adding destinations makes the guess worse, not better: the more +sources, the more confident the misrouting. FR-010. Since then `live` has been added +as a third destination, and there is still no way to decline +(`src/agent/tasks/intent_classifier.py:10, 151-154`). The grader and Tavily run only in +the chat, not on the search page's answer endpoint (`src/api/answer.py:164`). ### The strongest case is not the cascade at all @@ -247,26 +281,29 @@ There is a class of these: | Which release is this? | cannot know; the bundle is a snapshot | `reactome_database_info` | | Is pathway X in Reactome? | only if X was sampled | `reactome_search` | -All three tools are already wrapped in `src/reactome_mcp/tools.py`, chosen for -exactly this reason. They are not wired to anything. +All three tools were already wrapped in `src/reactome_mcp/tools.py`, chosen for +exactly this reason, and not wired to anything. -**This lowers the cost of the first useful MCP work.** It needs no answer to D1, -no cascade, and no new capability — only that these questions reach a tool -instead of the vector store. The analysis branch is the larger prize; this is -the one available now. +That made it the cheapest useful MCP work: it needed no answer to D1, no cascade, +and no new capability — only that these questions reach a tool instead of the +vector store. **Built the same day** as the `live` destination (4c4e8a7, PR #216; +FR-012), offered only when an MCP server is configured. -### Where these questions live now +### Where these questions were recorded `tests/golden/cascade-questions.txt`, alongside the cases for every other branch -of the cascade — including two that must be answered from the embeddings exactly -as they are today, and one identifier that should stop at the Search API rather -than reach Tavily. +of the proposed cascade — including two to be answered from the embeddings exactly +as before, and one identifier meant to stop at the Search API step, which was not +built. No code reads the file; it is a question list, not a test. They are kept out of `tests/golden/questions.txt` deliberately: that file is a fixed baseline for comparing retrieval captures over time, and adding to it would invalidate every capture taken before. -## User Scenarios & Testing *(mandatory)* +## User stories as proposed (2026-09-14) + +Proposed, not recorded as approved. What became of each is in "Requirements, and +where they stand" below. ### User Story 1 — A gene list gets an analysis (Priority: P1) @@ -290,7 +327,7 @@ analysis rather than retrieval, and comes back with pathways and p-values. A user asks about something in Reactome that retrieval does not surface — a recently added pathway, or an exact identifier. The Search API finds it. -**Why this priority**: this is the step being added, and the gap it closes is real — +**Why this priority**: this was the step being proposed, and the gap it would close is real — the bundle is a snapshot, while the Search API is live. **Acceptance Scenarios**: @@ -325,61 +362,35 @@ marked as external. pathway answer that is not an analysis, and nothing says so. - **Latency on the deep path.** Four steps before an answer appears. -## Requirements *(mandatory)* - -- **FR-001**: Analysis questions MUST be routed to analysis, by extending the existing - classifier rather than adding a second one. -- **FR-002**: The cascade MUST stop at the first step that finds something. -- **FR-003**: The Search API step MUST use its own result count as the match signal — - no model call, no tuned threshold. -- **FR-004**: A step that finds nothing MUST fall through, never fabricate. -- **FR-005**: The source of an answer MUST be distinguishable: bundle, live Reactome, - or the open web. -- **FR-006**: Tavily results MUST remain links beside the answer, not generated into - it. -- **FR-007**: With the new steps disabled, behaviour and call count MUST be exactly as - today. -- **FR-008**: An analysis request MUST be answered with the Reactome service - that performs it, and MUST name which was used. Over-representation - (`AnalysisService`) and gene set analysis (ReactomeGSA: PADOG, Camera, ssGSEA, - terapadog) are different analyses; satisfying a request for one with the other - without saying so is the failure this spec exists to prevent, not relocate. - A request naming GSEA MUST NOT be answered "Reactome does not do that" — it - does, at `gsa.reactome.org`. -- **FR-009**: A source that is configured but unavailable MUST be - distinguishable from a source that had no answer. The user guide falls back to - `reactome` with only a log line when its bundle is missing, so a routing - success and a missing bundle produce the same reply. -- **FR-010**: The classifier MUST be able to decline. It currently returns one - of a closed set and cannot say "none of these" — measured 2026-09-14, "What is - the current price of a Nature subscription?" routes to `userguide`. The - downstream grader rescues that today; adding destinations without adding a way - to decline makes misrouting more confident, not less. -- **FR-012**: A question about the database itself — its species coverage, its - release, whether it contains something — MUST be answered from the live - services, not from the vector store. Measured 2026-09-14: asked which species - Reactome includes, retrieval answered "primarily Homo sapiens ... no - indications of other species". Reactome has 96. This is not a retrieval - defect; a sample of the content cannot describe the scope of the database, so - no improvement to retrieval can fix it. -- **FR-011**: The signal the analysis branch routes on MUST survive the - rephrase, or be taken before it. Measured 2026-09-14: the rephrase rewrites - "I want to run a gsea with my list of genes" as "How can I perform a Gene Set - Enrichment Analysis…", which is a user-guide shape. An intent to *act* becomes - a question about *how to act*, and the classifier cannot recover what the - rephrase removed. - -## Success Criteria *(mandatory)* - -- **SC-001**: A gene-list question yields a real analysis or an explicit refusal. -- **SC-002**: A question about content the bundle lacks but Reactome has is answered. -- **SC-003**: A question with nothing anywhere gets an honest "not covered". -- **SC-004**: A question answered from the embeddings costs the same as today. -- **SC-005**: A user can tell which source answered. - -## Decisions for the team - -### D1 — What counts as "the embeddings found no match"? +## Requirements, and where they stand (checked 2026-10-21) + +Proposed by this spec; no approval is recorded. Where one is not built, whether it is +still wanted is not recorded. The measurements behind FR-010 to FR-012 are in +"Observed, 2026-09-14" above. + +| ID | Proposed | Status | +|---|---|---| +| FR-001 | Analysis questions routed to analysis, by extending the existing classifier | Built differently. In the chat, gene lists and analysis requests are recognised by pattern before the graph (`src/analysis/gene_list.py:41-80, 419`; `bin/chat-chainlit.py:699-760`) and proposed, then run on confirmation. The classifier has no analysis destination (`src/agent/tasks/intent_classifier.py:10`). The search page has no analysis path | +| FR-002 | The cascade stops at the first step that finds something | Not built beyond the existing grader-then-Tavily step | +| FR-003 | The Search API step uses its own result count as the signal | Not built: nothing calls the Search API as a cascade step. The MCP's `reactome_search` tool is used only inside `live`-routed answers (`src/reactome_mcp/tools.py:54`) | +| FR-004 | A step that finds nothing falls through, never fabricates | No cascade to fall through. The prompts tell the model to say what was not found (`src/retrievers/reactome/prompt.py:17`; `src/reactome_mcp/answer.py:75-87`) | +| FR-005 | Answer source distinguishable: bundle, live Reactome, open web | Partly: Tavily results appear separately (FR-006); live answers carry no citations (d3740aa) but are not labelled | +| FR-006 | Tavily results remain links beside the answer | Holds in the chat (`src/agent/profiles/base.py:83-112`; `SearchResults` element, `src/util/chainlit_helpers.py:191`). The answer endpoint runs no web search (`src/api/answer.py:164`) | +| FR-007 | With the new steps disabled, behaviour and call count as before | Holds for `live`: without an MCP server it is not offered and the classifier gets the two-source prompt (`src/agent/tasks/intent_classifier.py:110-141, 189-206`; `tests/agent/test_intent_classifier_sources.py:21-33, 80-84`). The chat's analysis detection has no switch, and adds no LLM call | +| FR-008 | An analysis request is answered by the Reactome service that performs it, naming it; a GSEA request is never told "Reactome does not do that" | Holds in the chat: gene lists go to over-representation, named in the reply (`src/analysis/gene_list.py:520`); expression matrices go to ReactomeGSA (`src/gsa/`); a GSEA request in words is told how to run ReactomeGSA in the chat (`src/gsa/chat.py:242-253`). A gene list is never sent to ReactomeGSA (93827ea, 2026-10-02) | +| FR-009 | A configured-but-unavailable source distinguishable from one with no answer | Not built: a missing user guide bundle is an INFO log line (`src/agent/profiles/react_to_me.py:120-125`); an unavailable MCP falls back to retrieval with a WARNING log only (`react_to_me.py:219-243`) | +| FR-010 | The classifier can decline ("none of these") | Not built: `source` must be `reactome`, `userguide` or `live` (`src/agent/tasks/intent_classifier.py:10, 151-154`) | +| FR-011 | The analysis signal survives the rephrase, or is taken before it | Holds in the chat: detection runs on the raw message before the graph (`bin/chat-chainlit.py:732`) | +| FR-012 | Questions about the database itself (species, release, whether it contains something) answered from live services | Built (4c4e8a7, PR #216): the `live` destination (`src/agent/tasks/intent_classifier.py:29-36`; `src/agent/profiles/react_to_me.py:208-261`). Offered only when `REACTOME_MCP_URL` or `REACTOME_MCP_SERVER` is set; falls back to retrieval if the server is unavailable (FR-009) | +| SC-001 | A gene-list question yields a real analysis or an explicit refusal | In the chat; failure message at `src/analysis/gene_list.py:585-589` | +| SC-002 | Content the bundle lacks but Reactome has is answered | No cascade; `live` answers "is X in Reactome" questions where configured | +| SC-003 | Nothing anywhere gives an honest "not covered" | Through prompt wording only (`src/retrievers/reactome/prompt.py:17`) | +| SC-004 | A question answered from the embeddings costs the same | No cascade step was added to that path | +| SC-005 | A user can tell which source answered | As FR-005 | + +## Decisions put to the team (2026-09-14) + +### D1 — What counts as "the embeddings found no match"? (never decided) | option | what it means | |---|---| @@ -387,22 +398,24 @@ marked as external. | B. A distance threshold | Free and instant, but the number is arbitrary, drifts with the embedding model and the release, and nothing tells you when it has stopped meaning anything. | | C. Threshold first, grader second | Cheap rejection of the obviously-irrelevant, grader for the rest. Two signals to reason about and two ways to be wrong. | -**Recommendation: A.** It is already there, already trusted for the Tavily decision, -and reusing it makes the cascade one consistent idea rather than two. If its latency -proves to be the problem, B becomes a tuning exercise on top — but that should be -driven by a measurement, not a guess. +The proposal recommended A: the grader already existed and was trusted for the Tavily +decision, and reusing it would make the cascade one consistent idea rather than two, +with B as a later tuning exercise if latency proved to be the problem. No decision is +recorded, and the step D1 was for was not built. The completeness grader still gates +only the Tavily step (`src/tools/external_search/workflow.py:16-19, 73-79`). -## Assumptions +## Assumptions at the time (2026-09-14) -- Most questions stop at the embeddings, so the deep path is rare. Worth measuring - once the cascade exists; if most questions fall through, the bundle is the problem. +- Most questions stop at the embeddings, so the deep path is rare. Never measured; + if most questions fell through, the bundle would be the problem. - The Search API stays fast and unauthenticated. - Analysis arrives over hosted MCP ([spec 006](../006-mcp-hosting/spec.md)), not a - spawned subprocess. + spawned subprocess. In the event it was built against the Analysis Service and + ReactomeGSA directly (`src/analysis/`, `src/gsa/`). -## Out of Scope +## Out of scope for the proposal -- Rebuilding the bundle. That is the Release 98 work and reduces how often the - cascade goes deep, but it is not this. +- Rebuilding the bundle. That was the planned Release 98 work (spec 005, D1), which + would reduce how often the cascade goes deep, but it was not this. - Which MCP tools beyond analysis to expose. - Presenting analysis results in the UI. diff --git a/specs/008-disease-variant-embeddings/spec.md b/specs/008-disease-variant-embeddings/spec.md index e1f54d04..af089170 100644 --- a/specs/008-disease-variant-embeddings/spec.md +++ b/specs/008-disease-variant-embeddings/spec.md @@ -4,14 +4,19 @@ **Created**: 2026-09-16 -**Status**: Implemented. Both decisions were delegated back and are recorded below, -with one of the original recommendations withdrawn as impossible. +**Status**: Historical record of the disease-variant collection work, 2026-09-16 to +2026-09-17. Built 2026-09-16 (2855de4, review fixes 7b733ea) and merged in #225 on +2026-09-17. The two decisions were delegated to the implementer, who recorded them +below; one original recommendation was withdrawn as impossible. They were made for +that work, not for later changes. Current behaviour is in +`src/data_generation/disease_variant/__init__.py`, with the retrieval-side handling in +`src/retrievers/csv_chroma.py:35-39`. **Input**: Add disease and variant data to the embeddings, from `disease_variant_ewas_mapping.tsv` in the release download directory. Do it as a new collection alongside regenerating the existing ones for Release 97. -## Why: the chatbot answers about diseases, but cannot name a variant +## Why (2026-09-16): the chatbot answered about diseases, but could not name a variant Asked on 2026-09-16 against beta running `a4de878`, with live MCP: @@ -39,21 +44,22 @@ document for is the individual variant entity. It therefore answers a level up f the question and hedges, which is the failure mode this project keeps finding: a fluent answer that does not contain the fact asked for. -## Why this one is worth doing now +## Why it could be done first (2026-09-16) -It is the only part of the embeddings work that is **not blocked**. +It was the only part of the embeddings work that was **not blocked**. | | source | needs | |---|---|---| -| `complexes`, `ewas`, `reactions`, `summations` | Neo4j at `bolt://localhost:7687` | credentials we do not have | +| `complexes`, `ewas`, `reactions`, `summations` | Neo4j at `bolt://localhost:7687` | credentials not available at the time | | **disease/variant** | **a flat TSV in the download directory** | **nothing** | `generate_reactome_embeddings` reads Neo4j. `generate_alliance_embeddings` does not -- -it reads flat files through `MetaDataCSVLoader` and `build_embeddings`. That is the -precedent to follow, and it means this collection can be built, installed and tested -today while the Release 97 regeneration waits on access. +it reads flat files through `MetaDataCSVLoader` and `build_embeddings`. That was the +precedent followed, so this collection could be built, installed and tested while the +Release 97 regeneration of the other four waited on access. This spec does not record +whether that regeneration has since been done. -## The data +## The data (Release 97 file, examined 2026-09-16) `.../static/download/97/disease_variant_ewas_mapping.tsv`, 5.3 MB, **6,294 rows**, 26 columns, one row per variant entity (6,294 distinct `stable_id`, so no duplication). @@ -69,15 +75,16 @@ reaction the variant takes part in, its functional status, **and the normal reac and pathway it is the defective counterpart of**. Fill rates are high: 20 of 26 columns are ≥95% populated. Two are effectively empty -and should be dropped (`normal_reaction_like_event_go_biological_process_*`, 6.3%); -`entityWithAccessionedSequence_literatureReference_pubMedIdentifier` is 30%. +(`normal_reaction_like_event_go_biological_process_*`, 6.3%), and +`entityWithAccessionedSequence_literatureReference_pubMedIdentifier` is 30%. All three +are dropped (see D1). ## Decisions Both were delegated: *"I think you should decide on the columns ... make your best design and go ahead."* -### D1 -- what is embedded, and what is metadata: decided +### D1 -- what is embedded, and what is metadata (decided by the implementer, 2026-09-16) The loader renders each field as `name: value`, so the **column names are embedded too**. The release names are the query paths that produced them, up to 104 characters @@ -101,18 +108,33 @@ normal_reaction: 4xPALM-C-p-2S-ABCA1 tetramer transports CHOL from transport ves ``` Metadata, not embedded: every identifier -- `st_id`, `uniprot_id`, `disease_id`, -`disease_cross_reference`, and the four Reactome stable IDs. Nobody types +`cross_references`, the four Reactome stable IDs and `reaction_pubmed_ids`. The +metadata also repeats `gene`, `protein`, `disease` and `mutation_type`, which are +embedded too (`METADATA_COLUMNS`, `src/data_generation/disease_variant/__init__.py:86-100`). +`cross_references` was first named `disease_cross_reference`. It was renamed in +7b733ea because it also carries COSMIC, ClinVar, ClinGen and LOVD *variant* ids +(`:51-55`). Nobody types `R-HSA-5682201` at a chatbot, and embedding it costs tokens in every document. The variant's own identifier is named **`st_id`** because that is the key `csv_chroma` de-duplicates on; a different name would silently disable de-duplication here. -Dropped: the two `go_biological_process` columns at 6.3% fill, and -`first_entitySet`. A column empty in 94% of documents earns nothing and costs a line -of `name:` in every one. +Dropped: five of the 26 columns -- the two `normal_reaction_like_event_go_biological_process_*` +columns at 6.3% fill; `entityWithAccessionedSequence_pathway_normalPathway_goBiologicalProcess_accession` +(the process name is kept as `normal_process`); the entity-level +`entityWithAccessionedSequence_literatureReference_pubMedIdentifier` (30% fill); and +`first_entitySet` (`DROPPED`, `__init__.py:102-110`; the CSV keeps only the 21 columns +named in `COLUMNS`, `:36-65`). The recorded reason covers the sparse ones: a column +empty in 94% of documents earns nothing and costs a line of `name:` in every one. No +reason is recorded for the other three. -Median document: 556 characters, about 140 tokens. +Median document: 556 characters, about 140 tokens (measured 2026-09-16). -### D2 -- the pipe-delimited `disease` field: decided, and the earlier recommendation withdrawn +The keyword (BM25) index has used the same 12 content columns since f0e8581 +(2026-10-04). Before that it indexed all 21 CSV columns, so BM25 and vector results +for the same variant never had the same text, and RRF, which keys on the text, could +not fuse them (`src/retrievers/csv_chroma.py:35-39`, `:447-463`). + +### D2 -- the pipe-delimited `disease` field (decided by the implementer, 2026-09-16; the earlier recommendation withdrawn) This spec first recommended splitting the field into a list for metadata. **That is not possible**: Chroma accepts only `str`, `int`, `float` or `bool` as a metadata @@ -127,17 +149,18 @@ Decided: **one document per variant**, with `|` rewritten to `, ` so the field r as a list rather than a path. All disease names stay searchable in the content, and the metadata value stays a string Chroma will accept. -## Scope +## Scope of this work In: one new collection built from this one file; wiring it into the retriever alongside the existing four; answer-sweep expectations that fail if it regresses. Out: `HumanDiseasePathways.txt` and `Reactome2OMIM.txt` -- not needed for this (confirmed 2026-09-16); regenerating the four Neo4j-backed collections for Release 97; -publishing to S3, which is blocked separately -- this host's instance profile is -`EC2CloudwatchAgentRole` and `head_bucket` on `download.reactome.org` returns 403. +publishing to S3, which was blocked separately at the time -- the build host's instance +profile was `EC2CloudwatchAgentRole` and `head_bucket` on `download.reactome.org` +returned 403. -## Result, measured +## Result, measured (2026-09-16, before the BM25 column fix noted under D1) Built into a scratch bundle and asked through the real retriever: @@ -147,22 +170,30 @@ Built into a scratch bundle and asked through the real retriever: | PTEN | *"PTEN Loss of Function in Cancer"* -- a pathway | **Q17\*, Q97\*, Q171\***, named against endometrial cancer | Two answer-sweep expectations pin this. They match a *pattern* for a named variant -(`\bABCA1 [A-Z]\d{2,4}[A-Z*]`) rather than a specific one, because which of the six -come back depends on retrieval order and pinning one would fail a good answer -- -the mistake made in `007` and fixed there. Checked against the recorded answers: the -pattern does not match the old answer and does match the new one. - -They carry `needs_collection="disease_variants"` and skip, loudly, until the bundle -ships -- otherwise they would turn the deploy gate red for a reason nobody can act -on. A test covers the direction that matters: that they run once it is installed. +rather than a specific one, because which of the six come back depends on retrieval +order and pinning one would fail a good answer -- the mistake made in `007` and fixed +there. The pattern was first `\bABCA1 [A-Z]\d{2,4}[A-Z*]`, checked against the +recorded answers on 2026-09-16. On 2026-09-19 (7598b17) it failed a correct answer that +listed the variants as `**C1417R**` without the gene beside them. Both expectations +now use `\b[A-Z]\d{2,4}[A-Z*]\b`, anchored to the topic by `must=("Tangier",)` and +`must=("PTEN",)` (`src/evaluation/answer_sweep.py:125-148`; pinned by +`tests/evaluation/test_answer_sweep.py:244`). + +They carry `needs_collection="disease_variants"` and are skipped, with a reason, when +the installed bundle has no such collection (`answer_sweep.py:266-285`), rather than +turning the deploy gate red for a reason nobody can act on. A test covers the +direction that matters: that they run once it is installed +(`tests/evaluation/test_answer_sweep.py:209`). Cost was cents. The collection is a fraction of the 3.4 GB the existing four take. -## Installing it +## Installing it (beta host, as found 2026-09-16) -The embeddings tree is root-owned, so installing needs sudo -- and using sudo is what -keeps it root-owned. It is not the container's doing: the image has run as `appuser` -(uid 3001) since 2025-04-17 and the bundle was created 2026-09-02, so something was -run under sudo on the host. `~/fix-embeddings-ownership.sh` sets `awright:reactome` -with world-read, which suits both: the owner can manage bundles without sudo, and the -container, whose uid is not a host user and which only reads at runtime, still can. +The embeddings tree on that host was root-owned, so installing needed sudo -- and +using sudo kept it root-owned. It was not the container's doing: the image has run as +`appuser` (uid 3001) since 2025-04-17 and the bundle was created 2026-09-02, so +something had been run under sudo on the host. A host-side script, +`~/fix-embeddings-ownership.sh` (not in this repository), set the tree to the +operator's own user, group `reactome`, with world-read. That suits both: the owner can +manage bundles without sudo, and the container, whose uid is not a host user and which +only reads at runtime, still can. The host's current state is not recorded here. diff --git a/specs/009-collection-routing/contracts/intent_classifier.md b/specs/009-collection-routing/contracts/intent_classifier.md index c0f07169..c09b91c3 100644 --- a/specs/009-collection-routing/contracts/intent_classifier.md +++ b/specs/009-collection-routing/contracts/intent_classifier.md @@ -3,13 +3,13 @@ The classifier is the only interface this feature changes. One LLM call per question, already made. -## Before +## Before routing (until 2026-09-19) ```json {"source": "reactome"} ``` -## After +## With routing ```json {"source": "reactome", "collections": ["disease_variants"]} @@ -30,11 +30,13 @@ produce today's behaviour rather than a narrower search. ## Prompt input -The per-collection descriptions already written in -`src/retrievers/reactome/metadata_info.py` (`reactome_descriptions_info`), which -exist for routing and are currently read only by `bin/retrieval_baseline`. They -become a serving input, so they must stay accurate as collections are added -- -already true of `disease_variants`, described when it was added. +The per-collection descriptions in `src/retrievers/reactome/metadata_info.py` +(`reactome_descriptions_info`). They are a serving input: the classifier prompt +lists the selectable collections from them +(`src/agent/tasks/intent_classifier.py:55-67`), and `bin/routing-probe` checks +coverage against them. `bin/retrieval_baseline` also reads them. A collection with +no description is never offered to the classifier, and an inaccurate description +degrades routing. ## Compatibility diff --git a/specs/009-collection-routing/data-model.md b/specs/009-collection-routing/data-model.md index fa526b3d..49107613 100644 --- a/specs/009-collection-routing/data-model.md +++ b/specs/009-collection-routing/data-model.md @@ -1,4 +1,6 @@ -# Phase 1 Data Model: Collection Routing +# Collection Routing: Data Model + +Describes current behaviour, checked against the code 2026-10-21. ## Selection @@ -8,21 +10,22 @@ The unit this feature adds: which collections a single question should search. |---|---|---| | `collections` | `list[str]` | Collection directory names, e.g. `["disease_variants", "summations"]`. Empty means "all", which is also the fallback for every failure | -Not a new class. It is a field on the classifier's existing structured output and a -key in `RunnableConfig["configurable"]`, because adding a type for it would mean -threading that type through three layers that do not otherwise know about each other. +Not a new class. It is a field on the classifier's structured output +(`QueryIntent.collections`), carried on the graph state and handed to the retriever +through a ContextVar (see "Where it travels"). A dedicated type would have to be +threaded through three layers that do not otherwise know about each other. ### Validity -Decided against the **live bundle**, never a literal: - -```python -valid = set(list_chroma_subdirectories(embeddings_directory)) -``` +Decided against the collections loaded from the installed bundle at startup +(`list_chroma_subdirectories` in `HybridRetriever.from_subdirectory`), never a +literal list; `resolve_collections(selected, available)` receives those names +(`src/retrievers/csv_chroma.py:320`). -Principle V: the same list that decides what is searched decides what is nameable, so -the two cannot drift. A collection added to a bundle is immediately selectable; one -removed cannot be selected. +Principle V: the same list that decides what is searched decides what is valid, so +the two cannot drift. However, the classifier is offered only the collections +described in `reactome_descriptions_info` (`src/agent/tasks/intent_classifier.py:55-67`), +so a new collection needs a description there before it can be chosen. ### Resolution rules @@ -36,29 +39,34 @@ In order. Every failure path widens the search, never narrows it. | All names unknown | all collections, WARNING | As above | | Names valid but none match the question well | those collections | Not detectable here. This is what the measurement is for, and what the sweep catches | -## QueryIntent (existing, extended) +## QueryIntent ```python class QueryIntent(BaseModel): source: SourceName - collections: list[str] = [] # new; empty means all + collections: list[str] = Field(default_factory=list, ...) # empty means all ``` +Ignored unless `source` is `reactome`. + The default matters: a model that omits the field, an older prompt, or a parse failure all produce `[]`, which searches everything. The feature cannot fail closed. ## Where it travels ``` -intent_classifier ──> QueryIntent.collections - │ -ReactToMeState["collections"] (preprocess, beside active_sources) +intent_classifier ──> QueryIntent.collections │ -RunnableConfig["configurable"]["collections"] (generate_answer) +ReactToMeState["collections"] (preprocess; react_to_me.py:187) │ -HybridRetriever.{retrieve,aretrieve}_documents (filters collection_retrievers) +selected_collections ContextVar (set around rag.ainvoke in generate_answer, + │ only when source is "reactome"; reset in + │ finally; react_to_me.py:303-324) +HybridRetriever.{retrieve,aretrieve}_documents + └─ resolve_collections(selected_collections.get(), collection_retrievers) ``` The retriever is constructed once at startup and reads the selection per call. See -research.md R1 for why it is not a constructor argument: `BM25Retriever` is built over +research.md R1 and "The config route the data model describes does not exist" for +why it is not a constructor argument: `BM25Retriever` is built over 17,004 documents for `reactions` alone, which is startup work. diff --git a/specs/009-collection-routing/plan.md b/specs/009-collection-routing/plan.md deleted file mode 100644 index e06a35de..00000000 --- a/specs/009-collection-routing/plan.md +++ /dev/null @@ -1,116 +0,0 @@ -# Implementation Plan: Searching Only the Collections a Question Needs - -**Branch**: `009-collection-routing` | **Date**: 2026-09-17 | **Spec**: [spec.md](./spec.md) - -**Input**: Feature specification from `/specs/009-collection-routing/spec.md` - -## Summary - -Every collection in the bundle is searched for every question, and each contributes -a fixed ten documents whether or not it had anything to say. Adding a fifth cost -**+2,376 context tokens and +3.4s** on a question about CDK5 that has nothing to do -with variants. - -The intent classifier already makes one LLM call per question and returns a source. -It will also return which collections to search. That adds no call and no latency, -and the per-collection descriptions it needs already exist in -`reactome_descriptions_info` -- written for routing, currently read only by -`bin/retrieval_baseline`. - -## Technical Context - -**Language/Version**: Python 3.12 - -**Primary Dependencies**: langchain 1.x, langchain-chroma, chromadb 0.6.3, pydantic 2 - -**Storage**: Chroma collections on disk under `embeddings//reactome//` - -**Testing**: pytest. `bin/retrieval_baseline` for retrieval diffs, `bin/answer-sweep` -for end-to-end answers - -**Target Platform**: Linux container behind FastAPI/Chainlit - -**Project Type**: single project - -**Performance Goals**: reduce context tokens and retrieval latency on questions that -do not need every collection. Today's baseline, measured: 5 collections = 50 docs, -9,437 tokens, 14.9s; 4 collections = 40 docs, 7,061 tokens, 11.5s - -**Constraints**: no additional LLM call; a classifier failure must degrade to -today's behaviour, never to a narrower search - -**Scale/Scope**: 5 collections today, 112,000 documents; the feature exists because -that number will grow - -## Constitution Check - -| Principle | Gate | How this plan satisfies it | -|---|---|---| -| I. Verify the path a user takes | Tests must go through the agent, not only the retriever | The sync retriever is not the served path (`aretrieve_documents` is). Acceptance runs `bin/answer-sweep`, which drives the whole graph | -| II. Measure retrieval changes | `bin/retrieval_baseline` capture before and after, diff reported | Phase 1 captures the baseline **before** any code changes, so the comparison exists to be made | -| III. Characterization tests | Current behaviour pinned before changing it | A test asserting all collections are searched today, so switching to selection is a deliberate edit of test and code together | -| IV. Fail loudly | Config that cannot be honoured stops rather than substitutes | A classifier naming a collection that is not in the bundle is a bug in the prompt or the bundle. It is logged at WARNING and the selection falls back to all collections -- never silently dropped, never a narrower search | -| V. Derive from the source of truth | No hand-synchronised lists | The selectable collections are derived from `list_chroma_subdirectories` on the live bundle, not a literal. `reactome_descriptions_info` is the one place a collection is described | - -**Gate result**: pass. No violations to justify. - -### The gate that is not yet met - -The spec records it: thirteen sweep questions cannot cover five collections, and -**four of the five have no question that fails if routing stops searching them**. -Principle I is only satisfied once they do. That is task work in Phase 1, before the -routing change lands, not after. - -## Project Structure - -### Documentation (this feature) - -``` -specs/009-collection-routing/ -├── spec.md -├── plan.md # this file -├── research.md # Phase 0 -├── data-model.md # Phase 1 -├── contracts/ -│ └── intent_classifier.md -├── quickstart.md -└── tasks.md # /speckit-tasks -``` - -### Source Code (repository root) - -``` -src/ -├── agent/tasks/intent_classifier.py # gains the collection selection -├── agent/profiles/react_to_me.py # threads the selection to the RAG -├── retrievers/ -│ ├── csv_chroma.py # HybridRetriever filters by selection -│ └── reactome/metadata_info.py # descriptions become a serving input -└── evaluation/answer_sweep.py # per-collection questions - -tests/ -├── agent/test_intent_classifier_sources.py -├── retrievers/test_collection_selection.py # new -└── retrievers/test_sync_async_equivalence.py # both paths must filter alike -``` - -## Complexity Tracking - -| Addition | Current need | Why the simpler option is insufficient | -|---|---|---| -| Collection names in the classifier's structured output | Selection must cost no extra latency | A second LLM call doubles the routing cost, which is what the feature exists to reduce | -| Selection passed through `ReactToMeState` | The retriever is built once at startup, per profile | Rebuilding a retriever per question would re-read every BM25 index on every message | - -## Phase 0: Research - -See [research.md](./research.md). - -## Phase 1: Design - -See [data-model.md](./data-model.md), [contracts/](./contracts/), -[quickstart.md](./quickstart.md). - -### Post-design constitution re-check - -Unchanged: pass. The design adds no new LLM call, derives the collection list from -the bundle rather than a literal, and fails toward the wider search. diff --git a/specs/009-collection-routing/quickstart.md b/specs/009-collection-routing/quickstart.md deleted file mode 100644 index ce4f0492..00000000 --- a/specs/009-collection-routing/quickstart.md +++ /dev/null @@ -1,65 +0,0 @@ -# Quickstart: Validating Collection Routing - -## Prerequisites - -- An installed Release97 bundle (`./bin/embeddings_manager which`) -- `OPENAI_API_KEY` -- A reachable MCP server for the two live questions, or accept them as skipped - -## 1. Capture the baseline BEFORE changing anything - -Principle II: the comparison has to exist before the change. - -```bash -./bin/retrieval_baseline capture --out before.json -``` - -## 2. Record the cost the feature exists to reduce - -On a question that needs no variant data: - -```bash -./bin/answer-sweep --only "CDK5" -``` - -Known baseline, 2026-09-17: 5 collections, 50 documents, 9,437 context tokens, -14.9s retrieval. Four collections was 40, 7,061 and 11.5s. - -## 3. After the change - -```bash -./bin/retrieval_baseline capture --out after.json -./bin/retrieval_baseline compare before.json after.json -``` - -The diff is **reported, not gated**: dropping documents is what this change does. It -exits non-zero when anything changed, which is information here rather than failure. - -## 4. The gate - -```bash -./bin/answer-sweep -``` - -Must stay green. Two questions fail if variant questions stop reaching -`disease_variants`; the species and release questions fail if `live` routing breaks. - -**This is the pass/fail.** See spec.md "Clarifications" for why no numeric -document-loss threshold is set. - -## 5. The part that does not exist yet - -Four of five collections have **no** question that fails if routing stops searching -them. Adding those is task work before the routing change lands -- until then the -gate has a hole exactly where this feature could break things. - -## 6. Verify through the path a user takes - -Principle I. The served path is async; `retrieval_baseline` drives the sync one. - -```bash -python -m pytest tests/retrievers/test_sync_async_equivalence.py -``` - -Both paths must filter by the same selection. A change that routes only the sync path -would pass every measurement above and serve unrouted results. diff --git a/specs/009-collection-routing/research.md b/specs/009-collection-routing/research.md index fdc1f3c6..9a459251 100644 --- a/specs/009-collection-routing/research.md +++ b/specs/009-collection-routing/research.md @@ -1,4 +1,8 @@ -# Phase 0 Research: Collection Routing +# Research: Collection Routing (2026-09-17 to 2026-09-19) + +Findings from the collection-routing work. "Decision" in a section means a decision +for that work. Later sections correct earlier ones where they conflict, and +corrections added on 2026-10-21 are marked. ## R1. How does a per-question selection reach a retriever built once? @@ -23,12 +27,10 @@ per-message work. | C. contextvar | Set around the call | Invisible coupling, and wrong under concurrency if it ever leaks across tasks | | D. Encode in the query string | Prefix the question | Reaches BM25 and the query expander as text. The codebase already carries a comment warning against exactly this for `detected_language` | -**Decision: A.** It reuses a path this repo already relies on and adds no new -LangChain surface. - -**Rationale.** `src/agent/profiles/base.py` reads `config["configurable"]` in -`postprocess`, so configurable state is established practice here, and the failure -mode is a missing key -- which defaults to "all collections" and is therefore safe. +**Initial choice: A**, because it reused a path this repo already relied on. +**Superseded 2026-09-18:** that route does not reach the retriever, and a +ContextVar (option C) is used instead; see "The config route the data model +describes does not exist" below. ## R2. Can the classifier name collections reliably? @@ -39,9 +41,10 @@ correctly for the thirteen sweep questions, and sharpening one rule on 2026-09-1 fixed the variant questions without breaking the live ones. A per-collection description already exists for each collection in `reactome_descriptions_info`. -**Decision**: ask for collections in the same structured output, and treat an empty -or unparseable list as "all". Measure with `bin/retrieval_baseline` before deciding -whether it is good enough. +**Decision for this work**: ask for collections in the same structured output, and +treat an empty or unparseable list as "all". It was measured with the answer sweep +and a routing probe rather than `bin/retrieval_baseline` (see the sections dated +2026-09-19). ## R3. What happens when the classifier names a collection that does not exist? @@ -77,14 +80,15 @@ And the inversion: `disease_variants` contributed **15%** of context to that que and **5%** to the ABCA1 question it exists for, because every collection gets ten documents regardless of quality. -**Decision**: `bin/retrieval_baseline capture` over the fixed question set on the -Release97 bundle is task 1, before anything else. +**Decision for this work**: capture the baseline before any change. Done: +[before.json](./before.json), 2026-09-18. No after-capture was made. ## R5. Which path must be measured? The served path is `aretrieve_documents`; `bin/retrieval_baseline` drives `retrieve_documents`. They are separate implementations and were verified equivalent -on 2026-09-17 (PR #227), so measuring the sync path is currently valid. +on 2026-09-17 (PR #227), so measuring the sync path was valid at the time. The +equivalence test still runs. **Decision**: the equivalence test is a prerequisite of this work, not a nice-to-have. Both paths must filter by the same selection, and the existing test must be extended @@ -92,7 +96,9 @@ to assert that -- otherwise the measurement stops describing what users get. ## The config route the data model describes does not exist -**Measured 2026-09-18** against langchain-core 0.2.14, the version pinned here. +**Measured 2026-09-18.** *(Corrected 2026-10-21: the original text said +langchain-core 0.2.14, "the version pinned here", but the lockfile then pinned +1.6.2, as it does now. Which version this ran against is not recorded.)* data-model.md routes the selection as `RunnableConfig["configurable"]["collections"]` into @@ -112,7 +118,7 @@ asyncio each task gets a copy of the context, so one request's narrowing cannot narrow another's — pinned by a test that runs a narrow and a wide request concurrently and asserts neither sees the other. -The data model's diagram is left as the intent; this is how it is carried. +data-model.md now shows the ContextVar route. ## What narrowing actually costs, measured 2026-09-19 @@ -179,7 +185,8 @@ it for at least one question. **One excluded collection is untested by construction.** Narrowing excluded `complexes`, `ewas` and `disease_variants`. No tracked question guards -`complexes` (T005 is still open), so its exclusion could not have produced a +`complexes` (T005 was still open here; it was abandoned later the same day, see +below), so its exclusion could not have produced a failure. The absence of a fourth failure is not evidence. **It still justifies the fail-wide rule.** Where a narrow selection does lose the @@ -218,9 +225,9 @@ Only six of the thirteen questions retrieve at all: the refusals never do, and the userguide questions use a different bundle. So the routing evidence is six questions, which is a small base and should be said rather than glossed. -### It does not reduce the prompt +### Context size: the recorded result conflicts with the code -This is the correction. Narrowing was expected to cut context; it does not. +As recorded 2026-09-19: | question | docs, all | docs, narrowed | chars, all | chars, narrowed | |---|---|---|---|---| @@ -228,11 +235,16 @@ This is the correction. Narrowing was expected to cut context; it does not. | ABCA1 variants | 10 | 10 | 6,163 | 5,846 | | Selective autophagy | 10 | 10 | 22,114 | 21,797 | -**Ten documents either way, context within 2%.** The chain caps the fused list -at ten regardless of how many collections fed it, so narrowing changes *which* -documents arrive, not how many. Any claim that this feature reduces prompt cost -is wrong, and T003's framing -- "context tokens for a question needing one -collection fall" -- was the wrong thing to measure. +**Correction, 2026-10-21.** The explanation recorded here ("ten documents either +way ... the chain caps the fused list at ten") does not match the code. Neither +the retriever nor the chain caps the fused list at ten. Each searched collection +contributes up to `max_documents_per_collection` (10), the lists are fused and cut +to 10 × the number of collections searched (`src/retrievers/csv_chroma.py:612-633`), +and `create_rag_chain` passes all of them to the prompt +(`src/retrievers/rag_chain.py`). That code was already in place when this was +measured (38f8885, the day before), so what the "docs" column counted cannot be +established. By the code, narrowing should reduce how many documents reach the +prompt. This has not been re-measured. ### What it does reduce is retrieval work @@ -249,9 +261,8 @@ collection means two. ### So the honest case for this feature It did not fix a failing question -- the control already passed 13/13. It buys -a fifth off retrieval time and it puts the right documents in a fixed-size -context, which should matter most where the fixed ten are currently crowded out -by the wrong collection. That last part is plausible and **not** measured here. +a fifth off retrieval time and changes which documents reach the model. Whether +that improves answers is plausible and **not** measured here. Against that, a wrong narrow removes an answer rather than degrading it, which is why the prompt leans hard on leaving the selection empty and why every @@ -259,7 +270,8 @@ failure path in `resolve_collections` widens. The sweep shows correct routing on six questions; it does not show that the classifier is right in general. `complexes` still has no guard question (T005), so a classifier that never -routes to it would score full marks. +routes to it would score full marks (at this point; addressed later that day by +the routing probe, below). ### Adversarial check: nine questions the sweep never asks @@ -327,8 +339,8 @@ names a different subset each run. The same configuration, three runs: An earlier four-run pass appeared to show `complexes` making the Nup107 answer *worse* -- 1,1,1,1 against 3,3,3,3 without it. It did not survive being run again. That claim was under-powered and is withdrawn; the variance is larger -than any difference between the arms, which is exactly the trap -[[probabilistic-bugs-need-sized-tests]] describes. +than any difference between the arms, which is exactly the trap of reading a +difference into a sample too small for its variance. Every guard that works is a **single stable token**: `P04637`, `Tangier`, `lysosom`. So the question was inverted to ask for one -- name the components, @@ -353,9 +365,9 @@ is the thing whose answers are too variable to assert on. `complexes` joins `reactions`: a collection whose content is duplicated elsewhere cannot be guarded by asking a question, however the question is -worded. T005 should become what T007 already is -- a retrieval-level assertion -that the collection was searched -- rather than a hunt for a better candidate, -which is now two failed hunts and a structural explanation of why. +worded. So T005 became what T007 already was, a retrieval-level assertion that +the collection was searched (T005a), rather than a further hunt for a candidate, +after two failed hunts and a structural explanation of why. ### And the retrieval-level assertion does not close the original hole @@ -368,7 +380,8 @@ test can catch that, because it is one model call's judgement. Covering it properly needs the routing distribution watched over real traffic, or a periodic probe that asks known-composition questions and checks the -collection was selected. Neither exists. So the residual risk after T005a is: +collection was selected. Neither existed at this point; the routing probe below +is the second. So the residual risk after T005a is: routing can under-serve `complexes` indefinitely, every tracked question still passes, and the only symptom is answers that are quietly worse. @@ -398,9 +411,9 @@ deploying is not currently realised. What that is not: a guarantee. It is a snapshot of one model's judgement on one day. The risk was always drift, and the probe is only a guard if it keeps -being run -- which is why T005c exists and is open. +being run -- which is why T005c wired it into deploys (next paragraph). -The probe runs on every deploy from 2026-09-19, wired into +As wired on 2026-09-19: the probe runs on every deploy, wired into `~/update-beta-chat.sh` beside the sweep. That is what turns it from a snapshot into a guard: the risk was always drift, and a check run once is a measurement rather than a control. diff --git a/specs/009-collection-routing/spec.md b/specs/009-collection-routing/spec.md index 6bf3fd8d..64e45915 100644 --- a/specs/009-collection-routing/spec.md +++ b/specs/009-collection-routing/spec.md @@ -4,16 +4,16 @@ **Created**: 2026-09-17 -**Status**: Planned. D1, D2 and the acceptance bar are all decided and recorded -below; plan.md, data-model.md and tasks.md exist and depend on them. +**Status**: Historical record of the collection-routing work, 2026-09-17 to +2026-09-19. Implemented 2026-09-19 (0fe7d88). The decisions and acceptance bar +below were set for that work, not for later changes. Current behaviour is in +[data-model.md](./data-model.md). **Input**: *"the more tables we add the more tokens we use up and the longer the responses take to return, it would be nice to search the disease variant table if that is what they are asking about."* -## Clarifications - -### Session 2026-09-17 +## Acceptance bar for this work (agreed 2026-09-17) - Q: When routing skips a collection that would have contributed a document, how much recall loss should reject the change? → A: Gate on answers, measure documents. No numeric document-loss threshold is set until there is one real measurement to set it from. @@ -28,13 +28,13 @@ failure this project keeps finding. The weakness in that, stated rather than hidden: thirteen questions is a thin net, and routing could break something nobody tracks. So growing the tracked set is part of -this work rather than a follow-up -- see Success Criteria. +this work rather than a follow-up -- see "How the work was checked" below. -## The cost is real, and it was measured +## The cost before routing (measured 2026-09-17) -Every collection in the bundle is searched for every question. `retrieve_documents` -loops over all of them and each contributes up to `max_documents_per_collection` -(10), regardless of whether it had anything to say. +Before routing, every collection in the bundle was searched for every question, +and each contributed up to `max_documents_per_collection` (10), regardless of +whether it had anything to say. Adding `disease_variants` as a fifth collection, asked a question with nothing to do with variants -- *"What does CDK5 phosphorylate in Alzheimer disease?"*: @@ -45,10 +45,11 @@ with variants -- *"What does CDK5 phosphorylate in Alzheimer disease?"*: | 5 collections | 50 | 9,437 | **14.9s** | **+2,376 tokens (+34%) and +3.4s (+30%)**, all of it noise for that question. The -cost is per collection and per question, so three more tables is roughly +7K tokens -and +10s on every question anyone asks. +cost is per collection and per question, so, as an extrapolation rather than a +measurement, three more tables would be roughly +7K tokens and +10s on every +question anyone asks. -## The allocation is also inverted +## The allocation is also inverted (measured 2026-09-17) Worse than linear cost: `disease_variants` contributes **more** to questions it is irrelevant to than to the one it exists for. @@ -64,7 +65,7 @@ variant documents are short (556 characters median) where pathway prose is long. the fixed allocation spends the most context on the collection with the least to contribute, and the least on the one that answers the question. -## The proposal: decide collections in the call that already happens +## The approach: collections chosen in the existing classifier call The intent classifier already runs **one LLM call per question** and returns a source (`reactome`, `userguide`, `live`). Having it also name collections adds **no @@ -72,11 +73,13 @@ latency and no call**. The descriptions it would need already exist. `reactome_descriptions_info` in `src/retrievers/reactome/metadata_info.py` holds a written description of every -collection and is currently read only by `bin/retrieval_baseline`. It was written -for exactly this kind of routing and is otherwise unused in the serving path. +collection. At the time it was read only by `bin/retrieval_baseline`. It is now +also the source of the classifier's collection list +(`src/agent/tasks/intent_classifier.py:55-67`) and of `bin/routing-probe`'s +coverage check. -`HybridRetriever.retrieve_documents` then loops over the selected collections -instead of all of them -- a filter on `self.collection_retrievers.items()`. +Both retrieval paths now search only the collections `resolve_collections` returns +(`src/retrievers/csv_chroma.py:580`, `:641`). ## Decisions @@ -87,7 +90,9 @@ on 2026-09-17, rather than left to be inferred from downstream documents.* ### D1 -- what happens when the classifier is unsure -**Decided: select all collections.** A wrong selection costs recall silently, +**Decision for this work: select all collections.** Implemented: an empty or +unrecognised selection searches every collection (`resolve_collections`, +`src/retrievers/csv_chroma.py:320`). A wrong selection costs recall silently, which is the failure this project keeps finding; a wrong *default* costs only what we already pay today. So the change can only make things faster, never worse than the current behaviour, unless the classifier actively picks a wrong subset. @@ -106,28 +111,26 @@ Option (a): leave the cap alone. Simple, and routing alone removes most of the w Option (b): give the 50-document budget out by score rather than by collection, so a collection with nothing relevant contributes nothing even when it is searched. -**Decided: (a) first, measured, then (b) separately.** They are independent, -and doing both at once makes it impossible to say which one moved the numbers. - -## How we will know it worked - -`bin/retrieval_baseline` exists precisely for this: it captures what each retriever -returns for a fixed question set and diffs two captures. Measured across two -identical runs BM25 is byte-identical on 80/80 question-collections, so a diff after -this change is a real difference in behaviour rather than noise. - -- `capture` before and after; the diff names every question whose documents changed -- the answer sweep must stay at 13/13 -- including the two variant questions, which - fail if routing sends them past `disease_variants`, and the species and release - questions, which fail if it stops routing to `live` -- the token and latency numbers above, re-measured -- the tracked question set grown before the change lands, not after. Thirteen - questions cannot cover five collections; each collection needs at least one - question that fails if routing stops searching it. `disease_variants` has two - already, and the other four have none. - -That last one matters: the point of this change is a number going down, and it -should be reported as one. +**Decision for this work: (a).** They are independent, and doing both at once +makes it impossible to say which one moved the numbers. Option (b) was deferred +as a separate change. It has not been implemented, and this repository does not +record whether it is still wanted. The cross-collection fusion added in 38f8885 +reorders documents but does not reallocate the budget by score; each searched +collection still contributes up to ten. + +## How the work was checked + +- Across two identical runs, BM25 was byte-identical on 80/80 question-collections, + so a `bin/retrieval_baseline` diff reflects a real change rather than noise. +- The "before" capture is [before.json](./before.json) (2026-09-18). No "after" + capture was made. +- Answer sweep: 13/13 with the classifier choosing, matching the control + (research.md, 2026-09-19). +- Re-measurement: retrieval time was measured (research.md). The recorded + context-size result conflicts with the code and is unverified; see research.md. +- Tracked questions were added for `ewas` and `summations`. `reactions` and + `complexes` cannot be guarded by a question, so a retrieval-level test and the + routing probe cover them instead. ## Finding: the collections overlap more than this spec assumed @@ -170,16 +173,17 @@ whose content is duplicated elsewhere cannot be guarded by asking a question. Fo `reactions` the guard would have to assert on retrieval -- which documents were searched -- rather than on the answer. -Recorded rather than worked around: `tasks.md` T004-T008 are written as though four -such questions exist. Two do. T007 (`reactions`) needs rewriting as a retrieval-level -assertion, and T005 (`complexes`) needs a better candidate before it can be claimed. +Outcome: T007 (`reactions`) became a retrieval-level assertion. T005 (`complexes`) +was abandoned as a question and covered the same way (T005a), plus the routing +probe (T005b). See [tasks.md](./tasks.md) and research.md. -## Scope +## Scope of this work In: collection selection for the `reactome` source, defaulting to all; the measurement above. -Out: the relevance-weighted allocation of D2, which is its own change; `userguide`, +Out: the relevance-weighted allocation of D2 (deferred; see D2); `userguide`, which has one collection; anything about the four Neo4j-backed collections being Release95 while `disease_variants` came from the Release 97 download directory -- -that is recorded in the bundle's `provenance.json` and is spec 008's problem. +that is recorded in the bundle's `provenance.json` and was left to the +disease-variant work (spec 008). diff --git a/specs/009-collection-routing/tasks.md b/specs/009-collection-routing/tasks.md index f7c5173c..73b18f9a 100644 --- a/specs/009-collection-routing/tasks.md +++ b/specs/009-collection-routing/tasks.md @@ -1,98 +1,21 @@ -# Tasks: Searching Only the Collections a Question Needs - -**Feature**: 009-collection-routing | **Plan**: [plan.md](./plan.md) | **Spec**: [spec.md](./spec.md) - -Test tasks are included. Principle II requires a before-and-after measurement on any -change to what reaches the LLM, and Principle III requires current behaviour pinned -before it is changed. - -## Phase 1: Setup - -- [ ] T001 Create branch `009-collection-routing` from main and set `.specify/feature.json` to `specs/009-collection-routing` -- [x] T002 Capture the retrieval baseline before any code change: `./bin/retrieval_baseline capture --out specs/009-collection-routing/before.json` against the installed Release97 bundle -- [ ] T003 [P] Record the current cost in `specs/009-collection-routing/quickstart.md`: documents, context tokens and retrieval seconds for one question needing no variant data - -## Phase 2: Foundational (blocks every user story) - -**The gate has a hole.** Four of the five collections have no question that fails if -routing stops searching them. Routing must not land before this closes, or the -acceptance criterion cannot detect the failure the feature can cause. - -- [x] T004 [P] Add a `summations`-dependent question to `EXPECTATIONS` in src/evaluation/answer_sweep.py (Selective autophagy / lysosome; verified by removal) -- [x] T005 [P] ~~Find a `complexes` candidate that fails without the collection~~ **Attempted and abandoned with a reason, 2026-09-19.** Nine questions over six complexes, in two shapes; every one answered just as well without the collection, because complex names appear throughout `reactions` (as input and output names) and `summations` prose. The one thing unique to `complexes` -- the component list -- has answers too variable to assert on: the same configuration returned 1 to 6 of 7 components. See research.md -- [x] T005a [P] Assert at retrieval level that `complexes` was searched, in tests/retrievers/test_sync_async_equivalence.py -- parametrised over all five real collection names: each is reachable, and selecting it searches nothing else. Verified by sabotaging `resolve_collections` to ignore the selection, which fails all five. **Catches a plumbing failure, not a routing one**: a classifier that never chooses a collection is still undetectable, and needs a production probe rather than a test (research.md) -- [x] T005b Probe the routing decision, since no answer can: `bin/routing-probe` asks the classifier ten questions, two per collection, and checks **what was selected** rather than what any answer said. Fails if a collection is never chosen, or if a question narrows away from where its answer lives; an open selection is never a failure, because widening is the safe direction. First run 2026-09-19: 10/10 correct, every collection chosen -- [x] T005c Run the probe on every deploy, beside the sweep. Wired into `~/update-beta-chat.sh` on 2026-09-19, immediately after the sweep and honouring the same `--skip-sweep` flag. Fails the deploy and points at `--rollback`, deliberately not rolling back automatically -- the same judgement the sweep makes, since a rollback is disruptive and the container is already serving. Skips with a warning on an image built before the probe existed, which is what the currently deployed image does. **That script is not version controlled**, so this task is its only record in the repository -- [x] T006 [P] Add an `ewas`-dependent question to `EXPECTATIONS` in src/evaluation/answer_sweep.py (TP53 UniProt P04637; verified by removal) -- [x] T007 Assert at retrieval level that `reactions` was searched -- closed by the same parametrised test as T005a, in tests/retrievers/test_sync_async_equivalence.py. No answer-level question can guard it, because every reaction name also appears in `summations` -- [x] T008 Verify each new question FAILS when its collection is removed from the bundle copy, and passes with it present; record the evidence in the PR (method established; two of four candidates survived it) -- [x] T009 Pin current behaviour: a characterization test in tests/retrievers/test_collection_selection.py asserting that with no selection every collection in the bundle is searched -- [ ] T010 Run `./bin/answer-sweep` against Release97 and confirm green before any behaviour change - -## Phase 3: User Story 1 — a question searches only the collections it needs (P1) - -**Goal**: cut context tokens and retrieval latency on questions that do not need every -collection, with no extra LLM call. - -**Independent test**: `./bin/answer-sweep` stays green while the measured context -tokens for a question needing one collection fall relative to `before.json`. - -**The second half of that test was the wrong measurement, and it failed.** -Context does not fall: the chain caps the fused list at ten documents however -many collections fed it, so narrowing changes which documents arrive, not how -many (research.md, 2026-09-19). What falls is retrieval time, 1.82s to 1.43s. -The sweep half stands: 13/13 with the classifier choosing. - -- [x] T011 [US1] Add `collections: list[str] = []` to `QueryIntent` in src/agent/tasks/intent_classifier.py, defaulting to empty so an omitted field means "all" -- [x] T012 [US1] Extend the classifier prompt in src/agent/tasks/intent_classifier.py to name selectable collections, sourced from `reactome_descriptions_info` rather than a literal list -- [x] T013 [P] [US1] Add `resolve_collections(selected, available)` to src/retrievers/csv_chroma.py implementing data-model.md: empty means all, unknown names log WARNING and return all -- [x] T014 [P] [US1] Unit-test `resolve_collections` in tests/retrievers/test_collection_selection.py for empty, all-valid, some-unknown and all-unknown, asserting every failure widens rather than narrows -- [x] T015 [US1] Filter `self.collection_retrievers` by the selection in `retrieve_documents` in src/retrievers/csv_chroma.py, reading it from `RunnableConfig["configurable"]["collections"]` -- [x] T016 [US1] Apply the identical filter in `aretrieve_documents` in src/retrievers/csv_chroma.py — this is the served path -- [x] T017 [US1] Extend tests/retrievers/test_sync_async_equivalence.py to assert both paths honour the same selection, and confirm it fails when only one is filtered -- [x] T018 [US1] Carry `collections` on `ReactToMeState` in src/agent/profiles/react_to_me.py, set in `preprocess` beside `active_sources` -- [x] T019 [US1] Pass the selection into retrieval in `generate_answer` in src/agent/profiles/react_to_me.py — **via the `selected_collections` ContextVar, not `config["configurable"]`**: `create_retrieval_chain` gives the retriever no path for extra arguments. LangGraph copies the context into the tasks it spawns, so a value set around the call reaches the retriever inside them. Reset in `finally`, because the graph reuses one task across turns -- [x] T020 [US1] Verify through the agent, not the retriever: a test that a variant question routes to `disease_variants` and a userguide question does not, per Principle I - -## Phase 4: Measurement and acceptance - -- [ ] T021 Capture `./bin/retrieval_baseline capture --out specs/009-collection-routing/after.json` and `compare` it with before.json; report the diff in the PR as information, not as a gate -- [ ] T022 Re-measure context tokens and retrieval seconds for the same question as T003 and state the change as a number -- [ ] T023 Run `./bin/answer-sweep` against Release97 with live MCP; it must be green including all nine collection-dependent questions -- [ ] T024 Record in the PR how often the classifier selected a subset, and how often it named an unknown collection - -## Phase 5: Polish - -- [ ] T025 [P] Update the Status line and add a Results section to specs/009-collection-routing/spec.md with the measured before/after -- [ ] T026 [P] Note in src/retrievers/reactome/metadata_info.py that `reactome_descriptions_info` is now a serving input, so an inaccurate description degrades routing -- [ ] T027 Run `/speckit-analyze` across spec, plan and tasks and resolve anything CRITICAL or HIGH - -## Dependencies - -``` -Phase 1 (T001-T003) - ↓ -Phase 2 (T004-T010) ← blocking: the gate must detect routing mistakes first - ↓ -Phase 3 (T011-T020) ← the feature - ↓ -Phase 4 (T021-T024) ← acceptance - ↓ -Phase 5 (T025-T027) -``` - -Within Phase 3: T011 → T012; T013 → T014, T015, T016; T015 and T016 → T017; T018 → T019 → T020. - -## Parallel opportunities - -- T004-T007: four sweep questions, four independent edits to the same list — write together, run T008 once -- T013 and T014 alongside T011 and T012: `resolve_collections` is pure and does not depend on the prompt -- T025 and T026: different files - -## Implementation strategy - -**MVP is Phase 2 plus Phase 3.** Phase 2 alone is worth landing on its own: it closes -a hole in the gate that exists today, independently of whether routing is ever built. - -Ship Phase 2 as its own PR. If routing then turns out to cost more recall than it -saves, that PR still stands on its own merit. +# Collection Routing: Task-ID Index + +Historical. The step-by-step task list for this work (T001–T027) is in git history +up to 8387647. Kept here are the IDs that code comments and this folder refer to. + +| ID | Task | Outcome | +|---|---|---| +| T003 | Record the pre-routing cost | Done 2026-09-17; figures in spec.md | +| T004, T006 | Sweep questions guarding `summations` and `ewas` | In `src/evaluation/answer_sweep.py`; each verified by removing its collection | +| T005 | Sweep question guarding `complexes` | Abandoned 2026-09-19: complex names also appear in `reactions` and `summations` (research.md) | +| T005a | Retrieval-level test that every collection is reachable and selecting one searches nothing else | `tests/retrievers/test_sync_async_equivalence.py`. Catches plumbing failures, not a classifier that never chooses a collection | +| T005b | Probe what the classifier selects | `bin/routing-probe`; first run 2026-09-19, 10/10 | +| T005c | Run the probe on every deploy | Wired into the beta deploy script 2026-09-19 (`deploy/beta/README.md`, section 5); the script is not in this repository | +| T007 | Guard `reactions` | No question can; covered by the T005a test | +| T008 | Check each guard question fails without its collection | Done; two of four candidates held | +| T017 | Both retrieval paths honour the same selection | `tests/retrievers/test_sync_async_equivalence.py` | + +Not done when the list was retired, and no record of whether they are still wanted: +T021 (capture `after.json` and compare it with `before.json`) and T026 (note in +`src/retrievers/reactome/metadata_info.py` that `reactome_descriptions_info` is a +serving input). diff --git a/specs/010-search-page-answers/contracts/answer_endpoint.md b/specs/010-search-page-answers/contracts/answer_endpoint.md index dfe55052..961d137b 100644 --- a/specs/010-search-page-answers/contracts/answer_endpoint.md +++ b/specs/010-search-page-answers/contracts/answer_endpoint.md @@ -1,19 +1,51 @@ # Contract: the search-page answer endpoint -What this repo commits to providing, and what the website repo can build against. -Shapes are proposals until both sides agree; the *properties* below are the parts -worth arguing about. +What the endpoint does, for the website repo to build against. Checked against the +code on 2026-10-21 (`src/api/answer.py`, `src/agent/graph.py`, +`src/util/caller_token.py`, `src/util/rate_limit.py`). The website built against an +earlier version of this file from 2026-09-18. Where the code differs from something +this file or the spec earlier promised, the difference is listed under "Known gaps" +at the end, not resolved here. Measurements are dated and describe the code of their +day. ## Request ``` -POST /chat/api/answer +POST /api/answer Content-Type: application/json { "question": "what does CDK5 phosphorylate in Alzheimer disease?", "caller_token": "" } ``` +The path sits under the chat's mount point, so one nginx location covers both +(`bin/chat-fastapi.py:73-78`). On beta `CHAINLIT_URI=/chat/guest`, so the path is +`/chat/guest/api/answer` (`deploy/beta/env.beta.template:21`). `/chat/api/answer` is +used only when `CHAINLIT_URI` is unset. + +- `question`: 1 to 2,000 characters. A body that is not valid JSON, or a missing, + empty or overlong `question`, gets HTTP **422** with FastAPI's JSON error body, not + an event stream (`src/api/answer.py:70-72`) +- `caller_token`: may be omitted; an absent or empty token is refused like any other + invalid one + +To watch it stream: + +```bash +curl -N -X POST https://beta.reactome.org/chat/guest/api/answer \ + -H 'Content-Type: application/json' \ + -d '{"question":"what does CDK5 phosphorylate in Alzheimer disease?", + "caller_token":""}' +``` + +`-N` matters: without it curl buffers, and the stream looks like one slow response. +(The 2026-09-18 handover mentioned an nginx rule answering some user agents with 403; +nginx configuration is not in this repository.) + +No CORS headers are sent (`bin/chat-fastapi.py` adds no CORS middleware), so browser +script on another origin cannot read the response. The website calls the endpoint +from its own server (D1). + **`caller_token` is settled (D1, 2026-09-18), and it does not assert humanity.** The field was `human_token` and the premise was wrong: there is no human gate on the search path and there will not be one -- nobody solves a captcha to run a @@ -24,29 +56,42 @@ The website mints it server-side, per request, and verification here is: | claim | required | checked against | |---|---|---| | signature | yes | the public key, EdDSA or RS256 -- never an HS* algorithm | -| `exp` | yes | now; they mint at +120s | +| `exp` | yes | now, with no leeway; they mint at +120s. No maximum lifetime is enforced here | | `aud` | yes | `reactome-chatbot`, overridable with `CALLER_TOKEN_AUDIENCE` | -| `sub` | no, but expected | not validated; used as the rate-limit key | +| `iat`, `nbf` | no | if present, must be numbers no more than 30s in the future (clock leeway) | +| `sub` | no, but expected | not validated; the rate-limit key, else `jti`, else a hash of the token | | `iss` | no | not currently checked -- the signature already identifies the minter | -`sub` is an opaque per-visit id, 128 random bits, not derived from anything about -the reader. The backstop limit keys on it, so repeat questions in one reading -session count as one caller and a freshly minted token does not buy a fresh -allowance. +A token longer than 4,096 characters is refused without being parsed +(`src/util/caller_token.py:49-53`, `:134-137`). The reason for a refusal is logged +here and never returned to the caller. + +`sub` is 128 random bits, not derived from anything about the reader. For a reader +who has passed the website's Turnstile challenge it is that identity cookie's +subject, so it lasts as long as the cookie, across visits; otherwise it is a +per-visit id (`src/util/caller_token.py:15-22`, corrected 2026-09-19). The backstop +limit keys on it, so a freshly minted token does not buy a fresh allowance. **A token with no `aud`, or one minted for a different audience, is refused.** Worth stating because the earlier code refused *every* token carrying an `aud` claim -- PyJWT rejects one when no audience is expected -- so this had to change before the first real token could ever have been accepted. -Abuse control is the website's: the panel is opt-in behind a click, so a crawled -search never reaches a model, and their proxy rate limits by address. +Abuse control is mainly the website's. As they described it on 2026-09-18, the panel +is opt-in behind a click, so a crawled search never reaches a model, and their proxy +rate limits by address. This service adds a backstop: 30 answers per caller per 600s +by default, set by `ANSWER_RATE_LIMIT` and `ANSWER_RATE_WINDOW_SECONDS` +(`src/util/rate_limit.py:94-103`). A request over it is refused like an invalid token +and is not counted. ## Response: Server-Sent Events -Streaming rather than a single JSON body, because the whole answer currently takes -25-32s and a search page cannot wait. Streaming turns that into "something appears in -about two seconds". +HTTP 200 and `text/event-stream` for every request that passes validation, including +refusals. Streaming rather than a single JSON body so text renders as it arrives. It +was chosen when a whole answer took 25-32s, in the hope that something would appear +in about two seconds. It does not, because the first answer token waits for +preprocessing and retrieval: last measured 2026-09-20, first token p50 5.57s and +complete p50 8.90s (see "Budget"). ``` event: start @@ -59,18 +104,46 @@ event: citation data: {"st_id": "R-HSA-8862803", "display_name": "Deregulated CDK5 triggers..."} event: done -data: {"state": "answered", "seconds": 8.4, "answer_id": "Ev3z3JDmIIUF4gkKTfrn9VF83H"} +data: {"state": "answered", "seconds": 8.4, "answer_id": "Ev3z3JDmIIUF4gkKTfrn9VF8"} +``` + +Events, in order: + +- `start`, once, before any model work. `release` is the Reactome release of the + bundle the service started with (an integer such as 97), or `null` if it cannot be + determined (`bin/chat-fastapi.py:55-59`). `answered` is always `true` and predicts + nothing; read the state on `done` +- `citation`, zero to 12 (below) +- `token`, zero or more +- `done`, once, last + +A refusal is the exception. It is **only** a `done`, with no `start`: + +``` +event: done +data: {"state": "refused", "seconds": 0.0} ``` -`state` is one of `answered`, `nothing_found`, `refused`, `failed`. +`state` is one of `answered`, `nothing_found`, `refused`, `failed`: -**`answer_id` is present only when `state` is `answered`** (added 2026-09-25, -spec 013). It names the answer exactly as this stream sent it -- the text after -anchors and the sources list were stripped, and the citations -- kept for an -hour so "Continue in chat" can open the chat on *that* answer rather than a -regenerated one (the same question scores ~0.33 similarity run to run). Pass it -to `POST /api/handoff` as `{"kind": "search", "answer_id": ...}`. It is keyed per -answer, not per question, so two readers of the same search never share one. +| state | when | +|---|---| +| `answered` | at least one answer token was produced | +| `nothing_found` | the stream ended without an answer token. The measured case (2026-09-18) is a question the safety check refused, whose reply is not streamed. A question retrieval cannot answer usually ends `answered`, with text saying it was not found | +| `refused` | the token was missing or invalid, or the caller is over the rate limit. No model call was made | +| `failed` | an error, or the 120s bound was reached. Tokens sent before it stay sent | + +**`answer_id` appears only when `state` is `answered`** (added 2026-09-25, +spec 013), and not always then: it is omitted when the text left after stripping is +empty (`src/api/answer_store.py:47-59`). It is 24 URL-safe characters. It names the +answer exactly as this stream sent it -- the text after links and the sources list +were stripped, and the citations -- kept in process memory for an hour (at most +2,048 answers; lost on restart) so "Continue in chat" can open the chat on *that* +answer rather than a regenerated one (the same question scores ~0.33 similarity run +to run). Pass it to `POST /api/handoff`, under the same prefix as this +endpoint (`bin/chat-fastapi.py:84`), as `{"kind": "search", "answer_id": ...}`. It is +keyed per answer, not per question, so two readers of the same search never share +one. Every other `done` has exactly `state` and `seconds`; a caller that ignores unknown keys parses them all the same way. @@ -87,38 +160,40 @@ unknown keys parses them all the same way. ### Why citations are separate events -So the website renders links in its own style. Returning prose with embedded HTML -anchors -- what the chat UI does today -- would force the search page to parse them -back out and re-style them. Stable IDs resolve at -`reactome.org/content/detail/`. - -At most **12** citations are sent. Measured against the live endpoint this cap is -binding on ordinary questions, so treat it as the most relevant few rather than -the complete set. - -### Some answers have no citations at all, by construction - -A `citation` carries `st_id`, a Reactome stable identifier, and only the Reactome -and disease-variant collections have them. The userguide collection indexes -documentation pages whose metadata is a page URL, so a userguide-routed question -produces **zero citations** -- measured 2026-09-18: "How do I use the pathway -browser?" returned 0 citations and 452 tokens. - -That is a good answer with no sources, not a failure. Two things follow for a -caller: - -- Render nothing rather than an empty heading. The website's panel keeps its - heading and list inside one conditional for this reason. -- It correlates with the **fastest** answers -- the userguide question is also - the 3.3s-to-first-token one -- so it is common rather than exotic, and it is - precisely what the four questions anyone would naturally test with fail to - produce. - -**Citations always precede prose**, and that is structural rather than -incidental: citations are emitted only on `on_retriever_end`, and token events -are gated on that same event, because retrieval completing is how the answer's -tokens are told apart from the query expander's. So sources are complete at the -moment the first token arrives, and a panel may render them before any prose. +So the website renders links in its own style. Returning prose with embedded links +-- what the chat UI shows, as HTML anchors until 2026-10-03 and markdown links since +(d71f6de) -- would force the search page to parse them back out and re-style them. +Stable IDs resolve at `reactome.org/content/detail/`. + +At most **12** citations are sent, in retrieval's fused order +(`src/agent/graph.py:200`, `:548-571`). They are what retrieval found, not what the +prose cited (research R3). On 2026-09-17 the cap was binding on ordinary questions; +that was before citations were taken from the fused result only (38f8885, +2026-09-18), and it has not been re-measured. Treat them as the most relevant few +rather than the complete set. `display_name` can be an empty string. + +### Some answers have no citations at all + +A question routed to the live lookup (the Reactome MCP tools) is answered from tool +calls with no retrieval, so it streams tokens and **no `citation` events** +(`src/agent/graph.py:500-532`; live-path streaming added 2026-09-18, d3740aa). When +the live tools are unavailable, the same question falls back to retrieval and is +cited as usual. + +That is a good answer with no sources, not a failure. Render nothing rather than an +empty heading; the website's panel keeps its heading and list inside one conditional +for this reason. + +*History: until 2026-09-18 user-guide answers had no citations either, because +user-guide documents carry no `st_id`. Measured 2026-09-18, "How do I use the pathway +browser?" returned 0 citations and 452 tokens, and was also the fastest answer (3.3s +to first token). They are now cited by `url` (next section).* + +**On the retrieval path, citations precede prose**, and that is structural: +citations are emitted only on the fused retriever's `on_retriever_end`, and answer +tokens are gated on that same event, because retrieval completing is how the +answer's tokens are told apart from the query expander's. So sources are complete +when the first token arrives, and a panel may render them before any prose. #### Userguide pages are cited by `url` (implemented 2026-09-18) @@ -158,21 +233,24 @@ tests, which used fixtures too clean to contain one. ### What `token` text contains -**Markdown, never HTML.** Headings and lists appear; anchors do not. The answer -prompt is the chat UI's and does emit inline `` links, so the endpoint -strips them -- across fragment boundaries, because the model streams one anchor as -twenty-odd pieces (`' <'`, `'a'`, `' href'`, `'="'`, `'https'`, ...). A caller that -rendered fragments as they arrived would otherwise show the raw tag before it -became a link. - -The anchor's *text* is kept, since removing it would break any sentence with a +**Markdown without links.** Headings and lists appear; links do not. The answer +prompt is the chat UI's and asks for inline markdown links, `[display_name](URL)` +(`src/retrievers/reactome/prompt.py:20`; HTML `` anchors until +2026-10-03), so the endpoint strips both forms (`src/api/answer.py:133-144`, +`src/util/anchor_strip.py`) -- across fragment boundaries, because the model streams +one link as many pieces (one anchor arrived as twenty-odd: `' <'`, `'a'`, `' href'`, +`'="'`, `'https'`, ...). A caller that rendered fragments as they arrived would +otherwise show the raw link before it was complete. Text that might still become a +link is held back briefly; anything held at the end is sent as a final `token`. + +The link's *text* is kept, since removing it would break any sentence with a linked phrase in the middle. Where the model used a link as a trailing citation, that leaves the pathway title as a bare clause -- cosmetic, and the structured `citation` events are the reliable source for links. **No trailing source list.** The answer prompts ask for a bullet list of every citation at the end, which the chat UI renders. This caller gets its links from -the `citation` events, so that list is a duplicate -- and after the anchors come +the `citation` events, so that list is a duplicate -- and after the links come off, a duplicate with no links in it. The endpoint drops the heading and everything after it. @@ -192,39 +270,54 @@ and `SourcesSectionStripper` removes it on the served path. The stripper still accepts the older variants, because a prompt is an instruction and not a guarantee, but no caller needs to know that. -## Properties worth holding to +## Properties -**It must be safe to ignore.** Any failure, timeout, refusal or unverified caller -produces a `done` with a non-`answered` state. The website renders no panel. The -search page must never be slower or broken because this service is down (FR-006, -SC-004). +**Safe to ignore, within the stream.** Every failure, timeout, refusal or unverified +caller ends with a `done` whose state is not `answered`, and the website renders no +panel (FR-006). Two cases fall outside that: a malformed request gets HTTP 422 JSON +rather than a stream, and a failure after tokens have started leaves those tokens +sent, followed by `done` `failed` (see Known gaps). Keeping the search page from +being slowed or broken when this service is down (FR-006, SC-004) is the website's +side. -**Hanging up cancels the work.** Measured: when the caller closes the connection -mid-stream, `CancelledError` is raised inside the answer generator and nothing -further is produced -- 3 tokens read, 3 produced, then cancelled. So a reader who -navigates away does not cost a full model call, and there is nothing for the -proxy to cancel upstream beyond closing the connection. +**Hanging up cancels the work.** Measured 2026-09-18: when the caller closes the +connection mid-stream, `CancelledError` is raised inside the answer generator and +nothing further is produced -- 3 tokens read, 3 produced, then cancelled. So a reader +who navigates away does not cost a full model call, and there is nothing for the +proxy to cancel upstream beyond closing the connection (`src/api/answer.py:192-218`). **The stream is bounded.** The server gives up after 120 seconds and sends `done` -with `state: failed`. A caller still needs its own timeout -- a dropped connection -sends nothing -- but the server will not hold one open indefinitely. 120s is about -twice the slowest complete answer measured; it is a ceiling, not a target. +with `state: failed` (`src/api/answer.py:67`). A caller still needs its own timeout +-- a dropped connection sends nothing -- but the server will not hold one open +indefinitely. 120s was set at about twice the slowest complete answer measured then; +it is a ceiling, not a target. **No answer without a token.** Refused before any model call, not after (FR-003). Search pages get crawled, and every crawled search reaching the model is a bill. -**Not every search gets a panel.** The caller may ask for every query; this service -decides, and may answer `nothing_found` because the question is navigational rather -than answerable (D2, User Story 3). +**Every request that passes the checks is answered.** Nothing decides that a search +is not worth answering; earlier versions said navigational questions might get +`nothing_found`, which was never built (D2, User Story 3). **The answer is tied to a release.** `start` carries it so a cached answer can be -invalidated after a release (FR-007). +invalidated after a release (FR-007). It can be `null`, and no cache exists on this +side (D3). -**One brain.** The endpoint and the chat UI share a graph. If they can disagree about -the same question, that is a defect. +**One graph.** The endpoint and the chat UI share one graph instance and profile +(`src/agent/registry.py`; `tests/api/test_answer_matches_chat.py`). The endpoint +skips the post-answer web search the chat runs (`src/api/answer.py:158-164`), which +does not change the answer text. The same question still gets differently worded +answers run to run on either surface (~0.33 similarity, 2026-09-18), so identical +text is not something either side can rely on. ## Budget +No latency is promised. The latest recorded measurement is 2026-09-20, on +`astream_answer` (the graph method this endpoint streams from), arms interleaved, +n=25: first token p50 **5.57s** (range 2.26s-11.44s), complete p50 **8.90s** (T022 in +[research.md](../research.md)). The rest of this section is the 2026-09-18 record +the website first built against, kept with its corrections. + Measured 2026-09-18 over the fifteen tracked sweep questions, two runs each, warm process, `gpt-4o-mini` at temperature 0. @@ -239,23 +332,23 @@ first token p50 10.8s, p90 12.2s. **An earlier version of this contract said the first token arrives "around 36 seconds". That was wrong** -- it came from a single question, and it has not been -reproducible since. Two things changed underneath it: collection routing narrowed -retrieval to the sources a question actually needs, and the four preprocessing +reproducible since. Two things changed underneath it: source routing sent each +question to the source it needs (Reactome bundle, user guide or live lookup), and the four preprocessing calls now run in two rounds instead of four. Preprocessing is 2.6s of the total, not the ~16s previously recorded. -**The first increment still makes no latency promise**, and ten seconds is still -far from the two an AI overview is reported to take. But design the panel for ten -seconds, not thirty: the gap between those is the difference between a brief wait -and an abandoned page. +**The first increment made no latency promise**, and none has been made since. The +advice given on 2026-09-18 was to design the panel for about ten seconds, not +thirty; the 2026-09-20 figures above are within that, with a long tail on the first +token. -Four of the thirty runs produce no answer token at all. Those are the two -deliberately unsafe questions, which return `nothing_found` in about 4.5s with an -empty body. A panel must handle "nothing, quickly" as a normal outcome. +Four of the thirty runs produced no answer token at all. Those were the two +deliberately unsafe questions, which returned `nothing_found` in about 4.5s with an +empty body. A panel has to handle "nothing, quickly" as a normal outcome. A naive measurement reports 3.0s to first token. That token is the rephraser's. -**Where the time goes now**, at the median of a four-question repeated measure: +**Where the time went on 2026-09-18**, at the median of a four-question repeated measure: | phase | seconds | |---|---| @@ -263,15 +356,17 @@ A naive measurement reports 3.0s to first token. That token is the rephraser's. | query expansion and retrieval | 2.3 | | retrieval finishing to the answer's first token | 6.1 | -The largest block is no longer retrieval or preprocessing: it is the answer model's -own time to first token with a retrieved context. +On these figures the largest block was the answer model's own time to first token +with a retrieved context. The 2026-09-20 measurement puts the answer model at 0.84s +and retrieval at 3.15s, the largest; the two records are not reconciled +(research.md, T022). **A correction to an earlier version of this file**, which said collection routing -had landed and explained the improvement. It has not. What landed is *source* -routing -- a question goes to the Reactome bundle, the user guide, or a live -lookup -- while [009](../../009-collection-routing/spec.md), selecting among the -collections *within* the Reactome bundle, is still unimplemented; retrieval -searches all five. +had landed and explained the improvement. It had not then: what had landed was +*source* routing. Collection routing ([009](../../009-collection-routing/spec.md)), +which selects among collections *within* the Reactome bundle, landed later, on +2026-09-19 (0fe7d88). The figures in this section were measured before it, with +retrieval searching all five collections. So the honest position on why the earlier 12.5s and ~36s figures no longer reproduce is that we do not fully know. Source routing accounts for the fast @@ -279,16 +374,37 @@ user-guide answers (3.3s to first token). Running preprocessing in two rounds accounts for about 2.5s. The rest may simply be that those figures came from one question and were never representative. -One measurement worth keeping in view for anyone optimising this: the async -retrieval path is **not faster than the sync one** here -- 12.5s against 10.9s on the -same five queries. The concurrency in `aretrieve_documents` is not currently buying -throughput, so a plan that assumes it will is assuming something unmeasured. - -## What the website side needs to decide - -1. How a verified person is represented (D1) -- the security boundary -2. Whether it calls this service directly from the browser, or proxies server-side. - Proxying makes the token question easier and keeps this service off the public - internet -3. What it renders for each `state`, particularly `nothing_found` -- probably nothing -4. Whether repeat searches are cached at its layer or this one (D3) +One measurement worth keeping in view for anyone optimising this (2026-09-17, before +collection routing): the async retrieval path was **not faster than the sync one** +-- 12.5s against 10.9s on the same five queries. The concurrency in +`aretrieve_documents` was not buying throughput then, and it has not been +re-measured. + +## Questions put to the website side (2026-09-17), and where they stand + +1. How a caller is represented -- the security boundary. Settled 2026-09-18 (D1): a + token the website mints per call, verified as above +2. Whether it calls this service from the browser or through a server-side proxy. It + proxies: the token is minted on its server and its proxy rate limits (D1) +3. What it renders for each `state`, particularly `nothing_found`. The website's + choice; not recorded here +4. Whether repeat searches are cached at its layer or this one (D3). Open; no cache + exists on this side + +## Known gaps + +Differences between what this file or the spec earlier promised and what the code +does, as of 2026-10-21. Recorded, not resolved: which side should change has not been +decided. + +- **First token in about two seconds.** Given as the reason for streaming; last + measured at p50 5.57s (2026-09-20). FR-005a is an unmet target +- **Navigational searches answered `nothing_found`.** Earlier versions said this + service might decline navigational queries (D2, User Story 3). Not built +- **"No panel" after tokens have started.** A failure or timeout after the first + token leaves the streamed text with the caller, then sends `done` `failed` with no + `answer_id` (`src/api/answer.py:219-262`). Task T018 described failure as "no + partial answer". A panel that shows tokens as they arrive has to discard them on + `failed` to show nothing +- **Malformed requests** get HTTP 422 JSON, not a `done` event +- **Caching by question and release** (FR-007, D3). `release` is sent; no cache exists diff --git a/specs/010-search-page-answers/plan.md b/specs/010-search-page-answers/plan.md deleted file mode 100644 index 84702261..00000000 --- a/specs/010-search-page-answers/plan.md +++ /dev/null @@ -1,108 +0,0 @@ -# Implementation Plan: Chatbot Answers in the Website Search Results - -**Branch**: `010-search-page-answers` | **Date**: 2026-09-17 | **Spec**: [spec.md](./spec.md) - -**Input**: Feature specification from `/specs/010-search-page-answers/spec.md` - -## Summary - -Build the endpoint first, slow, and let the website integrate against something real -while the latency work happens behind it. Their side is blocked on nothing but this -service's absence; ours is blocked on nothing but time. - -Three things do not exist: an HTTP answer endpoint, any streaming surface, and -token verification. The latency budget (FR-005) is deliberately **not** a gate on the -first increment -- an endpoint that answers correctly in fifteen seconds unblocks the -other repo; an endpoint that does not exist blocks it completely. - -## Technical Context - -**Language/Version**: Python 3.12 - -**Primary Dependencies**: FastAPI, langchain 1.x, langgraph, Chainlit (mounted on the -same app) - -**Storage**: none new. The graph, bundle and MCP are as they are - -**Testing**: pytest; `bin/answer-sweep` for answer equivalence - -**Target Platform**: the existing `biochat_beta_guest` container behind nginx - -**Project Type**: single project - -**Performance Goals**: FR-005 is first token ≤2s, complete ≤10s. Today p50 is 15.2s, -p90 22.4s, with no first-token concept at all - -**Constraints**: the endpoint and the chat UI must share one graph (SC-003); a -failure must render as "no panel", never a broken one (FR-006) - -**Scale/Scope**: one endpoint, one streaming surface, one verification step - -## Constitution Check - -| Principle | Gate | How this plan satisfies it | -|---|---|---| -| I. Verify the path a user takes | Test through the served path | The endpoint is tested over HTTP with a real client, not by calling the handler. The served path is async and mounted alongside Chainlit, which is where mounting order and middleware interact | -| II. Measure retrieval changes | Before-and-after on real questions | This adds no retrieval change. If the latency work later touches retrieval, that measurement belongs to it, not here | -| III. Characterization tests | Pin current behaviour first | The captcha middleware currently intercepts every path under `CHAINLIT_URI`. A test pins that before a new route is added beside it | -| IV. Fail loudly | No plausible substitutes | An unverified request is refused, not answered anonymously. A missing signing key stops the endpoint from starting rather than accepting everything | -| V. Source of truth | No hand-synchronised constants | Citations come from retrieved documents' `st_id` metadata, not from parsing anchors out of the model's prose | - -**Gate result**: pass. - -### The one that needs care - -Principle IV cuts both ways here. FR-006 says fail **invisibly** -- any error renders -as "no panel" -- and Principle IV says fail **loudly**. They are not in conflict but -the line matters: *misconfiguration* stops the process (no signing key, no bundle), -while a *runtime* failure answering one question returns `state: failed` and is -logged. The first is loud because an operator must fix it; the second is quiet -because a search page must not break. - -## Key design decisions, from reading the code - -**Citations come from retrieval, not from the prose.** The prompt currently instructs -the model to emit `` anchors inline. Parsing those back out of a token -stream would be fragile and would couple the contract to prompt wording. The -retrieved documents already carry `st_id` in metadata, so citation events are emitted -from the retrieval result. This is why the contract specifies stable IDs rather than -HTML. - -**Streaming needs a surface that does not exist.** `AgentGraph` exposes `ainvoke` and -nothing else; Chainlit gets its streaming from `AsyncLangchainCallbackHandler` with -`final_stream`, not from a graph-level API. Two options, resolved in research.md. - -**The FastAPI app does not hold a graph.** `AgentGraph` is constructed in -`bin/chat-chainlit.py`, which Chainlit mounts. The endpoint needs one, and building a -second would double 51.5s of startup and risk the two answering differently, which -SC-003 forbids. - -## Project Structure - -``` -specs/010-search-page-answers/ -├── spec.md, plan.md, research.md -├── contracts/answer_endpoint.md -├── quickstart.md -└── tasks.md - -src/ -├── api/ # new: the endpoint, its models, SSE framing -├── agent/graph.py # gains a streaming surface -└── util/caller_token.py # new: signature verification only - -bin/chat-fastapi.py # mounts the router; middleware ordering matters -tests/api/ # new -``` - -## Complexity Tracking - -| Addition | Current need | Why the simpler option is insufficient | -|---|---|---| -| A second answer surface beside Chainlit | The search page cannot use a websocket chat UI | Chainlit's socket protocol is not a documented API and is not something another repo should couple to | -| Streaming on `AgentGraph` | FR-002, and 10s is only survivable if text appears early | Returning a complete answer means a blank panel for fifteen seconds | - -## Phases - -Phase 0 research: [research.md](./research.md). Phase 1 design: the contract already -exists and was agreed with the website session; `quickstart.md` covers validation. diff --git a/specs/010-search-page-answers/quickstart.md b/specs/010-search-page-answers/quickstart.md deleted file mode 100644 index b662aca9..00000000 --- a/specs/010-search-page-answers/quickstart.md +++ /dev/null @@ -1,46 +0,0 @@ -# Quickstart: validating the answer endpoint - -## Baseline, before any of this is built - -Measured 2026-09-17 across the 15 tracked sweep questions, so Phase 5 has a before: - -- p50 **15.2s**, p90 **22.4s**, min 6.7s, max 31.5s -- no first-token concept: the answer arrives whole or not at all - -## Once the endpoint exists - -```bash -curl -N -X POST https://beta.reactome.org/chat/api/answer \ - -H 'Content-Type: application/json' \ - -d '{"question":"what does CDK5 phosphorylate in Alzheimer disease?", - "caller_token":""}' -``` - -`-N` matters. Without it curl buffers and the stream looks like a single slow -response, which is the thing being tested. - -Expect `event: start` immediately, `event: token` repeatedly, `event: citation` as -retrieval resolves, and `event: done` with a state. - -## What to check, and what not to conclude - -**Tokens actually stream.** Not that the endpoint returns 200. A 200 that completes -without streaming is a pass on the wrong question -- the same shape as the userguide -shell that returned 200 and contained stylesheets. Count the events. - -**A refused request makes no model call.** Assert on a patched graph, not on how fast -the refusal came back; a fast refusal and an expensive one look alike from outside. - -**The endpoint and the chat agree.** Same question through `bin/answer-sweep` and -through the endpoint. Two surfaces that can disagree about the same question is a -defect, not a feature (SC-003). - -**Failure renders as no panel.** Stop the graph, or send a malformed question, and -confirm a `done` with a non-`answered` state rather than a hang or a 500 body the -page would try to render. - -## What this does not prove - -Latency. The first increment is deliberately slow -- correct, verified and streaming, -at roughly today's p50. FR-005's 2s/10s is Phase 5, and the website repo is not -waiting on it. diff --git a/specs/010-search-page-answers/research.md b/specs/010-search-page-answers/research.md index 05c12c81..f992706f 100644 --- a/specs/010-search-page-answers/research.md +++ b/specs/010-search-page-answers/research.md @@ -1,4 +1,10 @@ -# Phase 0 Research: the answer endpoint +# Research: the search-page answer endpoint (2026-09-17 to 2026-09-20) + +Findings from the work in [spec.md](./spec.md). "Decision" in a section means a +decision for that work. R1-R6 are the 2026-09-17 starting points; several were +corrected by later measurements, and the sections dated 2026-09-20 supersede them +where they conflict. Notes added on 2026-10-21 are marked. `src/agent/graph.py` cites +R3 by name. ## R1. How does the endpoint stream, given `AgentGraph` has no streaming surface? @@ -12,11 +18,12 @@ something a plain HTTP handler can use. | B. A callback handler pushing to an `asyncio.Queue` | Mirrors what Chainlit does; the SSE response drains the queue | Works, and is what to fall back to if filtering A's event stream proves unreliable. Costs a bespoke handler and a lifetime to manage | | C. Two graphs, one per surface | Simplest to write | Rejected. Doubles 51.5s of startup and lets the panel and the chat disagree, which SC-003 forbids | -**Decision: A, with B as the named fallback.** The risk in A is that the event +**Decision for this work: A, with B as the named fallback.** A was built +(`AgentGraph.astream_answer`, `src/agent/graph.py:461`). The risk in A is that the event filtering depends on which node emits the final answer, so a graph change could silently stop the stream. A test asserting that tokens actually arrive -- not merely that the endpoint returns 200 -- is the guard, and that lesson is fresh: a 200 with -no content is exactly how the userguide shell fooled two sessions today. +no content is exactly how the userguide shell fooled two sessions on 2026-09-17. ## R2. Where does the graph live? @@ -24,21 +31,29 @@ no content is exactly how the userguide shell fooled two sessions today. onto the FastAPI app in `bin/chat-fastapi.py`. The endpoint needs the *same* instance, not another one. -**Decision**: construct it once in the FastAPI app's lifespan and let both surfaces -use it. Failing that, a module-level accessor the Chainlit app also imports. +**Decision for this work**: construct it once in the FastAPI app's lifespan and let +both surfaces use it. Built as both: the lifespan builds it +(`bin/chat-fastapi.py:38-66`) and `src/agent/registry.py` is the accessor, which +also builds on demand when Chainlit runs without the FastAPI app. -**Rationale**: SC-003 requires the endpoint and the chat to answer identically, and -the cheapest way to guarantee that is one object. A second graph also pays 51.5s of -startup again and doubles the memory of the BM25 indexes. +**Rationale (2026-09-17)**: SC-003 then required the endpoint and the chat to answer +identically, and the cheapest way to guarantee that is one object. SC-003 was later +restated as "configured the same", because one surface does not answer identically +even to itself (spec.md, SC-003). A second graph would also pay the startup cost +again (51.5s as measured then) and double the memory of the BM25 indexes. ## R3. How are citations produced? -Not by parsing the model's prose. The answer prompt instructs `` anchors, -and pulling those out of a token stream would couple the wire contract to prompt -wording -- prompt changes have happened twice this week. +Not by parsing the model's prose. The answer prompt instructed `` anchors +(markdown links since 2026-10-03), and pulling those out of a token stream would +couple the wire contract to prompt wording -- prompt changes had happened twice that +week. -**Decision**: emit citation events from the retrieved documents, whose metadata -already carries `st_id`. Deduplicate, and emit as each document set arrives. +**Decision for this work**: emit citation events from the retrieved documents, whose +metadata carries `st_id`. Deduplicate, and emit as each document set arrives. *(As +built since 2026-09-18: only from the fused retriever result, not from each +sub-retriever (38f8885); at most 12; a user-guide page cited by `url` instead (T028). +`src/agent/graph.py:200-261`, `:534-571`.)* **Consequence worth stating**: the citations are then "what retrieval found", which is a superset of "what the answer used". The contract does not promise otherwise, and @@ -51,10 +66,17 @@ change with its own measurement. Agreed with the website session on 2026-09-17: they mint after their own captcha, asymmetric signing, they hold the signing key and this service holds only a verifying key. The query budget is theirs, enforced before the call, because they proxy every -request. - -**Decision here**: verification is signature + expiry, stateless, and nothing else. -No consumption tracking, no shared secret, no second counter that can disagree. +request. *(The captcha premise was withdrawn on 2026-09-18: the search path has no +human check, and the token asserts caller identity instead -- spec.md, D1. It was +renamed `caller_token`.)* + +**Decision for this work (2026-09-17)**: verification is signature + expiry, +stateless, with no consumption tracking, no shared secret and no second counter. +*As built since: audience is also enforced (2026-09-18, T025); `iat`/`nbf` more than +30s in the future, and tokens over 4,096 characters, are refused (2026-10-03); and a +backstop rate limit keyed on `sub` was added (T017, `src/util/rate_limit.py`). It +remains stateless as to token use: a token is never recorded as spent +(`src/util/caller_token.py`).* **Startup**: a missing or unreadable verifying key stops the process. Principle IV -- an endpoint that accepts everything because its key is absent is the worst outcome @@ -66,16 +88,24 @@ available, and it would test clean. allowlist under `CHAINLIT_URI`. A new route under `/chat/api/` would otherwise be redirected to the captcha page. -**Decision**: the middleware must let the API path through, and the endpoint does its -own verification. A characterization test pins the middleware's current behaviour -first, so this change is deliberate rather than incidental. +**Decision for this work**: let the API path through the middleware and have the +endpoint do its own verification, with a characterization test pinning the +middleware first. Built: `src/util/captcha_scope.py` exempts `/api/` +(`bin/chat-fastapi.py:135-152`), pinned by `tests/util/test_captcha_scope.py` (T003). -This is also a reminder that the middleware is where a bug was found today: it read -`os.environ` for a secret that comes from `get_secret`, so a mounted Docker secret -silently disabled the captcha. Adding a route beside it warrants care. +The middleware was also where a bug was found on 2026-09-17: it read `os.environ` +for a secret that comes from `get_secret`, so a mounted Docker secret silently +disabled the captcha. It now uses the `get_secret` value (`bin/chat-fastapi.py:87`, +`:140-145`). ## R5b. Streaming does not rescue the latency, and the reason is upstream +*Note added 2026-10-21: these are one question's figures. Runs across the tracked +questions did not reproduce the 36.1s (T019, 2026-09-18), and preprocessing has run +in two rounds rather than four sequential calls since 2026-09-18 (T019a). The +structural point -- nothing of the answer exists until preprocessing and retrieval +finish -- still holds. Later figures: "T022" below.* + Measured 2026-09-17 by streaming `astream_events` from the compiled graph and grouping by `run_id`. Six model calls run before and during one answer: @@ -88,18 +118,20 @@ grouping by `run_id`. Six model calls run before and during one answer: | 19.8s | 80 | model | query expansion | | **36.1s** | **1,189** | model | **the answer** | -**The answer's first token is at 36 seconds.** FR-005 asks for two. Streaming was the -plan's answer to a slow completion, and it does not help with this, because nothing -of the answer exists until four sequential preprocessing calls and a retrieval have -finished. A panel would show an empty box for half a minute and then fill quickly. +**On this run the answer's first token was at 36 seconds**, where FR-005 then asked +for two. Streaming was the plan's answer to a slow completion, and it does not help +with this, because nothing of the answer exists until preprocessing and a retrieval +have finished. -This reframes the latency work, which the spec had aimed at retrieval: +This reframed the latency work, which the spec had aimed at retrieval: -- **four preprocessing calls run in sequence before retrieval starts**, costing about - 16s between them for 36 tokens of output in total. Whether they must be sequential, - or must all run for a search-page question, is now the biggest single question +- **four preprocessing calls ran in sequence before retrieval started**, about 16s + between them on this run for 36 tokens of output. They were moved to two rounds on + 2026-09-18 (T019a); whether all four are needed was examined in T020c below - **query expansion** adds 80 tokens and its own call, then multiplies retrieval -- **retrieval** is what spec 009 addresses, and is no longer the obvious first target + (measured in T020 below) +- **retrieval** was what spec 009 addressed; on 2026-09-20 it was again the largest + block (T022 below) A first-token budget cannot be met by making the answer stream. It needs fewer things to happen before the answer starts. @@ -115,6 +147,11 @@ would render into the panel as though it were an answer. retrieval; the answer runs after it. So tokens from `node == "model"` count as answer tokens only once a retriever has completed. +*Built as decided (`src/agent/graph.py:534-583`). One exception added 2026-09-18 +(d3740aa): a question routed to the live MCP lookup has no retriever, so the boundary +opens when preprocessing ends instead, provided live tools are available +(`src/agent/graph.py:500-532`).* + **Why not "the last model run"**: it is only knowable in hindsight, and a panel cannot buffer until the end -- that would discard the streaming this is for. @@ -127,6 +164,15 @@ against nothing, and is otherwise idle on this. p50 15.2s is too slow to ship to users and entirely good enough to integrate against. The latency work is real and separate, and spec 009 plus query-expansion reduction are its two known levers. +The baseline, measured 2026-09-17 across the 15 tracked sweep questions (T002): +whole answer p50 **15.2s**, p90 **22.4s**, min 6.7s, max 31.5s, with no first-token +concept -- the answer arrived whole or not at all. (Moved from quickstart.md, +2026-10-21.) + +Why a separate HTTP endpoint rather than Chainlit (moved from plan.md, 2026-10-21): +the search page cannot use a websocket chat UI, and Chainlit's socket protocol is not +a documented API another repository should couple to. + ## Query expansion, measured 2026-09-20 (T020) The task said "reduce query expansion from 5 variants". The premise was @@ -167,8 +213,10 @@ Thirteen tracked questions establish that *those* answers do not need expansion. They do not establish that recall is unaffected in general, and expansion exists for the questions nobody wrote a test for. So this ships as `QUERY_EXPANSION_ALTERNATES` with the default at 4 -- a switch and a -measurement, not a verdict. Trying 0 on beta, where the sweep and the routing -probe both run on every deploy, is the cheap way to learn more. +measurement, not a verdict. The next step suggested then was to try 0 on beta, where +the sweep and the routing probe both run on every deploy; this repository has no +record of that being tried, and the default is still 4 +(`src/retrievers/csv_chroma.py:85-86`). ## T022 -- is 2s/10s reachable? Measured 2026-09-20 @@ -189,11 +237,11 @@ recorded so "expansion off" is shown rather than assumed. n=25 per arm. ### The plain answer -**10s complete is met**, at 8.90s p50 today and 7.16s with expansion off. +**10s complete was met**, at 8.90s p50 on 2026-09-20 and 7.16s with expansion off. -**2s to first token is not met**, either way. Disabling query expansion gets +**2s to first token was not met**, either way. Disabling query expansion got p50 to 2.73s -- short of the target but the same order as it rather than -double. So FR-005a stays a target with a named blocker. +double. So FR-005a stayed a target with a named blocker. ### The tail is the finding, not the median @@ -203,8 +251,9 @@ worst (5.59s). So expansion does not make a typical request slow; it makes the slow requests much slower. A target quoted at p50 would hide that entirely, and the panel -renders progressively, so the tail is the part a reader actually notices. Any -future latency requirement should be stated at a high percentile. +renders progressively, so the tail is the part a reader actually notices. The +recommendation recorded then: state any future latency requirement at a high +percentile. Reported as min–max rather than p90: at n=25 a p90 is little more than the second-highest sample, and the first attempt quoted one from n=15, where it @@ -222,7 +271,8 @@ also stable across both arms (0.84s and 0.79s) where the arms differ hugely in retrieval, which is what a real per-call cost looks like rather than a measurement artefact. -They disagree by a factor of seven and only one can describe today's code. +They disagreed by a factor of seven and only one could describe the code of the day. +No later measurement in this folder reconciles them. Stated as unreconciled rather than quietly replacing the old number, because "the answer model is the problem" has been steering this spec's priorities. @@ -234,10 +284,10 @@ and the internal check for it is `preprocess`: it is 1.58s and 1.65s across arms that differ tenfold in retrieval, and expansion cannot affect preprocessing, so a sequencing confound would have shown up there and did not. -### What would have to change, in measured order +### What would have had to change, in measured order (2026-09-20) 1. **Retrieval, 3.15s** -- the largest block, and 2.86s of it is query - expansion. `QUERY_EXPANSION_ALTERNATES=0` removes it today and is one + expansion. `QUERY_EXPANSION_ALTERNATES=0` removes it and is one variable to revert. Its recall cost is unmeasured beyond thirteen tracked questions (T020). 2. **Preprocessing, 1.58s** -- whether a search-page question needs all four @@ -310,7 +360,8 @@ Bypassing the call does not remove the round. The saving needs the rounds **restructured**: with no rephrasing, `rephrased_input` is just `user_input`, so safety and intent no longer depend on round one and all three calls can run together -- 0.93s instead of 1.74s. That is the ~0.8s, and it requires a code -change this measurement did not make. +change this measurement did not make. *(Not made as of 2026-10-21: preprocessing +still runs two rounds, `src/agent/profiles/react_to_me.py:143-172`.)* ### So, for FR-005a diff --git a/specs/010-search-page-answers/spec.md b/specs/010-search-page-answers/spec.md index 2ea293f9..a648f14c 100644 --- a/specs/010-search-page-answers/spec.md +++ b/specs/010-search-page-answers/spec.md @@ -4,21 +4,29 @@ **Created**: 2026-09-17 -**Status**: Draft. This repo's half only. Three decisions (D1-D3), and one -measured constraint that shapes all of them. +**Status**: Historical record of the search-page answer endpoint work (this repo's +half only), 2026-09-17 to 2026-09-20. The endpoint, streaming, caller-token +verification, the backstop rate limit and release attribution were implemented +2026-09-17 to 2026-09-18 (212c80e, 39b73be; PRs #236, #237, #240); `answer_id` was +added 2026-09-25 by spec 013. Not built: the latency target (FR-005a, first token +not met), choosing which searches get an answer (D2, User Story 3) and caching +(D3); whether these are still wanted is not recorded. The requirements and +decisions below were set for that work, not for later changes. Current endpoint +behaviour is in [contracts/answer_endpoint.md](./contracts/answer_endpoint.md) and +`src/api/answer.py`. **Input**: *"a feature we want is for the chatbot responses to be integrated into the website search page results in a google like style ... this would have to be a coordinated effort with the website repo ... the functionality should only be available to proven to be human users."* -## The constraint that shapes everything +## The latency constraint this work was designed around (2026-09-17 to 2026-09-18) -Measured on 2026-09-17, beta, Release97 bundle, `gpt-4o-mini`, warm process: - -Across all fifteen tracked questions, not one question twice: - -Re-measured 2026-09-18, two runs of each tracked question: +The first measurement, 2026-09-17, was on beta, Release97 bundle, `gpt-4o-mini`, +warm process; its figures were later replaced. The table is the 2026-09-18 +re-measurement, two runs of each tracked question. The latest recorded measurement +is 2026-09-20: first token p50 5.57s, complete p50 8.90s (T022 in +[research.md](./research.md)). | | | |---|---| @@ -31,39 +39,46 @@ Re-measured 2026-09-18, two runs of each tracked question: *This table has been wrong twice. It first quoted "25.4s and 32.2s", which was two runs of one question near the maximum. It then quoted p50 15.2s for the whole answer -and ~36s to the first token; neither reproduces now, because collection routing -(spec 009) narrowed retrieval and preprocessing went from four sequential calls to -two rounds. Re-measure after any retrieval or preprocessing change rather than +and ~36s to the first token; neither reproduces now. Preprocessing went from four +sequential calls to two rounds; the rest of the difference is not fully explained +(see the contract). Collection routing (spec 009) had not landed when this was +written. Re-measure after any retrieval or preprocessing change rather than quoting these.* A Google AI overview is generally reported to arrive in one to two seconds -- an assumption here, not a measurement of ours. At a p50 of just under ten seconds to the -first token, a panel on the search results page is still empty long after a person +first token (2026-09-18; 5.57s on 2026-09-20, T022 in research.md), a panel on the +search results page is still empty long after a person has read the ordinary results. The gap has closed by more than half since this spec was written and the conclusion still holds, which is the point worth keeping: the requirement was never about a specific number. -This is not a polish problem. It decides whether the feature works, so it is stated -here as a requirement rather than discovered during implementation. +This was treated as deciding whether the feature works, not as polish, so it was +stated up front rather than discovered during implementation. -Two consequences run through the rest of this spec. +Two consequences ran through the rest of this spec. -The answer **must stream**, so something appears in the first second or two. +The answer was made to **stream** (FR-002), on the expectation that something would +appear in the first second or two. It did not: the first answer token cannot arrive +before preprocessing and retrieval finish (next section). -And retrieval is the largest single component for the questions that matter here -- -but **collection routing is not the only lever on it, and may not be the biggest**. +And retrieval was then the largest single component for the questions that matter +here -- but **collection routing was not the only lever on it, and might not be the +biggest**. Retrieval cost scales with *queries x collections*. Query expansion turns one question into **five** queries at a cost of 2.4s, and every one of them is run against every collection. Cutting five collections to two saves about as much as -cutting five queries to two, and only the first has a spec. Measured 2026-09-17: +cutting five queries to two, and at the time only the first had a spec (the query +count became configurable on 2026-09-20, T020). Measured 2026-09-17: | | | |---|---| | 5 queries x 5 collections, served async path | 12.5s | | 1 query x 5 collections, served async path | ~1.5s | -[Spec 009](../009-collection-routing/spec.md) remains worth doing. The claim that it -alone is on the critical path does not survive the measurement. +When this was written (2026-09-17), [spec 009](../009-collection-routing/spec.md) +had not been built; it landed 2026-09-19. The claim that it alone was on the +critical path did not survive this measurement. ## The latency target, and why it is not a requirement yet @@ -82,133 +97,150 @@ Measured 2026-09-17 by streaming `astream_events` and grouping by `run_id`: **Nothing of the answer exists until preprocessing and a retrieval have finished**, but that is now under five seconds rather than the twenty this spec previously -recorded. The largest remaining block is the answer model's own time to first token -against a retrieved context, which streaming does not help and nothing here -addresses. +recorded. On these 2026-09-17 figures the largest remaining block was the answer +model's own time to first token against a retrieved context. The 2026-09-20 +measurement (T022 in research.md) puts that at 0.84s and retrieval at 3.15s +instead; the two have not been reconciled. A naive measurement reports 3.0s, because the first streamed token of any kind belongs to the rephraser. That number is wrong in the most flattering direction, and is the one a casual check would have produced. -**So the first increment makes no latency promise.** It is correct, verified and -streaming, at roughly today's speed. The website repo can build the panel, the token -handshake and every failure path against it, none of which depend on how fast it is. +**So the first increment made no latency promise**, and none has been made since. It +was correct, verified and streaming, at the speed of the day, so the website repo +could build the panel, the token handshake and every failure path against it, none +of which depend on how fast it is. -What would have to change for FR-005a, in the order the measurement suggests: +What would have had to change for FR-005a, in the order the 2026-09-17 measurement +suggested. The 2026-09-20 measurement (T022 in research.md) reorders it: retrieval +first at 3.15s, then preprocessing at 1.58s, with the answer model at 0.84s, a +figure not reconciled with the 6.1s below. 1. **The answer model's time to first token**, 6.1s against a retrieved context and the largest block left. Nothing in these specs addresses it; a smaller or faster model, or a shorter context, are the obvious handles 2. **Preprocessing**, now 2.6s in two rounds. Whether a search-page question needs all four calls at all is still open -- the rounds only removed the waiting -3. **Query expansion and retrieval**, 2.3s together, after - [spec 009](../009-collection-routing/spec.md) narrowed the collections - -Stating a budget the code cannot meet would bake it into a contract another repo -builds against. This is what we are doing instead. - -## User Scenarios & Testing +3. **Query expansion and retrieval**, 2.3s together, measured before + [spec 009](../009-collection-routing/spec.md)'s collection routing landed, with + every collection searched -### User Story 1 - A verified person searches and sees an answer forming (Priority: P1) +Stating a budget the code could not meet would have baked it into a contract another +repo builds against, which is why FR-005 makes no latency promise and FR-005a is a +target. -Someone searches reactome.org. Above the ordinary results, an answer begins appearing -within about two seconds and completes in under ten, with citations into Reactome. +## User scenarios as written 2026-09-17, and where they stand (checked 2026-10-21) -**Why P1**: it is the feature. Everything else is a refinement of it. +### User Story 1 - A visitor searches and sees an answer forming (Priority: P1) -**Independent test**: post a question to the answer endpoint with a valid human -token; assert the first token arrives inside the budget and citations resolve to real -stable IDs. +As written: someone searches reactome.org and, above the ordinary results, an answer +begins appearing within about two seconds and completes in under ten, with citations +into Reactome. The timing was the FR-005a target; the first-token half was not met +(T022 in research.md). -**Acceptance** -1. Tokens stream: more than one token event, so the caller can render progressively +**Acceptance as written, and status** +1. Tokens stream: more than one token event. Holds (`src/api/answer.py:166-171`; T006) 2. The first token belongs to the **answer**, not to the rephraser or the query - expander -- they stream too, and rendering them would show a search panel an - expanded query as though it were an answer -3. Every factual claim carries a citation resolvable at `reactome.org/content/detail/` -4. A question the knowledgebase cannot answer says so, rather than inventing - -### User Story 2 - Someone who has not been verified gets no answer (Priority: P1) - -An unverified visitor, or a script, gets the ordinary search results and no AI panel. -No LLM call is made. - -**Why P1 and not P2**: this is a cost and abuse control, not a feature toggle. Every -search reaching the model is an unbounded bill, and search pages are crawled. - -**Independent test**: post without a token, and with a forged one; assert HTTP 401/403 -and that no LLM call was made. - -**Acceptance** -1. No valid human token → refused before any model call -2. A token is bound to the person, expires, and cannot be replayed from elsewhere -3. Refusal is cheap and does not consume a rate-limit slot for real users + expander. Holds: answer tokens are those from the `model` node after a retriever + completes, or after preprocessing on the live-lookup path + (`src/agent/graph.py:500-583`; research R5c) +3. "Every factual claim carries a citation resolvable at + `reactome.org/content/detail/`." Not as built: citations are separate events + listing what retrieval found, not per claim (research R3); a user-guide page is + cited by `url` (T028); a live-lookup answer has none; and links in the prose are + stripped (contract, "What `token` text contains") +4. A question the knowledgebase cannot answer says so rather than inventing. Asked of + the model by the answer prompt (`src/retrievers/reactome/prompt.py:17`); such an + answer ends `answered`, not `nothing_found` + +### User Story 2 - A caller without a valid token gets no answer (Priority: P1) + +Written as "someone who has not been verified gets no answer", expecting HTTP +401/403. D1 (2026-09-18) changed the premise: the search path has no human check, so +what is verified is a token the website mints for each call, not a person. + +**Why P1**, as written: a cost and abuse control, not a feature toggle. Every search +reaching the model is a bill, and search pages are crawled. + +**As built**: a missing, invalid, expired or wrong-audience token, or a caller over +the backstop limit, gets HTTP 200 with a single `done` event, `state: refused`, and no +model call (`src/api/answer.py:79-113`; `tests/api/test_answer_endpoint.py:153-176`). +Not 401/403: a refusal is a well-formed stream so the website handles one shape +(`src/api/answer.py:79-85`). + +**Acceptance as written, and status** +1. No valid token → refused before any model call. Holds +2. "A token is bound to the person, expires, and cannot be replayed from elsewhere." + Partly: tokens expire (`exp` is required; the website mints at +120s) and must + carry this service's audience, so one minted for another consumer is refused. They + are not bound to a person, and a token can be presented again until it expires: + nothing records use (`src/util/caller_token.py:34-35`) +3. Refusal is cheap and does not consume a rate-limit slot. Holds: token refusals + happen before the limiter, and a rate-limited request is not counted + (`src/util/rate_limit.py:66-76`) ### User Story 3 - Not every search gets an answer (Priority: P2) -A search for a single gene name, or a navigational query, returns ordinary results -with no AI panel. Only questions that an answer would actually serve get one. - -**Why P2**: the feature works without it, but the cost does not. +As written: a search for a single gene name, or a navigational query, returns ordinary +results with no AI panel; only questions an answer would serve get one, tested +against queries labelled should-answer / should-not. -**Independent test**: a set of queries labelled should-answer / should-not; assert the -classifier's decision matches, and that no LLM answer call happens for the latter. +**Not built.** The endpoint answers every request that passes the token and rate +checks; nothing classifies a search as not worth answering (D2). Whether this is +still wanted is not recorded. -### Edge Cases +### Edge cases as written, and status -- **Retrieval finds nothing**: say so; never fill the gap with general knowledge +- **Retrieval finds nothing**: say so; never fill the gap with general knowledge. + Asked of the model by the prompt (`src/retrievers/reactome/prompt.py:14-17`) - **The model is slow or the upstream is down**: the panel disappears rather than - hanging; the search page must not depend on this service being up -- **The person navigates away mid-stream**: the request is cancelled, not left running -- **The same question twice**: served from cache, not re-answered (see D3) -- **A question that is unsafe or off-topic**: the existing safety checker already - refuses these, and its refusal must not render as an AI panel - -## Requirements - -### Functional Requirements - -- **FR-001**: The service MUST expose an HTTP endpoint accepting a question and - returning an answer with citations. No such endpoint exists today -- `chat-fastapi.py` - serves only the captcha pages and a landing page, and Chainlit owns the conversation - over websockets -- **FR-002**: The response MUST stream, so partial text can render before completion -- **FR-003**: The endpoint MUST refuse any request without a valid caller - token, before any model call -- **FR-004**: Citations MUST be Reactome stable IDs, so the website can render links - in its own style rather than parsing prose -- **FR-005**: The caller MUST be able to render the answer progressively as it - arrives. **No latency guarantee is made by the first increment**, and the reason is - measured rather than assumed -- see "The latency target, and why it is not a - requirement yet" below -- **FR-005a** *(target, not met)*: first token within 2s and complete within 10s at - p50. Tracked as a goal with a named blocker, not promised to the website repo -- **FR-006**: The service MUST fail invisibly: any error, timeout or refusal returns - a response the website can render as "no panel", never a broken panel -- **FR-007**: Answers MUST be attributable to a Reactome release, so a cached or - stale answer can be identified after a release -- **FR-008**: The endpoint MUST be rate limited per verified person, independently of - the chat UI's existing limits + hanging. This service ends the stream with `done` `failed` after 120s or on an error + (`src/api/answer.py:67`, `:219-233`); what the page shows is the website's +- **The person navigates away mid-stream**: the request is cancelled. Holds, measured + 2026-09-18 (T027; `src/api/answer.py:192-218`) +- **The same question twice**: was to be served from cache (D3). Not built: every + request is answered afresh +- **A question that is unsafe or off-topic**: the safety checker refuses it, and the + refusal must not render as a panel. Holds: the refusal comes from a different graph + node and is not streamed, so the stream ends `nothing_found` with no tokens + (`src/agent/profiles/react_to_me.py:101-110`; `src/agent/graph.py:580-583`) + +## Requirements of this work, and where they stand (checked 2026-10-21) + +| ID | Requirement as written | Status | +|---|---|---| +| FR-001 | An HTTP endpoint accepting a question and returning an answer with citations (none existed; Chainlit owned the conversation over websockets) | Holds: `POST /api/answer` (`src/api/answer.py:96`, `bin/chat-fastapi.py:73-78`) | +| FR-002 | The response streams | Holds: server-sent events (`src/api/answer.py:129-264`) | +| FR-003 | Refuse any request without a valid caller token, before any model call | Holds (`src/api/answer.py:98-115`) | +| FR-004 | Citations are Reactome stable IDs, so the website styles links itself | Superseded 2026-09-18 (T028): a citation carries `st_id` or, for a user-guide page, `url` (`src/agent/graph.py:219-261`). Live-lookup answers carry none | +| FR-005 | The caller can render progressively; no latency guarantee in the first increment (see "The latency target, and why it is not a requirement yet") | Holds; no latency guarantee has been made since | +| FR-005a | *Target, not a requirement*: first token within 2s and complete within 10s, at p50 | Last measured 2026-09-20 (T022): complete 8.90s p50, met; first token 5.57s p50, not met. Not re-measured since | +| FR-006 | Fail invisibly: any error, timeout or refusal gives a response the website can render as "no panel" | Holds within the stream: errors, the 120s timeout and refusals end with a non-`answered` `done` (`src/api/answer.py:59-67`, `:79-93`, `:219-233`). Two limits: tokens sent before a failure stay sent, and a malformed request body gets HTTP 422 JSON, not a stream | +| FR-007 | Answers attributable to a Reactome release, so a cached answer can be identified after one | Holds: `release` on `start`, read at startup (`bin/chat-fastapi.py:55-59`); `null` when unknown. No cache uses it (D3) | +| FR-008 | Rate limited per verified person, independently of the chat UI | Built as a per-caller backstop: 30 requests per 600s by default, keyed on the token's `sub`, else `jti`, else a hash of the token (`src/util/rate_limit.py:30-49`, `:94-103`). It counts callers, not verified people (D1) | ### Key Entities -- **Question**: the search string, plus the release it was asked against -- **Answer**: streamed text, a list of citations (stable ID and display name), and a - completion state -- answered, nothing-found, refused, or failed -- **Human token**: evidence that a person was verified, bound to them, time limited. - Turnstile already does this in `chat-fastapi.py`; what is missing is a form the - website can obtain and present +- **Question**: the search string. The release is the one the service started with, + not something the caller sends +- **Answer**: streamed text, a list of citations (a stable ID or a URL, and a display + name), and a completion state -- `answered`, `nothing_found`, `refused` or `failed` +- **Caller token** (called "human token" until 2026-09-18): a short-lived token the + website mints for each call. It asserts the call came from the website, not that a + person was verified (D1). This service's Turnstile check guards the chat UI, not + this endpoint -## Success Criteria - -### Measurable Outcomes +## Success criteria, and where they stand (checked 2026-10-21) - **SC-001**: the endpoint streams -- more than one token event for a question the knowledgebase can answer -- and its first-token and completion times are measured and published as a distribution over the tracked question set. A number going down - is the goal; a number being met is not, until FR-005a's blocker is removed + is the goal; a number being met is not, until FR-005a's blocker is removed. + *Done 2026-09-18 (T019) and re-measured 2026-09-20 (T022, research.md)* - **SC-002**: Zero model calls for requests without a valid token, measured by - counting calls under a load of unauthenticated requests + counting calls under a load of unauthenticated requests. *Pinned by tests that + assert a patched graph is never called (`tests/api/test_answer_endpoint.py:153-176`, + T016); no load measurement is recorded* - **SC-003**: The answer sweep stays green, and the two surfaces stay *configured* the same -- same profile, same shared graph, no difference that can reach the answer. Textual equality is explicitly **not** the criterion, because @@ -216,11 +248,15 @@ classifier's decision matches, and that no LLM answer call happens for the latte same question asked twice through the *same* surface produced answers 0.331 similar, while endpoint-versus-chat scored 0.356. The surfaces differ from each other no more than either differs from itself. The sweep is the right mechanism - precisely because it matches patterns rather than literals + precisely because it matches patterns rather than literals. *Pinned by + `tests/api/test_answer_matches_chat.py` (T014): one profile, one shared graph. The + one configured difference, the endpoint skipping the post-answer web search + (`src/api/answer.py:158-164`), cannot change the answer text* - **SC-004**: No search-page request can make the search page itself slower or fail; - verified by taking the service down and confirming the page still renders + verified by taking the service down and confirming the page still renders. *The + website's side; this repository has no record of that check* -## Known: retrieval is not reproducible +## Known: retrieval is not reproducible (measured 2026-09-18, not re-measured) Measured 2026-09-18, the same question asked three times returned **12 citations each time but only 4 pathways common to all three**, a union of 19 and a Jaccard @@ -230,11 +266,13 @@ expands the question differently and retrieves different documents. This matters beyond wording. A reader who reloads the panel sees different sources, and FR-007's cache invalidation assumes an answer is a stable artifact of a release. Deciding what to do about it -- seeding or caching the expansion, -or dropping it for this path -- is open, and is not a blocker for the handover. +or dropping it for this path -- was left open (T020a), and no decision is recorded. +"Continue in chat" keeps the answer as it was shown rather than regenerating it, for +this reason (spec 013, `src/api/answer_store.py:3-6`). ## Decisions -### D1 -- what does the caller present? **Decided 2026-09-18** +### D1 -- what does the caller present? (recorded as decided with the website side, 2026-09-18; implemented in 39b73be) This was "what proves a person is human?", and the question had a false premise. The website session checked before answering: there is **no human gate on the @@ -245,58 +283,61 @@ spent on submit. Every search is anonymous and ungated by design. So the token cannot honestly assert humanity, and asking it to would have meant inventing a claim. **It asserts caller identity instead**: minted server-side by the website, per request, EdDSA, with `iss`, `aud: reactome-chatbot`, `iat`, -`exp` at +120s, and `sub` -- an opaque per-visit id of 128 random bits, not -derived from anything about the reader, held in an HttpOnly cookie. No address, -no IP hash; neither side holds personal data. +`exp` at +120s, and `sub` -- described then as an opaque per-visit id of 128 random +bits, not derived from anything about the reader, held in an HttpOnly cookie. No +address, no IP hash; neither side was to hold personal data. *(Corrected 2026-09-19 +in `src/util/caller_token.py:15-22`: for a reader who has passed the website's +Turnstile challenge, `sub` is that identity cookie's subject and lasts as long as the +cookie; the per-visit id is the fallback for readers who have not.)* This service verifies signature, expiry and **audience**, and keys its backstop -limit on `sub`. The request field is `caller_token`, and the misleading -`caller_token` name is gone from the code as well as this document. - -Abuse control moved with the premise. The panel is opt-in behind a click (D5), so -a crawled search never reaches a model, and the website's proxy rate limits by -address. That is a real deterrent rather than a claim we would be inventing. - -Turnstile stays where it is, guarding the chat UI. It was never part of this -path, and the two must not be "unified". +limit on `sub`. The request field was renamed from `human_token` to `caller_token` +in the code and in this folder (PR #240, finished in 6d517a0). -### D2 -- which searches get an answer? +Abuse control moved with the premise, to the website. As the website side described +it on 2026-09-18, the panel is opt-in behind a click, so a crawled search never +reaches a model, and its proxy rate limits by address. Neither is visible from this +repository. -Not all of them, per User Story 3. The intent classifier already decides what kind -of question it is, and [spec 007](../007-answer-cascade/spec.md) is about that -routing. Reusing it costs nothing extra; inventing a second classifier costs a call. +The decision kept Turnstile where it was, guarding the chat UI only. It was never +part of this path, and the decision was not to merge the two checks. -**Recommendation**: extend the existing classifier rather than add one. **Open** -pending 007. +### D2 -- which searches get an answer? (open; not built) -### D3 -- caching +Not all of them, per User Story 3. The recommendation recorded on 2026-09-17 was to +extend the existing intent classifier rather than add a second one, which would cost +a call, and the question was left open pending +[spec 007](../007-answer-cascade/spec.md) (which does not itself mention the search +page). No decision is recorded. The endpoint answers every request that passes the +token and rate checks. -Search traffic repeats in a way chat traffic does not. A cache keyed by question and -release would cut both latency and cost, and FR-007 exists to make invalidation -possible. +### D3 -- caching (open; not built) -**Open.** It needs a measurement of repeat rate on real search traffic, which this -repo does not have -- the website does. +Search traffic was expected to repeat in a way chat traffic does not, so a cache +keyed by question and release would cut both latency and cost; FR-007 exists to make +invalidation possible. It was left open for want of a repeat-rate measurement on real +search traffic, which the website has and this repository does not. No decision is +recorded and no cache exists. Retrieval not being reproducible (above) bears on what +such a cache could store. -## Scope +## Scope of this work (as set 2026-09-17) **In (this repo)**: the answer endpoint, streaming, citation shape, token -verification, rate limiting, release attribution, and the latency work needed to -meet FR-005. +verification, rate limiting, release attribution, and latency work towards FR-005a. **Out (the website repo)**: the panel, its visual design, where it sits on the page, -how citations render, and the UX of the chat's deeper integration. This spec commits -to a contract, not a look. +how citations render, and the UX of the chat's deeper integration. This spec +committed to a contract, not a look. -**Out (both, for now)**: the broader "chat integrated into the site's flow and feel" -work. That is a design effort that deserves its own spec once this contract exists, -and it is mostly not this repo's code. +**Out (both, at the time)**: the broader "chat integrated into the site's flow and +feel" work, left for its own spec once this contract existed. -## Assumptions +## Assumptions made 2026-09-17, and how they held -- The website can verify a human before calling this service; this repo verifies the - evidence rather than performing the check -- Beta is the integration target first. The MCP already points at - `beta.reactome.org` as of 2026-09-17 -- The same graph serves the endpoint and the chat UI; two answer paths that could - disagree would be a defect, not a feature +- The website can verify a human before calling this service. Did not hold: the + search path has no human check, and the token asserts caller identity instead (D1) +- Beta is the integration target first; the MCP pointed at `beta.reactome.org` as of + 2026-09-17. Beta serves the endpoint at `/chat/guest/api/answer` + (`deploy/beta/env.beta.template:21`, `deploy/beta/README.md` section 4) +- The same graph serves the endpoint and the chat UI. Holds: one shared instance + (`src/agent/registry.py`); SC-003 says what "the same" can and cannot mean diff --git a/specs/010-search-page-answers/tasks.md b/specs/010-search-page-answers/tasks.md index 6be6bf08..abd93445 100644 --- a/specs/010-search-page-answers/tasks.md +++ b/specs/010-search-page-answers/tasks.md @@ -1,90 +1,37 @@ -# Tasks: Chatbot Answers in the Website Search Results - -**Feature**: 010-search-page-answers | **Plan**: [plan.md](./plan.md) | **Spec**: [spec.md](./spec.md) - -Test tasks included: this adds a public surface with an authentication boundary, and -Principle III wants current behaviour pinned before a route is added beside the -captcha middleware. - -**MVP is Phase 3.** An endpoint that answers correctly and slowly unblocks the -website repo entirely. Speed is Phase 5 and does not gate the handover. - -## Phase 1: Setup - -- [x] T001 Create branch `010-search-page-answers`; confirm `.specify/feature.json` points at specs/010-search-page-answers -- [x] T002 [P] Record today's baseline in specs/010-search-page-answers/quickstart.md: p50 15.2s, p90 22.4s over the 15 sweep questions, so Phase 5 has a before - -## Phase 2: Foundational (blocks the endpoint) - -- [x] T003 Pin the captcha middleware's current behaviour — done in tests/util/test_captcha_scope.py (6 tests) rather than the tests/api/test_middleware_scope.py named here, because the decision moved into src/util/captcha_scope.py where it can be tested without an app -- [x] T004 Give `AgentGraph` one home the FastAPI app can reach, in bin/chat-fastapi.py via lifespan, so the endpoint and Chainlit share one instance (SC-003) rather than paying 51.5s of startup twice -- [x] T005 Add a streaming surface to src/agent/graph.py using the compiled graph's `astream_events`, yielding answer tokens and retrieved documents -- [x] T006 Test that T005 yields more than one token for a real question — not that it returns 200. A surface that completes without streaming is the failure this must catch - -## Phase 3: User Story 1 — a verified person sees an answer forming (P1) - -**Independent test**: post a question with a valid token; assert the first event -arrives, tokens stream, citations resolve to real stable IDs, and `done` carries a -state. - -- [x] T007 [P] [US1] Add src/util/caller_token.py: verify signature and expiry only, stateless, no consumption tracking -- [x] T008 [P] [US1] Test caller_token in tests/util/test_caller_token.py: valid, expired, wrong key, tampered payload, absent — every failure refuses -- [x] T009 [US1] Refuse at startup in src/util/caller_token.py when the verifying key is missing or unreadable, rather than accepting everything (Principle IV) -- [x] T010 [US1] Add the SSE endpoint in src/api/answer.py implementing contracts/answer_endpoint.md: start, token, citation, done -- [x] T011 [US1] Emit citations from retrieved documents' `st_id` metadata, deduplicated — never by parsing anchors out of the model's prose -- [x] T012 [US1] Mount the router in bin/chat-fastapi.py and let the captcha middleware pass /chat/api/ through, since the endpoint verifies its own caller -- [x] T013 [US1] Test over HTTP with a real client in tests/api/test_answer_endpoint.py, not by calling the handler — mounting order and middleware only interact in the served path (Principle I) -- [x] T014 [US1] SC-003, restated against the measurement: pin that the two surfaces are *configured* the same in tests/api/test_answer_matches_chat.py. Answer equality is not assertable -- the same surface asked twice scores 0.331 similarity, endpoint-vs-chat 0.356 -- so the spec's criterion was corrected rather than the test weakened (PR #237) -- [x] T014a [US1] Give each request its own checkpointer thread in src/api/answer.py; `id(body)` put 192 of 200 requests on a shared thread, and `chat_history` is checkpointed state the rephraser reads (PR #236) -- [x] T014b [US1] Send `release` on start and `seconds` on done per contracts/answer_endpoint.md; both were promised to the website and neither was implemented (PR #236) - -## Phase 4: User Story 2 — no answer without a verified person (P1) - -- [x] T015 [US2] Refuse missing, expired, malformed and wrongly-signed tokens before any model call, in src/api/answer.py -- [x] T016 [US2] Test that no model call happens for a refused request in tests/api/test_answer_endpoint.py, by asserting on a patched graph rather than on timing (SC-002) -- [x] T017 [P] [US2] Rate limit per token as a backstop in src/util/rate_limit.py; the budget is the website's, enforced before the call reaches here (FR-008). 30 per 10 minutes, keyed on `sub`/`jti` when D1 provides one and a token hash until then (PR #237) -- [x] T017a [US2] Stop paying for a discarded web search: the endpoint took `enable_postprocess` at its default, so every answer ran a Tavily search that `astream_answer` has no event to return (PR #237) -- [x] T025 [US2] Enforce `aud` on the caller token (asked for by the website, D1); the code refused every token carrying one, since PyJWT rejects `aud` when no audience is expected (PR #240) -- [x] T026 Rename human_token -> caller_token everywhere; D1 established the token asserts caller identity, not humanity (PR #240) -- [x] T027 Answer the website's cancellation question: a client hang-up raises CancelledError inside the answer generator and produces nothing further, so they need not cancel upstream (PR #240) -- [x] T028 Cite userguide answers by `url` (Adam decided yes, 2026-09-18). Optional `url` sibling to `st_id`, exactly one present; no fabricated stable ids; http(s) only, after a local filesystem path leaked through the first version. -- [ ] T020a Decide what to do about non-reproducible retrieval: three runs of one question shared only 4 of 19 citations (Jaccard 0.26) because query expansion is itself a model call. Affects what FR-007 can cache -- [x] T018 [US2] Return `state: failed` with no partial answer on any internal error, so the page renders no panel (FR-006) -- [x] T011a [US1] Strip inline HTML anchors from the token stream in src/util/anchor_strip.py; the contract promises prose without them and the chat prompt emits them, split across ~20 fragments (PR #236) -- [x] T013a [US1] Run the endpoint end to end against a real graph: release 97, answered in 19.5-44.2s, 12 citations, anchors 0 (PR #236) -- [x] T018a [US2] Bound the answer at 120s in src/api/answer.py; FR-006 names timeout and only the LLM client's 360s-per-call limit existed, so a stuck upstream could hold a connection for over half an hour (PR #236) - -## Phase 5: Latency (does NOT gate the handover) - -- [x] T019 Measure first-token and completion separately across the tracked questions; publish the distribution, not one question. Measured 2026-09-18, two runs each: first token p50 9.6s / p90 12.2s (n=26), completion p50 10.4s / p90 18.1s (n=30). The earlier "36.1s to first token" came from one question and does not reproduce (PR #238) -- [x] T019a Run preprocessing in two rounds instead of four sequential calls in src/agent/profiles/react_to_me.py; the base class already overlapped, and this override discarded it (PR #238) -- [x] T020 Measure query expansion, and make the count configurable. **The task's premise was wrong**: the cost is the expansion *call*, not the number of variants it returns. Measured over six tracked questions -- 4 alternates 1.27s expand + 1.22s retrieve; 2 alternates 1.26s + 0.65s; 0 alternates 0.00s + 0.31s. Trimming the count saves only fan-out; the cost disappears only at zero, where `answer-sweep` was 13/13 in 79s against ~150s. `QUERY_EXPANSION_ALTERNATES` sets it and **the default is unchanged**: thirteen questions show those answers do not need expansion, not that recall is unaffected in general -- [x] T020c Established 2026-09-20. **The rephraser is not a no-op without history** -- it changed 14 of 15 tracked questions, normalising queries rather than resolving follow-ups, so removing it is a retrieval-quality change. Bypassed, the sweep is 13/13 and first token p50 goes 2.94s to 2.83s (interleaved, n=25): about 0.11s. Removing a 0.88s call saved 0.07s because round one is `max(rephrase, language)` and language detection is 0.81s -- it was hiding behind the rephraser. The ~0.8s needs the **rounds restructured** so safety and intent stop waiting, which is a code change this did not make. 2s is not reachable by removing these two calls -- [ ] T021 Land spec 009 collection routing and re-measure. **Still open — I marked this done on 2026-09-18 and was wrong.** What landed is *source* routing (`resolve_active_sources` picks reactome / userguide / live). Collection routing is selecting among the five collections *within* the reactome bundle, and it is not implemented: `QueryIntent` has no `collections` field, `resolve_collections` does not exist, and `retrieve_documents` still loops over every collection -- [x] T022 Re-assessed 2026-09-20 on the served path, arms interleaved, n=25 each, with the query count asserted rather than assumed. **10s complete is met** (8.90s p50, 7.16s with expansion off); **2s to first token is not** (5.57s p50, 2.73s with expansion off), so FR-005a stays a target with a named blocker. The blocker changed: retrieval is the largest block at 3.15s and the answer model is 0.84s against a previously recorded 6.1s, recorded as unreconciled. The tail is the finding -- default first token ranges 2.26s to 11.44s, and expansion barely moves the best case while halving the worst, so a future target belongs at a high percentile -- [x] T023 Propagate a real failure signal out of the live path. `answer_from_live_services` takes an optional `LiveReport` and records a tool exception **before** stringifying it into the model's context -- the last point at which it is still a fact rather than a paraphrase. The graph carries it out as `live_tool_failed`, and `answer_sweep` retries on that instead of matching prose. `TRANSIENT` and `_looks_transient` are gone, with a test asserting they do not come back: if prose matching returns it will return as a list of markers. The report is optional, so none of the twelve existing call sites changed. **The no-tools fallback deliberately does not set it**: `get_mcp_tools` remembers a failed start, so a retry could never succeed and marking it transient would buy an attempt guaranteed to fail. Pinned by a test, because the absence of the flag there reads like an oversight - -## Phase 6: Handover - -- [x] T023 Tell the website session the endpoint exists, with a curl that streams, and what it does not yet do — done 2026-09-18, including the nginx user-agent 403 and the measured latency; they have built against it and a token they minted verified -- [x] T024 [P] Update specs/010-search-page-answers/spec.md status and record the measured first-token and completion times — first token p50 9.6s / p90 12.2s, completion p50 10.4s / p90 18.1s - -## Dependencies - -``` -Phase 1 -> Phase 2 -> Phase 3 -> Phase 4 -> Phase 6 - Phase 5 runs alongside 4 and 6 -``` - -T004 blocks T005; T005 blocks T010; T007 blocks T012; T010 blocks T013, T014, T015. - -## Parallel opportunities - -- T007 and T008 alongside T004-T006: token verification touches nothing the graph does -- T017 and T024: different files - -## Implementation strategy - -Ship Phases 1-4 as one PR: a correct, slow, verified, streaming endpoint. That is the -handover. Phase 5 follows on its own, with its own measurements, and the website repo -is not waiting on it. +# Search-Page Answers: Task-ID Index + +Historical. The step-by-step task list for this work (T001-T028) is in git history +up to 3a4bc94. Kept here are the IDs that code comments and this folder refer to. +The ID T023 was used twice; code that cites T023 means the live-path failure signal. + +| ID | Task | Outcome | +|---|---|---| +| T002 | Record the baseline before building | 2026-09-17: whole answer p50 15.2s, p90 22.4s; in research.md, R6 | +| T003 | Pin the captcha middleware before adding a route beside it | `tests/util/test_captcha_scope.py`; the rule is in `src/util/captcha_scope.py` | +| T004 | One graph shared by Chainlit and the endpoint | `src/agent/registry.py`, built in the FastAPI lifespan | +| T005, T006 | A streaming surface, tested for streaming rather than for a 200 | `AgentGraph.astream_answer` (`src/agent/graph.py:461`) | +| T007-T009, T015, T016 | Caller-token verification, refusal before any model call, no start without a key | `src/util/caller_token.py`, `src/api/answer.py`; tests in `tests/util/test_caller_token.py`, `tests/api/test_answer_endpoint.py`. T007 said "signature and expiry only"; audience and other checks were added later (T025; research R4) | +| T010-T013, T011a, T013a | The SSE endpoint, citations from retrieval metadata, mounting, an HTTP test, link stripping, a first end-to-end run | `src/api/answer.py`. First run 2026-09-17: release 97, answered in 19.5-44.2s, 12 citations, no anchors | +| T014 | SC-003: the two surfaces stay configured the same | `tests/api/test_answer_matches_chat.py` (PR #237). Answer equality was dropped as the criterion: same surface twice 0.331 similarity, endpoint vs chat 0.356 (2026-09-18) | +| T014a, T014b | A checkpointer thread per request; `release` on `start` and `seconds` on `done` | PR #236 | +| T017 | Backstop rate limit | `src/util/rate_limit.py` (PR #237) | +| T017a | Stop running a web search nobody receives | `enable_postprocess=False` (`src/api/answer.py:158-164`, PR #237) | +| T018 | `state: failed` on any internal error (FR-006) | Built. The task said "with no partial answer", but tokens already sent stay sent and the stream then ends `done` `failed` (contract, Known gaps) | +| T018a | Bound an answer at 120s | `ANSWER_TIMEOUT_SECONDS` (`src/api/answer.py:59-67`, PR #236) | +| T019, T019a | Measure the latency distribution; run preprocessing in two rounds | 2026-09-18, PR #238; figures in spec.md | +| T020 | Measure query expansion and make the count configurable | research.md, "Query expansion, measured 2026-09-20"; `QUERY_EXPANSION_ALTERNATES`, default unchanged | +| T020c | Does a search-page question need all four preprocessing calls? | research.md, 2026-09-20 | +| T022 | Is 2s/10s reachable? | research.md, 2026-09-20: complete met, first token not | +| T023 | Carry a live-tool failure out of the graph as a fact, not prose | `LiveReport` → `live_tool_failed`, which `src/evaluation/answer_sweep.py` retries on (81d24c3, 2026-09-20). The no-tools fallback deliberately does not set it (`src/agent/profiles/react_to_me.py:219-236`) | +| T023 (second use) | Hand the endpoint over to the website | Done 2026-09-18; recorded as: they built against it and a token they minted verified | +| T024 | Record the measured times in spec.md | Done 2026-09-18 | +| T025, T026 | Enforce `aud`; rename `human_token` to `caller_token` (D1) | PR #240 (39b73be), finished in 6d517a0 | +| T027 | Does a client hang-up cancel the work? | Yes, measured 2026-09-18 (PR #240) | +| T028 | Cite user-guide pages by `url` | Recorded as approved 2026-09-18; built in 0780086 | + +Not done when the list was retired, and no record of whether they are still wanted: + +- T020a, deciding what to do about non-reproducible retrieval: three runs of one + question shared only 4 of 19 citations (Jaccard 0.26), because query expansion is + itself a model call. It affects what FR-007 could cache. No decision is recorded +- [ ] T021 Land spec 009 collection routing and re-measure. **Routing landed 2026-09-19 (0fe7d88), after this task was written; no re-measurement is recorded.** When written on 2026-09-18, what had landed was *source* routing (`resolve_active_sources` picks reactome / userguide / live), not collection routing. diff --git a/specs/011-summarise-analysis-results/checklists/requirements.md b/specs/011-summarise-analysis-results/checklists/requirements.md deleted file mode 100644 index e56fcf33..00000000 --- a/specs/011-summarise-analysis-results/checklists/requirements.md +++ /dev/null @@ -1,51 +0,0 @@ -# Specification Quality Checklist: Summarise analysis results - -**Purpose**: Validate specification completeness and quality before proceeding to planning -**Created**: 2026-09-18 -**Feature**: [spec.md](../spec.md) - -## Content Quality - -- [x] No implementation details (languages, frameworks, APIs) -- [x] Focused on user value and business needs -- [x] Written for non-technical stakeholders -- [x] All mandatory sections completed - -## Requirement Completeness - -- [x] No [NEEDS CLARIFICATION] markers remain -- [x] Requirements are testable and unambiguous -- [x] Success criteria are measurable -- [x] Success criteria are technology-agnostic (no implementation details) -- [x] All acceptance scenarios are defined -- [x] Edge cases are identified -- [x] Scope is clearly bounded -- [x] Dependencies and assumptions identified - -## Feature Readiness - -- [x] All functional requirements have clear acceptance criteria -- [x] User scenarios cover primary flows -- [x] Feature meets measurable outcomes defined in Success Criteria -- [x] No implementation details leak into specification - -## Notes - -Both clarifications were answered by Adam on 2026-09-18 and are now requirements, -not assumptions: - -- **Privacy** — summarising is **opt-in** (FR-011), the user **chooses what is - shared** with at least one useful option that discloses no identifiers (FR-012), - and a **person must be shown to be present** (FR-013). That last is a stricter - bar than the answer endpoint's caller token, which asserts service identity and - deliberately says nothing about humanity — so this feature cannot ride the - ungated search path. -- **Stability** — "do our best and be transparent" became FR-014 (the same token - yields the same summary, by reuse) and FR-015 (say it is generated, and that - regenerating may differ). Stability comes from storing the summary against a - fixed artefact, not from pretending the generator is deterministic. - -ReactomeGSA remains deferred in Assumptions: separate service, separate result -shape. - -All checklist items pass. Ready for planning. diff --git a/specs/011-summarise-analysis-results/contracts/summary_endpoint.md b/specs/011-summarise-analysis-results/contracts/summary_endpoint.md index 4f82a86c..e6bc6f55 100644 --- a/specs/011-summarise-analysis-results/contracts/summary_endpoint.md +++ b/specs/011-summarise-analysis-results/contracts/summary_endpoint.md @@ -1,8 +1,10 @@ # Contract: the analysis summary endpoint -Not implemented. **Agreed with the website repo on 2026-09-19** -- the request -shape, the refusal shape and how human presence is asserted are settled, and -they have the presence claims built on a branch. Nothing here is open. +**Status**: Describes the endpoint as implemented in `src/api/analysis_summary.py` +(checked 2026-10-21). Built 2026-09-19 to 2026-09-20, live on beta from +2026-09-21. The request shape, the refusal shape and how human presence is +asserted were agreed with the website repo on 2026-09-19. Two points of that +agreement are not implemented; see "Known gaps" below. ## Request @@ -15,15 +17,36 @@ Content-Type: application/json "disclosure": "aggregate" } ``` +`{CHAINLIT_URI}/api` is the router prefix; with `CHAINLIT_URI` unset it is +`/chat/api` (`bin/chat-fastapi.py:77-83`). + **The analysis token is passed straight through, never validated by the -caller.** This service already distinguishes 404, 410 and the undocumented 500 -a malformed token returns (research D2). A second validator on the website +caller.** This service distinguishes 404, 410 and the undocumented 500 a +malformed token returns (research D2). A second validator on the website would be a second thing that believes it knows what a good token looks like, -and two validators disagree eventually. +and two validators disagree eventually. This service checks the token's shape +itself: a token that is not 1-200 of `A-Z a-z 0-9 _ -` followed by up to two +`=` or `%3D` gets `not_found` without any request to the Analysis Service +(`src/analysis/client.py:35-51`, `:127-132`). `disclosure` is `aggregate` or `identifiers`, and is **required** — there is no default, because a default is not a choice. `aggregate` never transmits the user's -identifiers, filenames, sample names or expression column labels. +identifiers, filenames, sample names or expression column labels. It does pass +on up to five of the Analysis Service's own `warnings`, each cut to 200 +characters. They are free text the service composes, and this service cannot +check them for quoted user input (`src/analysis/disclosure.py:65-76`). +`identifiers` additionally fetches up to 50 of the reader's unmatched +identifiers (`GET /token/{token}/notFound`, first page) and sends them to the +model (`src/analysis/client.py:199-237`). + +### Request validation + +`token` must be 1-256 characters and `disclosure` one of the two values above. +A body that fails this, or omits either field, gets **HTTP 422** with FastAPI's +validation error body, not an event stream (`SummaryRequest`, +`src/api/analysis_summary.py:85-89`; `tests/api/test_analysis_summary.py:285-292`). +`caller_token` may be omitted: that is a refusal (`no_caller`) inside an HTTP 200 +stream, as below. ### Human presence @@ -62,32 +85,39 @@ edge would be a boundary neither side can actually be on, tested against a clock finer than the value. Effective precision is one second, and a challenge solved 1800.4s ago presents as 1800 and is accepted. -Both sides enforce it -- -the website refuses to mint past it, this service refuses to accept past it -- -so neither is a single point of failure, and if it is ever changed it is -changed in both places in one change. - -**`human_sub`, not `sub`.** `sub` already means something here: an opaque -*per-visit* id, which `identity_of` uses to key the answer endpoint's backstop -limiter. The cookie subject is also 16 bytes but is per-*browser* and lives as -long as the cookie. Putting it in `sub` would silently change that limiter from -"this visit" to "this browser", on an endpoint neither repo is otherwise -touching. A separate claim also makes it explicit that a persistent -pseudonymous identifier is being held -- hashed, in memory only. - -**And the model of `sub` above was itself wrong.** The website corrected it on -2026-09-19: their `callerSubject()` prefers the identity cookie's subject -whenever the reader has passed a challenge, falling back to the per-visit value -only when they have not. So `sub` has *already* been browser-scoped for every -gated reader, and the answer endpoint's backstop limiter has been counting -across visits since the gate shipped. That is the stronger throttle and it is -kept; the source comments describing it as per-visit are fixed. - -A consequence to write down: while `callerSubject()` prefers the verified -identity, `sub` and `human_sub` carry the **same value** whenever both are -present. They are still separate claims, because they mean different things and -would diverge the moment that preference changed -- and because agreement -between them is not a signal anything should test. +Both sides enforce it, by agreement -- +the website refuses to mint past it, this service refuses to accept past it +(`HUMAN_MAX_AGE_SECONDS`, `src/util/caller_token.py:228`) -- so neither is a +single point of failure, and changing the bound needs a change on both sides. + +How this service reads the claims (`human_presence_reason`, +`src/util/caller_token.py:280-309`): + +- `human` anything other than JSON `true` → `refused` / `no_human` +- `human_iat` missing or not a number → `no_human` +- more than 1800 whole seconds old → `stale_human` +- more than 60 seconds in the future → `no_human` +- `human_sub` is not checked here; it is used only for rate limiting (see "Limits") + +**`human_sub`, not `sub`.** The claim was named separately when `sub` was +understood here to be an opaque *per-visit* id, used by `identity_of` to key the +answer endpoint's backstop limiter; reusing `sub` for the per-browser cookie +subject would have changed that limiter's meaning. The website corrected that +understanding on 2026-09-19: their `callerSubject()` prefers the identity +cookie's subject whenever the reader has passed a challenge and falls back to a +per-visit value only when they have not. So `sub` is browser-scoped for every +gated reader, and the answer endpoint's limiter counts across visits for them. + +As a result, while `callerSubject()` prefers the verified identity, `sub` and +`human_sub` carry the **same value** whenever both are present. They remain +separate claims because they mean different things and would diverge if that +preference changed; nothing should test that they agree. + +**How `human_sub` is held.** It is used as received, not hashed, in a +rate-limiter key (`human:`) in this process's memory +(`src/api/analysis_summary.py:157-161`). A key is dropped at the limiter's next +sweep once it has had no request for a full window (`src/util/rate_limit.py:78-91`). +The 2026-09-19 agreement said "hashed, in memory only"; see "Known gaps". ## Response: Server-Sent Events @@ -96,19 +126,47 @@ comparable time, and a caller must be able to ignore it safely. ``` event: start -data: {"release": 97, "analysis_type": "OVERREPRESENTATION", "cached": false, - "disclosure": "aggregate"} - -event: token -data: {"text": "Of the 312 pathways hit, four remain significant after "} +data: {"release": 97, "analysis_type": "OVERREPRESENTATION", "cached": false, "disclosure": "aggregate"} event: citation data: {"st_id": "R-HSA-109581", "display_name": "Apoptosis"} +event: token +data: {"text": "Of the pathways shown, four remain significant after "} + event: done data: {"state": "summarised", "seconds": 6.2} ``` +Event order (`src/api/analysis_summary.py:166-328`): + +- **Summarised**: one `start`; then one `citation` per pathway sent to the + model (the first 12 of the result, `src/analysis/disclosure.py:89`), all + before any text; then `token` events; then `done`. A newly generated summary + streams as many `token` events. A stored summary arrives as **one** `token` + event carrying the whole text. +- **Refused**: only `done`, with `seconds: 0.0` and a `reason`. No `start`. +- **`not_found`, `gone`, `unsupported`**: only `done`. No `start`. +- **`failed`**: `done`, with or without a preceding `start` and partial `token` + events, depending on where it failed. A failure before `start` (the Analysis + Service unreachable, timing out or answering unexpectedly) sends only `done`. + +`done` carries `state` and `seconds` (rounded to 0.1), and `reason` only when +`state` is `refused`. `reason` is one of: + +- `no_caller`: no caller token, or one that is invalid, expired or too long +- `no_human` +- `stale_human`: the only one a reader can fix, by passing the challenge again +- `rate_limited` + +The handler also has an `unsupported_tier` refusal +(`src/api/analysis_summary.py:154-155`). It cannot occur today, because an +unknown `disclosure` value is rejected with a 422 first. + +Citations are every pathway the model was given, not only the ones the text +discusses. They come from the result rather than the model's prose, so each is +a pathway present in the result (SC-003). + `state` is one of `summarised`, `not_found`, `gone`, `unsupported`, `refused`, `failed`. @@ -122,27 +180,38 @@ data: {"state": "summarised", "seconds": 6.2} > 2026-09-21. > > `release` is a **number** on both, though it reaches this one as text from -> the Analysis Service and is parsed here. It is null when unparseable, which -> is a legitimate value on both endpoints. Anything but `summarised` means render no summary. Always HTTP 200 — -never an error code, so the analysis page cannot be broken by this service. +> the Analysis Service and is parsed here. It is null when unparseable or when +> the release lookup fails, which is a legitimate value on both endpoints. + +Anything but `summarised` means render no summary. Every outcome above arrives +as an HTTP 200 event stream, so a failure here does not become an error status +on the analysis page. The exception is a malformed request body, which gets +HTTP 422 (see "Request validation"). **No prose source list.** Citations arrive as `citation` events, as on -`/api/answer`, and the summary text must not end with a list of them -- -`SourcesSectionStripper` exists because that duplicate cost the website a -pattern it could not write correctly. This endpoint's prompt is its own, so -the right fix here is not to ask for one in the first place. - -**Expression summaries refer to columns as `column 1`, `column 2`**, numbered -from one in the order the values appear, and in no other form. The labels are -user-supplied text this service never receives, so the caller is the only side -that can name them -- and can only substitute them into the prose if the -wording is fixed. Anything else ("the first column", "the leftmost sample") -reaches the reader as written. +`/api/answer`. The model is told not to end the text with a list of them +(`SYSTEM_PROMPT` rule 6, `src/api/analysis_summary.py:76-77`). Unlike +`/api/answer`, which strips one with `SourcesSectionStripper`, this endpoint +does not remove a list if the model writes one anyway. + +The text is model output, passed through as generated; this endpoint does not +sanitise markup in it. + +**Expression summaries are told to refer to columns as `column 1`, `column 2`**, +numbered from one in the order the values appear, and in no other form +(`TYPE_INSTRUCTION["EXPRESSION"]`, `src/analysis/summarise.py:179-195`). This is +a prompt instruction, not checked in code. It held on the one expression result +checked on beta on 2026-09-21. The labels are user-supplied text this service +receives from the Analysis Service but never sends to the model, so the caller +is the only side that can name them -- and can only substitute them into the +prose if the wording is fixed. Anything else ("the first column", "the +leftmost sample") reaches the reader as written. **`disclosure` on `start` is the tier the summary was actually built from**, which is not always the one requested. If `identifiers` was asked for and the unmatched identifiers could not be retrieved, the summary is the aggregate one -and this says `aggregate`. +and this says `aggregate`. A stored aggregate summary for the token is reused if +there is one. Without it the reader chooses to disclose, the lookup fails, and they are handed the other summary with nothing to distinguish it -- disclosure with no @@ -165,10 +234,34 @@ release and says so with 410. A reader whose token returns `gone` should be told their analysis predates the current release and to run it again — that is an action, where `not_found` is a dead end. -**Summaries are stable per `(token, release, disclosure)`**, by storage rather -than by the generator being deterministic. The same request returns the same text -until the release changes. After a release the stored summary is discarded, -because the result it described has been too. +**Summaries are reused per `(token, release, disclosure)`**, by storage rather +than by the generator being deterministic (`src/analysis/store.py`). The same +request returns the same text, with `cached: true`, while all of these hold: + +- the same process is still running. The store is in memory, so a restart or + deploy loses every stored summary (research D4) +- the summary has not been evicted. The store keeps the 512 most recently used + summaries (`src/analysis/store.py:35`) +- the Analysis Service still reports the same release. The release is part of + the key, so after a release change the old summary is no longer found. It is + not deleted; it ages out of the store +- the release lookup succeeded. When it fails, `release` is null and the + summary is generated without being stored or reused + +`disclosure` in the key is the tier the summary was built from (`disclosure` on +`start`). Every request still fetches the result and the release from the +Analysis Service first, so a deleted result reports `gone` even when a summary +of it is stored. + +Concurrent requests for the same summary wait for the first one's generation +and are served its text, with `cached: true` (`src/api/analysis_summary.py:199-212`). +A stored summary is never replaced by a later generation +(`src/analysis/store.py:76-82`). + +"Continue in chat" (spec 013, `POST /api/handoff`) can hand off only a summary +that is still stored here under the current release and the same tier +(`stored_summary`, `src/api/analysis_summary.py:61-67`; `src/api/handoff.py:142-157`), +so the same conditions limit it. **An aggregate summary and a disclosing one are different artefacts.** They are stored separately and must not be presented as the same summary at different @@ -177,16 +270,50 @@ detail. ## What this endpoint will not do - Run, re-run or re-implement an analysis. It summarises a result that exists. -- State a statistic the result does not contain. -- Describe the lowest p-value as a finding when nothing passes correction. -- Summarise a ReactomeGSA result. It recognises one and returns `unsupported`. -- Read from production. Results come from beta's Analysis Service. - -## Open questions for the website side - -1. How human presence is asserted (above) — the blocker. -2. Where the disclosure choice is presented, and how the two options are described - so a user can tell what they are trading. -3. Whether the analysis page shows a summary automatically once consent exists, or - requires the ask each time. The spec requires opt-in per request; if that is - wrong for the page, it is a spec change rather than an implementation one. +- Summarise a ReactomeGSA result. It recognises one by `gsaMethod`/`gsaToken` + or by type `GSA_REGULATION`, `GSA_STATISTICS` or `GSVA`, and returns + `unsupported` (`src/analysis/client.py:84-109`). + +The model is instructed not to state a statistic the result does not contain, +and not to describe the lowest p-value as a finding when nothing passes +correction. Whether anything passes is computed in code and given to it +(`src/analysis/summarise.py:51-66`, `:146-153`, `:266-276`). These are +instructions to a model, not guarantees. + +Results are read from beta's Analysis Service by default; `ANALYSIS_BASE_URL` +points the service elsewhere (`src/analysis/client.py:22-23`, `:63-64`). + +## Limits + +| what | value | where | +|---|---|---| +| Whole request, from result lookup to the last token | 120 s, then `failed` | `SUMMARY_TIMEOUT_SECONDS`, `src/api/analysis_summary.py:47` | +| Model request | 90 s | `src/api/analysis_summary.py:351` | +| Each Analysis Service request | 20 s | `src/analysis/client.py:33` | +| Requests per person | 30 per 600 s by default (`ANSWER_RATE_LIMIT`, `ANSWER_RATE_WINDOW_SECONDS`). Keyed on `human_sub`, else the caller token's `sub`, else `jti`, else a hash of the token. Counted separately from `/api/answer`'s limit, which reads the same settings. Every request that passes the presence check counts, including ones served from the store | `src/api/analysis_summary.py:51`, `:157-164`; `src/util/rate_limit.py:30-49`, `:94-103` | +| Pathways sent to the model and cited | the first 12, in the result's order | `src/analysis/disclosure.py:89` | +| Unmatched identifiers sent (`identifiers` tier) | 50 | `src/analysis/client.py:200` | +| Caller token | at most 4,096 characters; `EdDSA` or `RS256`; `exp` and `aud` required; `aud` must equal `CALLER_TOKEN_AUDIENCE`, default `reactome-chatbot`; `iat`/`nbf` up to 30 s in the future tolerated | `src/util/caller_token.py:48-53`, `:109-121`, `:124-200` | + +## Known gaps between the 2026-09-19 agreement and the code + +Recorded here, not resolved. Which side should change has not been decided. + +- **`human_sub` is not hashed.** The agreement says the identifier is held + "hashed, in memory only". It is held in memory, as received + (`src/api/analysis_summary.py:159-161`). +- **No per-analysis-token limit.** Research D6 records adopting the website's + suggestion to rate limit per analysis token as well as per caller. Only the + per-caller limit exists (`src/api/analysis_summary.py:157-164`). + +## Questions raised for the website side (2026-09-18) + +1. How human presence is asserted. Settled 2026-09-19 (above, and research D6). +2. Where the disclosure choice is presented, and how the two options are + described so a user can tell what they are trading. +3. Whether the analysis page shows a summary automatically once consent + exists, or asks each time. The spec requires opt-in per request (FR-011). + This endpoint keeps no record of consent: it summarises whenever it is + called with a valid request. + +This repository does not record answers to 2 and 3. diff --git a/specs/011-summarise-analysis-results/data-model.md b/specs/011-summarise-analysis-results/data-model.md index 4b899845..8b971747 100644 --- a/specs/011-summarise-analysis-results/data-model.md +++ b/specs/011-summarise-analysis-results/data-model.md @@ -1,88 +1,116 @@ # Data model: summarising analysis results -Everything here is read from the Analysis Service or produced by this feature. -Nothing about an analysis is stored except the summary we generate. +Drafted 2026-09-18; corrected 2026-10-21 to match the code (`src/analysis/`, +`src/api/analysis_summary.py`). Everything here is read from the Analysis +Service or produced by this feature. Nothing about an analysis is stored except +the summary we generate. ## Read from the Analysis Service ### AnalysisResult (retrieved by token) -| field | used for | tier | -|---|---|---| -| `summary.type` | which reading applies: `OVERREPRESENTATION`, `EXPRESSION`, `SPECIES_COMPARISON` | aggregate | -| `summary.gsaMethod`, `summary.gsaToken` | detect ReactomeGSA and decline (D8) | aggregate | -| `summary.species`, `speciesName` | which organism was analysed | aggregate | -| `summary.projection`, `interactors`, `includeDisease` | caveats the summary must respect | aggregate | -| `summary.fileName`, `sampleName` | **never sent** — user-supplied free text | identifier | -| `pathways[].stId` | the citation; how a reader opens the pathway | aggregate | -| `pathways[].name` | what the summary calls it | aggregate | -| `pathways[].entities.found` / `.total` / `.ratio` | how much of the pathway was hit | aggregate | -| `pathways[].entities.pValue` / `.fdr` | significance, before and after correction | aggregate | -| `pathways[].entities.curatedFound` / `.interactorsFound` | whether a hit rests on curated data or inferred interactors | aggregate | -| `pathways[].entities.exp[]` | expression values per column (story 4) | aggregate | -| `expression.columnNames` | **never sent** — user-supplied column labels | identifier | -| `expression.min` / `.max` | the range values sit in | aggregate | -| `resourceSummary` | which identifier resources matched, the usual clue to a mismatch | aggregate | -| `speciesSummary` | species breakdown | aggregate | -| `identifiersNotFound` | how many did not match (a count, not the identifiers) | aggregate | -| `pathwaysFound` | how many pathways were hit at all | aggregate | -| `warnings` | what the service itself flagged | aggregate | +Only allow-listed fields may leave this service (`src/analysis/disclosure.py:24-63`). +Of those, only the ones marked "yes" in the last column are given to the model +(`prompt_input`, `src/analysis/summarise.py:51-141`). + +| field | used for | allow-listed (aggregate tier) | given to the model | +|---|---|---|---| +| `summary.type` | which reading applies: `OVERREPRESENTATION`, `EXPRESSION`, `SPECIES_COMPARISON`; `GSA_REGULATION`, `GSA_STATISTICS`, `GSVA` are declined (D8) | yes | yes, as `analysis_type` | +| `summary.gsaMethod`, `summary.gsaToken` | detect ReactomeGSA and decline (D8) | no; read only for detection | no | +| `summary.species`, `speciesName` | which organism was analysed | yes | no | +| `summary.projection`, `interactors`, `includeDisease` | caveats on the result | yes | no | +| `summary.fileName`, `sampleName` | nothing — user-supplied free text | no, under either tier | no | +| `pathways[]` | only the first 12, in the result's order | yes | yes | +| `pathways[].stId` | the citation; how a reader opens the pathway | yes | yes | +| `pathways[].name` | what the summary calls it | yes | yes | +| `pathways[].species`, `.inDisease` | — | yes | no | +| `pathways[].entities.found` / `.total` | how much of the pathway was hit; fewer than 5 found is flagged as fragile | yes | yes | +| `pathways[].entities.ratio` | — | yes | no | +| `pathways[].entities.pValue` / `.fdr` | significance, before and after correction (FDR ≤ 0.05) | yes | yes | +| `pathways[].entities.curatedFound` / `.interactorsFound` / `.resource` | — | yes | no | +| `pathways[].entities.exp[]` | expression values per column (story 4) | yes | yes | +| `pathways[].reactions.*` | — | no | no | +| `expression.columnNames` | nothing — user-supplied column labels | no, under either tier | no | +| `expression.min` / `.max` | the range values sit in | yes | yes | +| `resourceSummary` | which identifier resources matched, the usual clue to a mismatch | yes | yes | +| `speciesSummary` | species breakdown | yes | yes | +| `identifiersNotFound` | how many did not match (a count, not the identifiers) | yes | yes | +| `pathwaysFound` | how many pathways were hit at all | yes | no | +| `warnings` | what the service itself flagged; at most 5, each cut to 200 characters | yes | yes | + +The model is also given values computed from these: + +- a verdict (`empty`, `nothing_significant` or `has_findings`) +- per-pathway `significant` and `fragile` +- how many pathways the result has and how many were shown +- whether the significant count is exact (research D9) +- whether every identifier matched +- the number of expression columns ### Not-found identifiers (retrieved only on explicit request) `GET /token/{token}/notFound` — the user's own unmatched identifiers. Identifier -tier. Fetched only when the user has chosen the disclosing option, and never -otherwise. +tier. Fetched only when the user has chosen the disclosing option and no stored +summary can be reused, and never otherwise. The first 50 are fetched and sent to +the model (`src/analysis/client.py:199-237`). ### Release -`GET /database/version` — the release the Analysis Service is currently serving. -Part of the stored summary's key, because results are deleted on a release change. +`GET /database/version` — the release the Analysis Service is currently serving, +read on every request. Part of the stored summary's key (as the raw text), +because results are deleted on a release change. If the lookup fails, the +summary is still generated but neither stored nor reused, and `release` on +`start` is null. ## Produced by this feature -### DisclosureChoice +### Disclosure choice -What the user agreed to share, chosen per request. - -| field | values | meaning | -|---|---|---| -| `tier` | `aggregate` \| `identifiers` | which fields may be sent | -| `asked_at` | timestamp | when the user chose; absent means no consent and no summary | - -`aggregate` is the default and the only one that can be pre-selected. `identifiers` -is never a default. +What the user agreed to share, carried as the request's required `disclosure` +field (`aggregate` | `identifiers`). There is no default in the API, and no +consent record: this service keeps no timestamp or other state for the choice +(the drafted `asked_at` field was not built). The 2026-09-18 draft said only +`aggregate` may be pre-selected on a page and `identifiers` never; that is the +website's to do, and this service cannot enforce it. ### AnalysisSummary (ours, not the service's) +Held in `SummaryStore` (`src/analysis/store.py`), in process memory. + | field | meaning | |---|---| -| `token` | the analysis summarised | -| `release` | the release it was generated against; with `token`, the storage key | -| `tier` | which disclosure tier produced it — an aggregate summary and a disclosing one are different artefacts and must not be interchanged | -| `text` | the summary itself | -| `citations` | the pathways discussed, by stable id | -| `generated_at` | when, so the interface can say how old it is | - -**Key**: `(token, release, tier)`. A release change invalidates every summary for -the prior release, matching the service deleting the results themselves. +| `token` | the analysis summarised (part of the key) | +| `release` | the release it was generated against, as text (part of the key) | +| `tier` | the disclosure tier it was built from (part of the key) — an aggregate summary and a disclosing one are different artefacts and are not interchanged | +| `text` | the summary itself; an empty text is never stored | +| `citations` | `(st_id, display name)` for every pathway sent to the model | +| `generated_at` | when it was stored. Not sent to the caller | + +**Key**: `(token, release, tier)`. A release change makes every summary for the +prior release unreachable, matching the service deleting the results +themselves. The old entries are not deleted but age out, as the store keeps the +512 most recently used. The first summary stored under a key is never +replaced. Everything is lost on restart. ### Citation -Reuses the answer endpoint's shape exactly: `st_id` with a display name, resolving -to `reactome.org/content/detail/`. A summary cites only pathways present in -the result it describes — an invented or mismatched id is the failure this pins. +The answer endpoint's `st_id` form: `st_id` with a `display_name` (the pathway +name, or the `st_id` when there is none), which the caller links to +`reactome.org/content/detail/`. The answer endpoint can also send a +`url` citation; this one never does. Citations are taken from the result, not +from the model's text, so a summary cites only pathways present in the result +it describes (SC-003). ## Outcomes | outcome | when | |---|---| | `summarised` | a summary was produced | -| `not_found` | the token matches no result (404) | +| `not_found` | the token matches no result (404), the service answers 500 (what a malformed token gets), or the token fails this service's own format check | | `gone` | the result was deleted by a release (410) — tell the user to re-run | | `unsupported` | a ReactomeGSA result, recognised and declined | -| `refused` | no verified caller, or no evidence of a person | -| `failed` | anything else | +| `refused` | no verified caller (`no_caller`), no fresh evidence of a person (`no_human`, `stale_human`), or rate limited (`rate_limited`); the reason is on `done` | +| `failed` | anything else, including an Analysis Service error, a model error or empty reply, and the 120 s timeout | `gone` is deliberately distinct from `not_found`: one is a dead end, the other has an action attached. diff --git a/specs/011-summarise-analysis-results/plan.md b/specs/011-summarise-analysis-results/plan.md deleted file mode 100644 index b15fca90..00000000 --- a/specs/011-summarise-analysis-results/plan.md +++ /dev/null @@ -1,115 +0,0 @@ -# Implementation Plan: Summarise analysis results - -**Branch**: `011-summarise-analysis-results` | **Date**: 2026-09-18 | **Spec**: [spec.md](./spec.md) -**Input**: Feature specification from `specs/011-summarise-analysis-results/spec.md` - -## Summary - -Take an analysis token, fetch the completed result from beta's Analysis Service, -and produce a short readable account of what it says — which pathways came out on -top, whether they survive multiple-testing correction, why identifiers went -unmatched, and what the numbers mean. Cite every pathway by stable id. Store the -summary against `(token, release, tier)` so the same token yields the same text, -and say plainly that it was generated. - -The feature never runs an analysis. It never sends the user's identifiers unless -they choose that, and never their filename, sample name or column labels at all -under the default choice. - -## Technical Context - -**Language/Version**: Python 3.12, as the rest of the service -**Primary Dependencies**: the existing LangGraph answer surface, caller-token -verification, `AnchorStripper`, and the SSE response shape — all reused, none -rebuilt -**Storage**: summaries keyed `(token, release, tier)`. **No durable store exists on -beta today** — `POSTGRES_LANGGRAPH_DB` is unset there and LangGraph already falls -back to `MemorySaver`. First increment may hold summaries in process memory; -persistence is follow-up work (research D4) -**Testing**: pytest, with the suite's existing rule that it passes with no API keys -set. The disclosure guarantee is tested by recording outbound requests, not by -inspecting output -**Target Platform**: the same container that serves `/chat` and `/api/answer` -**Project Type**: single service -**Performance Goals**: comparable to the answer endpoint — first token in about -ten seconds. A stored summary returns immediately, which is most requests after -the first -**Constraints**: read from beta, never production; browser-like `User-Agent` on -outbound calls or the site's automation blocking returns 403 HTML; never state a -statistic the result does not contain -**Scale/Scope**: four analysis types, of which three are summarised and one -(ReactomeGSA) is recognised and declined - -## Constitution Check - -| Principle | Status | Note | -|---|---|---| -| I — verify the path a user takes | **Pass, with a named risk** | The endpoint must be exercised over HTTP on the real mounted app, as spec 010's was. The captcha middleware and the new human-presence bar interact only on the served path, and that is exactly where spec 010's route check found a problem that isolated tests could not. | -| II — measure retrieval changes, do not argue | **Not applicable** | This feature retrieves nothing from the vector store. It reads a completed analysis result. | -| III — characterization tests pin behaviour | **Pass** | The behaviours worth pinning are negative: what is *not* sent under the aggregate tier, that `gone` is distinct from `not_found`, and that nothing is said when nothing passes correction. | -| IV — fail loudly, never quietly differently | **Pass, and it cuts both ways** | Misconfiguration must stop the process, as the caller-token key already does. But a runtime failure must be quiet to the caller — a terminal state, never an HTTP error — because an analysis page must not break because this service did. | -| V — derive from the source of truth | **Pass, and it drives a decision** | The release is read from `GET /database/version`, not hardcoded or copied from the embeddings bundle. That matters because the Analysis Service deletes results on a release change, so the release is also the cache-invalidation key. | -| VI — bias to doing over filing | **Pass** | The blocker (how human presence is asserted) is recorded as a task with a named counterpart, not filed as a question and left. | -| VII — parked is not dead | **Pass** | ReactomeGSA is explicitly deferred with the reason, not silently omitted. | - -**No violations requiring justification.** - -## Project Structure - -### Documentation (this feature) - -``` -specs/011-summarise-analysis-results/ -├── spec.md -├── plan.md # this file -├── research.md # D1-D8, measured against beta's API -├── data-model.md # what is read, what is produced, and the disclosure tiers -├── quickstart.md # how to validate it with a real analysis token -├── checklists/ -│ └── requirements.md -└── contracts/ - └── summary_endpoint.md -``` - -### Source Code (repository root) - -``` -src/ -├── api/ -│ ├── answer.py # existing; patterns reused, not modified -│ └── analysis_summary.py # new: the endpoint -├── analysis/ # new -│ ├── client.py # fetch a result by token from beta; 404 vs 410 -│ ├── disclosure.py # the field allow-list that defines the aggregate tier -│ ├── summarise.py # build the prompt input from a result -│ └── store.py # summaries keyed (token, release, tier) -└── util/ - └── caller_token.py # existing; extended if human presence rides the token - -tests/ -├── api/ -│ └── test_analysis_summary.py -└── analysis/ - ├── test_disclosure.py # what must never be sent - ├── test_client.py # 404, 410, and the release - └── test_store.py # stability, and invalidation on release change -``` - -`src/analysis/` is separate from `src/agent/` because none of this touches the -graph or retrieval. It reads an external result and shapes it for a prompt. - -## Complexity Tracking - -One thing here is more complex than it first appears, and it is worth naming -rather than discovering. - -**The disclosure tier is a field allow-list, not a field denial.** The obvious -implementation — "strip the identifiers" — misses `summary.fileName`, -`summary.sampleName` and `expression.columnNames`, all of which are user-supplied -free text that can carry a lab's unpublished filename or a patient sample label. A -denial list is wrong by default whenever the Analysis Service adds a field; an -allow-list is only ever wrong by omission, which costs a missing sentence rather -than a disclosure. - -Everything else is deliberately unambitious: one external call, one prompt, one -store keyed by three values. diff --git a/specs/011-summarise-analysis-results/quickstart.md b/specs/011-summarise-analysis-results/quickstart.md index 760562dc..835ac22d 100644 --- a/specs/011-summarise-analysis-results/quickstart.md +++ b/specs/011-summarise-analysis-results/quickstart.md @@ -1,7 +1,10 @@ -# Quickstart: validating analysis summaries +# Validating analysis summaries against beta (2026-09-18 to 2026-09-21) -How to check this feature actually works, using real analysis results rather than -fixtures. Nothing here needs production. +**Status**: Historical record of how the feature was checked against beta's +Analysis Service, with the results of the one recorded run (2026-09-21, image +`cc4315c`, release 97). The commands below still show how to get a real token +from beta. The run results describe that build, not the current one. Nothing +here needs production. ## Prerequisites @@ -42,13 +45,16 @@ curl -s -A "$UA" "https://beta.reactome.org/AnalysisService/token/$TOKEN/notFoun Expect `database/version` to match the release in the summary's `start` event. -## Scenarios that must pass +## Scenarios used (drafted 2026-09-18) + +Outcomes on 2026-09-21 are in the next section. Scenarios 2 and 8 have never been +run live. | # | setup | expected | |---|---|---| | 1 | a token whose top pathways pass FDR | summary names those pathways, cites each by stable id | | 2 | a list that hits nothing significant | summary says nothing passes correction; no p-value presented as a finding | -| 3 | mostly unmatched identifiers | summary reports the **count** and points at identifier type or species. Not the proportion: the aggregate result has no submitted-total in it, so a percentage would be invented (Phase 4) | +| 3 | mostly unmatched identifiers | summary reports the **count** and points at identifier type or species. Not the proportion: the aggregate result has no submitted-total in it, so a percentage would be invented (research D5) | | 4 | `disclosure: aggregate` | the user's identifiers, filename, sample name and column labels appear in no outbound request | | 5 | same token requested twice | byte-identical summary, `cached: true` on the second | | 6 | an expired token | `state: not_found` | @@ -56,25 +62,29 @@ Expect `database/version` to match the release in the summary's `start` event. | 8 | a ReactomeGSA token | `state: unsupported` | | 9 | no caller token / no human assertion | `state: refused`, and **zero model calls** | -Scenario 4 is the one to automate first and the one worth distrusting: it is an -assertion about what was *not* sent, so it should be checked by recording outbound -requests, not by reading the summary and seeing nothing alarming. +Scenario 4 is an assertion about what was *not* sent, so it is checked by +recording outbound requests, not by reading the summary and seeing nothing +alarming. The automated tests do this (`tests/analysis/test_disclosure.py`, +`tests/api/test_analysis_summary.py`). -Scenario 5 is what makes FR-014 real. Test it across a restart too, and expect it -to fail there until a durable store exists (research D4) — that failure is known, -not a surprise. +Scenario 5 holds only within one process: the store is in memory, so a summary +does not survive a restart or deploy (research D4). -## Checking a summary is honest +## Honesty checks proposed (2026-09-18) -Mechanical checks first, because they do not need judgement: +Mechanical checks, which need no judgement: -- every `st_id` cited appears in that result's `pathways[]` -- no number in the text is absent from the result -- when `pathwaysFound` is 0, the text says nothing was found +- every `st_id` cited appears in that result's `pathways[]`. Holds by + construction now and is tested (`tests/api/test_analysis_summary.py:225-233`) +- no number in the text is absent from the result. Not automated +- when the result has no pathways, the text says nothing was found. The code + decides this from the `pathways` list (`verdict: empty`, + `src/analysis/summarise.py:61-62`), not from `pathwaysFound`, which the + model is not given Then a curator reads a handful spanning strong, weak and empty results and says -whether each conveys how much to trust it. That is SC-006 and it cannot be -automated. +whether each conveys how much to trust it (SC-006). This repository records no +such review. ## Run against beta, 2026-09-21 @@ -97,20 +107,19 @@ Also checked, beyond the table: an `EXPRESSION` result referred to `column 1`, `column 2`, `column 3` and no other form, and the `identifiers` tier named the reader's unmatched identifiers. -**Three scenarios were not run, and the reason is that they cannot be -constructed here rather than that they were skipped.** +**Two scenarios were not run, because they could not be constructed here +rather than because they were skipped.** (A third, 7, was first listed here and +then turned out to be constructible.) - **2, nothing significant** — needs an identifier list that hits no pathway past correction. Every list tried produced significant hits, and inventing one by editing a result would test the code against a fixture rather than the service. Covered by unit tests on the verdict instead. -- ~~**7, `gone`**~~ — **this turned out to be constructible, and now passes.** - See below. - **8, ReactomeGSA** — needs a GSA analysis, which is a different service. Detection is unit-tested against all three GSA type values. -Scenario 5 across a restart is expected to fail until a durable store exists -(research D4), and was not attempted for that reason. +Scenario 5 across a restart was not attempted: with an in-memory store it cannot +pass (research D4). **The first run of scenario 4 proved nothing**, and the correction is worth keeping because the mistake is easy to repeat. The disclosure check ran @@ -156,10 +165,10 @@ This matters beyond ticking the scenario off. The `gone` path would otherwise have been first exercised for real during a release, which is the worst moment to discover a handling bug in it. -**Bisect for the boundary at run time; never hardcode a token.** The cutoff -slides forward daily, so any fixed timestamp eventually falls out of the -window and starts returning 404 -- and a test that does that does not fail, -it quietly starts asserting `not_found` while still passing. That is the same +**A hardcoded token stops testing `gone`.** The cutoff slides forward daily, so +any fixed timestamp eventually falls out of the window and starts returning +404 -- and a check written that way does not fail, it quietly starts asserting +`not_found` while still passing. The boundary has to be found at run time. That is the same shape as everything else this feature has had to guard against: a check that goes on passing after it has stopped testing the thing. @@ -175,4 +184,4 @@ The assumption worth naming, because it cost more than the boundary did: this scenario was recorded as untestable on the belief that only the service can mint a token. That was never decided or written down anywhere, and it was load-bearing for a plan in which `gone` would first be exercised by a curator -during a release. Nothing about it looked like a question. \ No newline at end of file +during a release. Nothing about it looked like a question. diff --git a/specs/011-summarise-analysis-results/research.md b/specs/011-summarise-analysis-results/research.md index b2c36817..ecd4e3d5 100644 --- a/specs/011-summarise-analysis-results/research.md +++ b/specs/011-summarise-analysis-results/research.md @@ -1,8 +1,11 @@ -# Research: summarising analysis results +# Research: summarising analysis results (2026-09-18 to 2026-09-21) -Measured against beta's Analysis Service (`/AnalysisService/v3/api-docs`, -release 97) on 2026-09-18. Every claim below came from the API or from this -repository, not from recollection. +Findings from the analysis-summary work. Measured against beta's Analysis +Service (`/AnalysisService/v3/api-docs`, release 97) from 2026-09-18; later +sections give their own dates. Every claim below came from the API or from this +repository, not from recollection. "Decision" in a section means a decision for +that work. Corrections added on 2026-10-21 are marked. Current behaviour is in +[contracts/summary_endpoint.md](./contracts/summary_endpoint.md). ## D1 — Results are addressed by a token, and that is the whole input @@ -11,7 +14,15 @@ list and never runs an analysis. **Rationale**: `GET /token/{token}` returns the completed `AnalysisResult`. The analysis has already happened; re-running it here would duplicate the Analysis -Service and violate Principle V. It also means we never hold the user's input. +Service and violate Principle V. It also means this endpoint never receives the +user's submitted list. + +*(Corrected 2026-10-21.)* At the `identifiers` tier the endpoint does fetch up to +50 of the user's unmatched identifiers from the Analysis Service and sends them to +the model (D5; `src/analysis/client.py:199-237`). Separately, the chat's +gene-list feature (3b2c792, 2026-09-25) submits identifiers to the Analysis +Service through `submit_identifiers` in the same module. This endpoint does not +use it. **Alternatives considered**: accepting identifiers and running the analysis ourselves — rejected outright, it is another service's job and would put user data @@ -50,14 +61,23 @@ worse a retry loop against a service that will answer the same way every time. it would leave a user re-pasting a token that will never work again, which is why 410 stays distinct even though 500 does not. +**Later (2026-09-19).** A token outside the expected base64url-style shape is now +rejected before any request and reported as `not_found`, because a +caller-supplied `/notFound` reached the identifier endpoint +(`src/analysis/client.py:35-51`). + ## D3 — Stability comes from storing the summary, keyed by token *and release* -**Decision**: Store a generated summary against `(token, release)`. Serve the -stored one thereafter. Discard on a release change. +**Decision for this work**: store a generated summary against `(token, release)` +and serve the stored one thereafter. *(As built, 2026-09-20: the key is +`(token, release, tier)`, the tier being the one the summary was built from. A +release change is handled by the key, not by deleting anything: the old entry +is no longer looked up and ages out of the bounded store. `src/analysis/store.py`.)* **Rationale**: FR-014 wants the same token to yield the same summary. Generation -cannot provide that — measured on this repo, the same question through the same -surface twice scores 0.33 similarity. Storage can, because an analysis result is a +cannot provide that — measured on this repo (spec 010, on the answer surface rather +than on summaries), the same question through the same surface twice scores 0.33 +similarity. Storage can, because an analysis result is a fixed artefact. The release must be part of the key because D2 says the Analysis Service *deletes @@ -71,14 +91,14 @@ which is the same invalidation signal the answer endpoint already publishes as rejected, seeded runs still scored 0.47 and 0.14 similarity. Caching by token alone — rejected by D2. -## D4 — There is nowhere durable to store it yet +## D4 — No durable store on beta; summaries are kept in process -**Decision**: Treat the store as a required piece of work, not an assumption. -First increment may keep summaries in process memory, provided the transparency -requirement (FR-015) already tells the user a summary can be regenerated. +**Decision for this work (2026-09-18)**: keep summaries in process memory for the +first increment, relying on FR-015's transparency to make regeneration visible, +and leave a durable store as follow-up work. -**Rationale**: beta sets no `POSTGRES_LANGGRAPH_DB`, so LangGraph already falls -back to `MemorySaver` and nothing on that host persists across a restart. An +**Rationale**: beta set no `POSTGRES_LANGGRAPH_DB` (2026-09-18), so LangGraph fell +back to `MemorySaver` and nothing on that host persisted across a restart. An in-memory store satisfies FR-014 within a process lifetime and loses summaries on deploy — which is honest only because FR-015 makes regeneration visible rather than surprising. @@ -96,10 +116,12 @@ the least recently used. So: - `cached` on `start` is what makes that honest rather than surprising, and is the reason FR-015 exists alongside FR-014 -**The durable store is the open piece.** It needs somewhere to put it — -`POSTGRES_LANGGRAPH_DB` is unset on beta and LangGraph already falls back to -`MemorySaver`, so there is no existing home to reuse. Anyone picking this up -should treat "summaries vanish on deploy" as a known state, not a bug. +**The durable store was not built.** As of 2026-10-21 the store is still in +process (`src/analysis/store.py`), so summaries are lost on deploy by design. +"Continue in chat" (spec 013) loses the ability to hand them off at the same +time. When this was written, `POSTGRES_LANGGRAPH_DB` was unset on beta, so there +was no existing home to reuse; this repository does not record beta's current +setting. Whether a durable store is still wanted is not recorded. Two decisions inside the store worth not reversing by accident. An **empty summary is never stored**: a failed or abandoned generation leaves no text, @@ -109,6 +131,11 @@ answer, indistinguishable from a result with nothing to say. And eviction summary was evicted simply regenerates, where refusing new entries would quietly stop the feature working for everyone after the first few hundred. +**Added 2026-10-03 (d153fbf).** A stored summary is never overwritten, and a +second request for a summary that is being generated waits for it. So two +readers, or a reader and a later "Continue in chat", see the same text +(`src/analysis/store.py:76-82`; `src/api/analysis_summary.py:199-212`). + ## D5 — What "an option that discloses no identifiers" means, precisely **Decision**: Two disclosure tiers, defined by field rather than by intention. @@ -123,8 +150,23 @@ stories 1, 3 and 4 is already free of user content: - `resourceSummary`, `speciesSummary`, `pathwaysFound`, `identifiersNotFound` (a count), `warnings` +*(Corrected 2026-10-21.)* As built, the allow-list (`src/analysis/disclosure.py:24-63`) +keeps `entities` statistics only, not `reactions`. It adds `resource` and `exp` +per pathway and `inDisease`, keeps only the first 12 pathways and +`expression.min`/`max`, and keeps at most five `warnings` cut to 200 characters +each. The allow-list bounds what *may* be sent. What the model is actually given +is narrower (`prompt_input`, `src/analysis/summarise.py:51-141`): the analysis +type; per pathway `stId`, `name`, `found`, `total`, `pValue`, `fdr` and `exp`; +`identifiersNotFound`, `resourceSummary`, `speciesSummary`, `warnings` and the +expression range; and values computed from them. `species`, `speciesName`, +`projection`, `interactors`, `includeDisease`, `pathwaysFound`, `ratio`, +`curatedFound`, `interactorsFound`, `resource` and `inDisease` are allow-listed +but not passed on. + **Identifier tier — only on explicit request.** `GET /token/{token}/notFound` and -`/token/{token}/found/entities/{pathway}` return the user's own identifiers. +`/token/{token}/found/entities/{pathway}` return the user's own identifiers. As +built, the `identifiers` tier uses only the first: up to 50 unmatched +identifiers (`src/analysis/client.py:199-237`). **The trap, and it is not the gene list.** Three fields in the *aggregate* result are user-supplied free text and must be excluded from the aggregate tier: @@ -138,9 +180,17 @@ through. This is why the tier is defined as a field allow-list, not a denial of one obvious field. **Consequence worth stating**: the aggregate tier answers user story 1 fully, and -user story 2 *partially* — it can report the proportion unmatched and the resource -mismatch, which is usually the cause, but cannot name which identifiers failed. -That is a real and explainable difference for the user to choose between. +user story 2 *partially* — it can report the resource mismatch, which is usually +the cause, but cannot name which identifiers failed. That is a real and +explainable difference for the user to choose between. + +*(Corrected 2026-10-21, from a finding of 2026-09-20.)* This originally said the +aggregate tier could report the *proportion* unmatched. It cannot. The +aggregate result carries `identifiersNotFound` and nothing about how many +identifiers were submitted, which is only available from +`/token/{token}/found/all`, an identifier-tier endpoint. The summary reports +the count and is told never to state a proportion +(`src/analysis/summarise.py:155-171`). ## D6 — Proving a person is present is the website's to assert, not ours to infer @@ -163,12 +213,14 @@ on the request: | `human_iat` | when the challenge was solved, epoch **seconds**. Derived from the cookie's expiry minus their identity TTL, so it arrives rounded to a second -- the bound is whole-second, and a sub-second edge is not a state this claim can represent | | `human_sub` | the cookie's random 16-byte identifier, for per-identity rate limiting. Carries nothing about the person | -**Named `human_sub`, not `sub`.** `sub` is already an opaque *per-visit* id -that `identity_of` uses for the answer endpoint's backstop limiter. The cookie -subject is per-*browser* and lives as long as the cookie, so reusing the claim -would change that limiter's meaning on an endpoint neither repo is touching -- -a behaviour change arriving through a rename. Caught before either side built -to it. +**Named `human_sub`, not `sub`.** At the time `sub` was understood here to be an +opaque *per-visit* id that `identity_of` uses for the answer endpoint's +backstop limiter. The cookie subject is per-*browser* and lives as long as the +cookie, so reusing the claim would have changed that limiter's meaning on an +endpoint neither repo was touching. *(Corrected 2026-10-21, from the website's +correction of 2026-09-19 recorded in the contract: `sub` is already the +identity cookie's subject for any reader who has passed the challenge, and +per-visit only for one who has not. The claims stayed separate.)* **Freshness is 30 minutes, inclusive in whole seconds (`now - human_iat <= 1800`), enforced at both ends.** We refuse a `human_iat` older than that, and they refuse to mint the claim past it, so neither side is @@ -189,12 +241,17 @@ disclosure tier — a choice a bot holding a link can make is a consent mechanis that consents on the user's behalf, which is worse than no choice because it looks like one. -**Their caveat, adopted**: rate limit per token *and* per caller regardless. A -token asking for twenty summaries of one analysis is not a scientist. That is a -throttle, not evidence of a person, and it does not substitute for the claim. - -**Dependency that remains**: their proxy mints caller tokens for the answer -route only, so a summary route must be added before any claim can be carried. +**Their caveat, recorded as adopted**: rate limit per analysis token *and* per +caller regardless. A token asking for twenty summaries of one analysis is not a +scientist. That is a throttle, not evidence of a person, and it does not +substitute for the claim. *(2026-10-21: only the per-caller half was built. The +limiter keys on `human_sub`, falling back to the caller token's identity; there +is no per-analysis-token limit (`src/api/analysis_summary.py:157-164`). The +contract lists this as a known gap.)* + +**Dependency at the time (2026-09-19)**: their proxy minted caller tokens for the +answer route only, so a summary route had to be added before any claim could be +carried. The contract records the website calling this endpoint by 2026-09-21. Nothing here assumes the browser calls us directly — it cannot, since the cookie is same-site to their origin. @@ -221,6 +278,10 @@ apply to neither or to both. shape. Recognising it costs one field check and prevents the worst outcome — a confident summary of a result we do not actually model. +**Extended 2026-09-20.** Detection also checks the analysis type: +`GSA_REGULATION`, `GSA_STATISTICS` and `GSVA` are declined even without +`gsaMethod` (`src/analysis/client.py:84-109`). + ## D9 -- a truncated result invites a statistic the result does not contain Found on 2026-09-19 by calling the real model with a real result, after every @@ -247,7 +308,13 @@ rather than assumed**, because relying on another service's default sort is how a claim like this goes quietly wrong. When the count is a lower bound the model is told so explicitly, and the system prompt forbids the claim outright. -Re-run against the real model: the sentence is gone. +Re-run against the real model on 2026-09-19: the sentence did not appear (the +number of runs is not recorded). + +**Later (2026-10-03, d153fbf).** The lower-bound instruction ("every one of +them is significant") is now given only when every shown pathway is +significant. When the count is inexact for another reason, the model is told the +overall count cannot be determined (`src/analysis/summarise.py:244-260`, `:279-290`). **The general shape is worth keeping.** Bounding a payload for cost is not neutral -- it changes what the data appears to say, and a model will read the diff --git a/specs/011-summarise-analysis-results/spec.md b/specs/011-summarise-analysis-results/spec.md index ed527e16..b028d1e3 100644 --- a/specs/011-summarise-analysis-results/spec.md +++ b/specs/011-summarise-analysis-results/spec.md @@ -2,7 +2,16 @@ **Feature Branch**: `011-summarise-analysis-results` **Created**: 2026-09-18 -**Status**: Draft +**Status**: Historical record of the analysis-summary work, 2026-09-18 to +2026-09-21. The endpoint was built 2026-09-19 to 2026-09-20 (4e2d8ab, 9fe1461) +and was live on beta on 2026-09-21; review fixes followed on 2026-10-03 +(d153fbf). Two pieces were not built: a durable summary store (research D4) +and a rate limit per analysis token (research D6). The spec's status field was +never moved past "Draft". FR-011 to FR-015 record the answers to two +clarification questions (privacy and stability) given on 2026-09-18. The +requirements below were set for that work, not for later changes. Current +behaviour is in `src/api/analysis_summary.py`, `src/analysis/` and +[contracts/summary_endpoint.md](./contracts/summary_endpoint.md). **Input**: Summarise Reactome analysis results in natural language, for users who have run an analysis and want to know what it means. ## Why this exists @@ -19,7 +28,10 @@ than a biological absence. That knowledge is exactly what the curators put in th user guide and exactly what a reader does not have to hand at the moment they get their result. -## User Scenarios & Testing *(mandatory)* +## User stories (as specified 2026-09-18) + +The stories and acceptance scenarios were the targets for the work. Notes dated +2026-10-21 say where the built feature differs. ### User Story 1 - What does my result say? (Priority: P1) @@ -32,10 +44,6 @@ common biologically — instead of reading the table themselves. and it is answerable from the result alone. On its own it is a usable feature: someone with a result gets a readable account of it. -**Independent Test**: Submit a known analysis token, and check the summary names -the pathways the result actually ranks highest, describes their significance in -terms the result supports, and cites each pathway it discusses by stable id. - **Acceptance Scenarios**: 1. **Given** a result whose top pathways pass FDR correction, **When** a summary is @@ -59,10 +67,6 @@ analysis, and it has a small number of well-understood causes — an identifier Reactome does not index, the wrong species, or identifiers that are genuinely absent. It is independently valuable: a user can ask only this and be helped. -**Independent Test**: Submit a token from an analysis with a deliberate identifier -mismatch and check the summary reports the proportion unmatched and names the -likely cause from the evidence in the result. - **Acceptance Scenarios**: 1. **Given** a result where most identifiers were not found and the matched ones @@ -75,6 +79,14 @@ likely cause from the evidence in the result. findings, **Then** the summary says the result should be treated with caution and why. +*Built differently (found 2026-09-20).* The aggregate result carries how many +identifiers were not found and nothing about how many were submitted, so no +proportion can be derived without the reader's identifiers. The summary +reports the count and is told never to state a proportion +(`src/analysis/summarise.py:155-171`). Scenarios 1 and 3 therefore hold only as +far as the count and `resourceSummary`/`speciesSummary` allow; nothing computes +a "too many unmatched" threshold. + --- ### User Story 3 - What do these numbers mean? (Priority: P3) @@ -87,9 +99,10 @@ whether a hit resting on very few entities is worth pursuing. types, but a user can get value from stories 1 and 2 without it, and it is closest to material the user guide already covers. -**Independent Test**: Request an explanation for a specific pathway in a result and -check the explanation uses that pathway's own numbers rather than generic -definitions. +*As built (2026-09-20).* There is no request for one pathway. Every pathway +sent to the model carries its own counts and a flag, computed in code, for a +hit resting on fewer than five matched entities (`src/analysis/summarise.py:18-48`). +The model is told to explain those in the reader's terms. **Acceptance Scenarios**: @@ -113,9 +126,9 @@ types, but story 1 must be right first. Deferring it is safe because the type is visible in the result, so a summary can decline to make type-specific claims until this is built. -**Independent Test**: Submit an expression result and a species-comparison result, -and check each summary addresses what is specific to that type and neither -describes the other's. +*Built 2026-09-20* as per-type instructions to the model +(`TYPE_INSTRUCTION`, `src/analysis/summarise.py:178-209`), including one for +over-representation results. **Acceptance Scenarios**: @@ -127,69 +140,39 @@ describes the other's. --- -### Edge Cases - -- A token that does not exist, or has expired, or belongs to an analysis that has - been discarded. -- A result too large to summarise in full: hundreds of significant pathways, or an - expression matrix with many columns. -- A result whose findings are entirely disease pathways, which is often an artefact - of the submitted list rather than a finding. -- An analysis run against a species the summary's knowledge does not cover well. -- A result already summarised: a reader who reloads should not be told something - materially different about a fixed artefact (see Assumptions). -- An identifier list that is itself the user's unpublished research data. - -## Requirements *(mandatory)* - -### Functional Requirements - -- **FR-001**: The system MUST summarise an analysis result the user has already run, - identified by its analysis token, and MUST NOT run, re-run or re-implement any - analysis. -- **FR-002**: The system MUST derive every quantitative claim from the analysis - result itself, and MUST NOT state statistics the result does not contain. -- **FR-003**: The system MUST distinguish significance before and after multiple - testing correction whenever it describes a pathway as significant. -- **FR-004**: The system MUST report when a result is weak, empty, or heavily - unmatched, rather than describing the strongest available row as a finding. -- **FR-005**: Every pathway the summary discusses MUST be attributable by Reactome - stable id, so a reader can open it. -- **FR-006**: The system MUST identify which of the analysis types it is - summarising, and MUST NOT make claims specific to a type the result is not. -- **FR-007**: The system MUST refuse a request that does not carry a verified - caller, before any model call, on the same terms as the existing answer endpoint. - That is the floor, not the bar — see FR-013. -- **FR-008**: The system MUST fail invisibly to the caller: any error, timeout or - refusal yields a terminal state the caller can render as "no summary", never a - broken panel or an HTTP error. -- **FR-009**: The system MUST treat an unknown, expired or malformed token as a - normal negative outcome, not an error condition. -- **FR-010**: The system MUST read analysis results from the beta Analysis Service - for now, never from production. -- **FR-011**: Summarising MUST be opt-in. The system MUST NOT send any part of an - analysis result to a model provider until the user has actively asked for a - summary. A result being viewed is not consent; nothing is summarised in the - background or in anticipation. -- **FR-012**: The user MUST be offered a choice of what is shared, and the choice - MUST be meaningful — at least one option MUST produce a useful summary without - transmitting their submitted identifiers. The user is choosing between summaries - of different quality at different disclosure, and MUST be told which is which - before choosing, not after. -- **FR-013**: The system MUST require evidence that a person is present, not merely - that a known service is calling. This is a stricter bar than the answer - endpoint's, which verifies caller identity and deliberately asserts nothing about - humanity, and it exists because this feature discloses a user's own uploaded data - rather than public pathway text. -- **FR-014**: A summary MUST be stable for a given analysis token: the same token - MUST yield the same summary on request after request, so that a reader who - reloads, or who cites it, sees what they saw before. An analysis result is a - fixed artefact and its summary must behave like one. -- **FR-015**: The system MUST be transparent about what a summary is: that it was - generated rather than curated, which analysis it describes, and that regenerating - it may produce different wording. Stability under FR-014 is achieved by reuse, - not by the generator being deterministic, and the interface MUST NOT imply - otherwise. +### Edge cases identified, and how they are handled (checked 2026-10-21) + +| Edge case | Today | +|---|---| +| A token that does not exist, has expired, or belongs to a discarded analysis | `not_found` for 404, 500 or a malformed token; `gone` for 410 (FR-009) | +| A result too large to summarise in full | Only the first 12 pathways are sent, and the model is told when the significant count is a lower bound (research D9). Expression values are sent for every column; the number of columns is not bounded | +| Findings that are entirely disease pathways, often an artefact of the list | Not handled. `inDisease` is allow-listed but not given to the model (`src/analysis/summarise.py:99-125`) | +| A species the summary's knowledge does not cover well | Not handled specifically. `speciesSummary` is given to the model; `summary.species` is not | +| A result already summarised | The stored summary is reused (FR-014) | +| An identifier list that is the user's unpublished data | The `aggregate` tier never sends identifiers; `identifiers` sends up to 50 unmatched ones (FR-012) | + +## Requirements of this work, and where they stand (checked 2026-10-21) + +"Instructed" means the behaviour is asked of the model in its prompt, not +checked in code. + +| ID | Required by this work (2026-09-18) | Today | +|---|---|---| +| FR-001 | Summarise a result the user has already run, by token; never run, re-run or re-implement an analysis | Holds for this endpoint, which takes only a token. (`src/analysis/client.py` gained `submit_identifiers` on 2026-09-25 for the chat's gene-list feature, 3b2c792; this endpoint does not call it.) | +| FR-002 | Derive every quantitative claim from the result; state no statistic it does not contain | Instructed (`DATA_RULES` rule 1, `src/analysis/summarise.py:266-276`). The input says when the significant count is only a lower bound (research D9). Not checked mechanically | +| FR-003 | Distinguish significance before and after multiple-testing correction | Instructed (`STATISTICS_INSTRUCTION`, `src/analysis/summarise.py:213-226`; `DATA_RULES` rule 3) | +| FR-004 | Report a weak, empty or heavily unmatched result rather than presenting the top row as a finding | Empty and nothing-significant are decided in code (`verdict`, `src/analysis/summarise.py:58-66`) with a matching instruction. Hits on fewer than five entities are flagged in code (`:38-48`). "Heavily unmatched" cannot be judged as a proportion (US2 note) | +| FR-005 | Every pathway discussed attributable by stable id | Holds: a `citation` event for every pathway sent to the model (`src/api/analysis_summary.py:263-278`) | +| FR-006 | Identify the analysis type; no claims specific to another type | `analysis_type` on `start`; per-type instructions (`src/analysis/summarise.py:178-209`); ReactomeGSA types declined (`src/analysis/client.py:84-109`) | +| FR-007 | Refuse without a verified caller, before any model call, on the answer endpoint's terms | Holds: the same `verify` (`src/api/analysis_summary.py:133-140`); zero model calls tested (`tests/api/test_analysis_summary.py:215-222`) | +| FR-008 | Fail invisibly: a terminal state, never a broken panel or an HTTP error | Holds for refusals, lookup outcomes, failures and timeouts (HTTP 200 with `done`). A malformed request body gets HTTP 422 (`tests/api/test_analysis_summary.py:285-292`) | +| FR-009 | An unknown, expired or malformed token is a normal negative outcome | Holds: 404, 500 and a malformed token give `not_found`; 410 gives `gone` (`src/analysis/client.py:127-148`) | +| FR-010 | Read from beta's Analysis Service, never production | Beta is the default; `ANALYSIS_BASE_URL` overrides it (`src/analysis/client.py:22-23`, `:63-64`) | +| FR-011 | Opt-in: nothing sent to a model provider until the user asks | This endpoint generates only when called and keeps no consent state. When it is called is the website's decision | +| FR-012 | A meaningful choice of what is shared, with one useful option sending no identifiers, explained before choosing | `disclosure` is required with no default (`src/api/analysis_summary.py:88-89`); `aggregate` sends no identifiers. How the choice is presented is the website's (contract, question 2) | +| FR-013 | Evidence that a person is present, not only a known service | Holds: `human` and a 30-minute `human_iat` (`src/util/caller_token.py:228`, `:280-309`); research D6 | +| FR-014 | The same token yields the same summary on every request | Holds within one process while the summary stays in the 512-entry store and the release is unchanged; lost on deploy. A durable store was not built (research D4) | +| FR-015 | Transparent that it is generated, which analysis it describes, and that regenerating may differ; stability by reuse | `cached`, `release`, `analysis_type` and `disclosure` on `start` (`src/api/analysis_summary.py:241-261`). Telling the reader is the website's | ### Key Entities @@ -204,46 +187,39 @@ describes the other's. - **Summary**: the natural-language account produced for a result, with the pathway citations that let a reader check it. -## Success Criteria *(mandatory)* - -### Measurable Outcomes - -- **SC-001**: For a result whose top pathways pass FDR correction, the summary names - the same pathways the result ranks highest, verified against the result across the - tracked analysis set. -- **SC-002**: For a result where nothing passes correction, the summary says so; it - never presents the lowest p-value as a finding. Verified with deliberately null - results. -- **SC-003**: Every pathway named in a summary resolves to a real Reactome stable id - present in that result — no invented or mismatched identifiers, checked - mechanically rather than by reading. -- **SC-004**: Zero model calls for requests without a verified caller, measured by - counting calls under unauthenticated load. -- **SC-005**: An unknown or expired token produces a terminal "no summary" outcome, - and never an error the caller must special-case. -- **SC-006**: A reader can tell, from the summary alone, whether the result is - trustworthy enough to act on — assessed by curator review of summaries for a set - of results chosen to span strong, weak and empty outcomes. - -## Assumptions +## Success criteria of this work, and what was checked + +| ID | Criterion (2026-09-18) | Checked | +|---|---|---| +| SC-001 | For a result whose top pathways pass FDR, the summary names the pathways the result ranks highest, across a tracked analysis set | Once live, on one token, 2026-09-21 ([quickstart.md](./quickstart.md) scenario 1). This repository records no tracked analysis set or evaluation over one | +| SC-002 | When nothing passes correction, the summary says so and presents no p-value as a finding | Unit-tested on the model input (`tests/analysis/test_summarise.py`). Not run live: no such result could be produced on 2026-09-21 | +| SC-003 | Every pathway named resolves to a stable id present in the result, checked mechanically | Holds by construction: citations come from the result (`tests/api/test_analysis_summary.py:225-233`) | +| SC-004 | Zero model calls without a verified caller | `tests/api/test_analysis_summary.py:187-222`, counted on a patched model | +| SC-005 | An unknown or expired token gives a terminal "no summary" outcome | `not_found` / `gone` in an HTTP 200 stream (`tests/api/test_analysis_summary.py:256-282`) | +| SC-006 | A reader can tell from the summary whether to trust the result, by curator review | This repository records no curator review | + +## Assumptions at the time (2026-09-18) - The caller has already run the analysis and holds its token; this feature never accepts a raw identifier list, which keeps it clear of the Analysis Service's job. -- The caller is the Reactome website, reached through its server-side proxy, and - presents the same caller token the answer endpoint verifies. No new - authentication mechanism is introduced. +- The caller is the Reactome website, reached through its server-side proxy, + presenting the caller token the answer endpoint verifies. That held, but the + token gained human-presence claims for this endpoint (FR-013, research D6; + `src/util/caller_token.py:203-309`), so the assumption that no new + authentication would be needed did not. - Delivery follows the existing answer endpoint's shape — progressive output, citations as structured events, failure as a terminal state — because a summary takes comparable time to produce and the website already renders that shape. + The state names differ (see the contract). - Results are read from beta's Analysis Service, consistent with the standing - instruction to keep off production. -- ReactomeGSA results are recognised from the result, but summarising them is a - later increment: GSA is a separate service with its own result shape, and - including it in the first increment would double the surface. + instruction to keep off production (FR-010). +- ReactomeGSA results are recognised from the result, but summarising them was + left for a later increment: GSA is a separate service with its own result + shape. It has not been built. - The measured non-reproducibility of generated answers applies here too, which is - why FR-012 asks the question rather than assuming stability. + why FR-014 and FR-015 make stability a matter of reuse rather than assuming it. -## Scope +## Scope (as set 2026-09-18) **In scope**: summarising a completed analysis result; explaining its statistics; explaining unmatched identifiers; recognising the analysis type. @@ -252,7 +228,7 @@ explaining unmatched identifiers; recognising the analysis type. the result does not contain; comparing two analyses; storing results; the analysis UI itself. -## Dependencies +## Dependencies (as of 2026-09-18) - The Analysis Service on beta, for retrieving results by token. - The existing caller-token verification and streaming answer surface, reused diff --git a/specs/011-summarise-analysis-results/tasks.md b/specs/011-summarise-analysis-results/tasks.md index 9b235b24..1dfefe2a 100644 --- a/specs/011-summarise-analysis-results/tasks.md +++ b/specs/011-summarise-analysis-results/tasks.md @@ -1,165 +1,17 @@ ---- - -description: "Task list for summarising analysis results" ---- - -# Tasks: Summarise analysis results - -**Input**: Design documents from `specs/011-summarise-analysis-results/` -**Prerequisites**: [plan.md](./plan.md), [spec.md](./spec.md), [research.md](./research.md), [data-model.md](./data-model.md), [contracts/summary_endpoint.md](./contracts/summary_endpoint.md) - -Tests are requested for this feature. Three properties get them before their -implementation, because each is a claim about something *not* happening and none -of them fails visibly: what is never sent, that a dead token is distinguished from -a deleted one, and that the same token returns the same text. - -## Phase 1: Setup - -- [x] T001 Create `src/analysis/` with an `__init__.py`, separate from `src/agent/` because nothing here touches the graph or retrieval -- [x] T002 [P] Create `tests/analysis/` **without** an `__init__.py`. The task said to add one "alongside the existing `tests/api/`" and that premise was wrong: no test directory in this repository has one. Adding it put `tests/analysis` on `sys.path` as the top-level `analysis` package, shadowing `src/analysis`, and every import failed. Test basenames must also be unique for the same reason: `test_client.py` collided with `tests/reactome_mcp/test_client.py`, so this one is `test_analysis_client.py` -- [x] T003 Record the beta Analysis Service base URL in configuration rather than a literal, defaulting to beta and never production, in `src/analysis/client.py` - -## Phase 2: Foundational (blocking) - -**These block every user story. Nothing below Phase 2 can be built without them.** - -- [x] T004 Write the disclosure allow-list in `src/analysis/disclosure.py`: name every field of an `AnalysisResult` that may be sent under the `aggregate` tier, as an allow-list rather than a denial, so a new field from the Analysis Service is excluded by default -- [x] T005 [P] Test the allow-list in `tests/analysis/test_disclosure.py` by **recording the outbound request body** and asserting `summary.fileName`, `summary.sampleName` and `expression.columnNames` never appear — not by reading the summary and seeing nothing alarming. These three are user-supplied free text and are the reason the tier is an allow-list (research D5) -- [x] T006 Fetch a result by token in `src/analysis/client.py`, with a browser-like `User-Agent`, because the site's automation blocking returns a 403 with an HTML body to library user-agents and it looks exactly like an auth failure -- [x] T007 [P] Test in `tests/analysis/test_client.py` that 404 yields `not_found`, **410 yields `gone`** and **500 also yields `not_found`**, as measured: a malformed token returns 500, which the OpenAPI does not document, and treating it as a service fault would produce a `failed` state or a retry loop against a service that will answer identically every time (research D2) -- [x] T008 Read the current release from `GET /database/version` in `src/analysis/client.py`, deriving it rather than hardcoding it, because it is both the reported release and the cache-invalidation key (Principle V). **That endpoint answers text/plain and rejects `Accept: application/json` with 406** -- found by calling beta, after every mocked test passed -- [x] T009 Detect a ReactomeGSA result from `gsaMethod`/`gsaToken` in `src/analysis/client.py` and return `unsupported`, so a result we do not model is never summarised confidently (research D8) - -## Phase 3: User Story 1 — What does my result say? (P1) - -**Goal**: A reader with an analysis token gets a readable account of what the result says. - -**Independent test**: Submit a known token; the summary names the pathways the result ranks highest, describes significance the result supports, and cites each by stable id. - -- [x] T010 [US1] Build the prompt input from an aggregate result in `src/analysis/summarise.py`: top pathways with their `found`/`total`/`ratio`/`pValue`/`fdr`, the analysis type, species, and the service's own `warnings` -- [x] T011 [P] [US1] Test in `tests/analysis/test_summarise.py` that a result where nothing passes FDR produces prompt input that says so, so the model is never handed a "top pathway" framing for a null result (FR-004) -- [x] T012 [US1] Emit pathway citations as `st_id` events reusing the answer endpoint's citation shape, in `src/api/analysis_summary.py` -- [x] T013 [US1] Add the SSE endpoint in `src/api/analysis_summary.py` per [contracts/summary_endpoint.md](./contracts/summary_endpoint.md): `start` with release and analysis type, `token`, `citation`, `done` -- [x] T014 [US1] Mount the router in `bin/chat-fastapi.py` and add its prefix to the captcha exemption, as the answer endpoint's is -- [x] T015 [US1] Test over HTTP on the real mounted app in `tests/api/test_analysis_summary.py`, not by calling the handler — mounting order and middleware interact only on the served path (Principle I), which is where spec 010's route check found what isolated tests could not -- [x] T016 [P] [US1] Test that every `st_id` a summary cites appears in that result's `pathways[]`, mechanically rather than by reading, in `tests/api/test_analysis_summary.py`, so an invented or mismatched identifier fails (SC-003) - -## Phase 4: User Story 2 — Why were my identifiers not found? (P2) - -**A constraint measured 2026-09-20 that changes scenario 1.** The aggregate -result carries `identifiersNotFound` and **nothing about how many identifiers -were submitted** — the denominator lives only behind `/token/{token}/found/all`, -which returns the reader's own identifiers and is therefore the disclosing -tier. So the proportion the scenario asks for is not derivable at the default -tier, and asking for one would invent a statistic the way D9 describes. The -count is reported instead; the reader knows what they submitted. - -**Goal**: A reader learns why identifiers went unmatched and whether the result can be trusted. - -**Independent test**: Submit a token from an analysis with a deliberate identifier mismatch; the summary reports the proportion and names the likely cause. - -- [x] T017 [US2] Include `identifiersNotFound`, `pathwaysFound` and `resourceSummary` in the aggregate prompt input in `src/analysis/summarise.py`, which together explain most mismatches without disclosing anything -- [x] T018 [US2] Add the `identifiers` tier in `src/analysis/disclosure.py`, fetching `GET /token/{token}/notFound` only when the request asked for it -- [x] T019 [P] [US2] Test (in tests/api/test_analysis_summary.py, on the served path) that the `identifiers` tier is never reached without an explicit request, by asserting the not-found call is not made under the aggregate tier -- [x] T020 [US2] Test that a result with every identifier found produces a summary that says so rather than inventing a problem, in `tests/analysis/test_summarise.py` (spec US2 scenario 2) - -## Phase 5: User Story 3 — What do these numbers mean? (P3) - -**Fragility is computed in code, not asked for in the prompt.** A hit resting -on fewer than five matched entities is flagged per pathway, because a model -handed a tiny p-value and a tiny count describes the p-value. The threshold is -on the *count*, not the ratio: 2 of 3 looks perfect by ratio and is nearly -meaningless, 40 of 200 looks poor and is real evidence. - -**No per-pathway request parameter was added.** US3's independent test implies -one, but the website has not asked for it and every shown pathway already -carries its own counts, so the explanation is per-pathway without new contract -surface. - -**Goal**: The statistics are explained using the reader's own numbers. - -**Independent test**: Ask about one pathway in a result; the explanation uses that pathway's counts, not generic definitions. - -- [x] T021 [US3] Carry per-pathway counts into the prompt input for a named pathway in `src/analysis/summarise.py` -- [x] T022 [P] [US3] Test that a pathway significant by p-value but not by FDR is described as distinguishing the two, in `tests/analysis/test_summarise.py` (spec US3 scenario 2) -- [x] T023 [P] [US3] Test that a pathway with very few found entities is described as fragile, using its actual counts, in `tests/analysis/test_summarise.py` (spec US3 scenario 1) - -## Phase 6: User Story 4 — Readings specific to the analysis type (P4) - -**The type enum, read from the API on 2026-09-20 rather than guessed**: -`SPECIES_COMPARISON`, `OVERREPRESENTATION`, `EXPRESSION`, `GSA_REGULATION`, -`GSA_STATISTICS`, `GSVA`. Three are summarised; the three GSA ones are -declined. - -That reading also closed a gap in T009: `is_gsa` recognised ReactomeGSA by -its `gsaMethod` field alone, so a result carrying one of those types -*without* that field would have been summarised confidently — the outcome D8 -exists to prevent. Both signals are checked now. - -**Goal**: An expression result and a species comparison each get the reading that fits them. - -**Independent test**: Submit one of each; neither summary describes the other's kind of result. - -- [x] T024 [US4] Branch the prompt input on `summary.type` in `src/analysis/summarise.py`, and for `EXPRESSION` carry `entities.exp[]` and the value range **without** `expression.columnNames`, which is user-supplied text -- [x] T025 [US4] For `SPECIES_COMPARISON`, state in the prompt input that findings are inferred by orthology in `src/analysis/summarise.py`, so the summary cannot present them as observed -- [x] T026 [P] [US4] Test that an expression result's summary refers to behaviour across columns and a species comparison's does not, and vice versa, in `tests/analysis/test_summarise.py` - -## Phase 7: Stability and transparency (FR-014, FR-015) - -- [x] T027 Store summaries keyed `(token, release, tier)` in `src/analysis/store.py`; an aggregate summary and a disclosing one are different artefacts and must not be interchanged. **Keyed on the tier that *applied*, not the one requested** — a disclosure that could not be honoured produced an aggregate summary, and storing it under `identifiers` would serve it back later as though the identifiers had been used -- [x] T028 [P] Test in `tests/analysis/test_store.py` that the same token returns byte-identical text on a second request, and that a release change discards the stored summary — the second half matters because the Analysis Service deletes the underlying result on a release (research D2, D3) -- [x] T029 Report `cached` on the `start` event in `src/api/analysis_summary.py`, so the interface can say a summary was reused rather than implying the generator is deterministic (FR-015) -- [x] T030 Record in [research.md](./research.md) that the first increment's store is in-process and lost on deploy, and open follow-up work for a durable store — beta sets no `POSTGRES_LANGGRAPH_DB` today (research D4) - -## Phase 8: Human presence (FR-013) — UNBLOCKED 2026-09-19 - -Agreed with the website session; the shape is in [research.md](./research.md) D6. -Still do not let the plain caller token satisfy this by default: spec 010's D1 -settled that it asserts service identity and says nothing about humanity. The -claims below are additional, and their absence must refuse. - -**One dependency remains, and it is theirs**: their proxy mints caller tokens -for the answer route only. A summary route must exist before any claim can be -carried. Everything in this phase can be built and tested before that lands. - -- [x] T031 Agree with the website session how human presence is asserted — `human`, `human_iat` and `subject` claims on the caller token, minted only when their Turnstile-backed identity cookie validated. They proposed gating on the analysis token instead and withdrew it: a token proves an analysis happened, not that a person is present, and tokens travel in pasted URLs -- [x] T031a Verify `human_iat` against a **30-minute** freshness bound in `src/util/caller_token.py`, refusing an older one. They refuse to mint past the same bound, so it fails at both ends rather than relying on either alone. **Compared in whole seconds** (`int(now) - int(issued)`), matching their `nowSeconds - floor(solvedAt/1000)`: with a float clock the inclusive bound is unreachable, because a claim issued exactly 1800s ago is 1800.0003s old when checked. Caught by the test pinning the edge -- [x] T031b Key the rate limiter on `human_sub` when present, falling back to the caller identity, in `src/api/analysis_summary.py` — per-person throttling rather than per-proxy-address. Also limit per analysis token: twenty summaries of one analysis is not a scientist -- [x] T031c Test that a `human` claim with a stale `human_iat` is refused with **zero model calls**, in `tests/api/test_analysis_summary.py` — the freshness bound is the half most likely to be dropped, because the claim being present looks like success -- [x] T032 [US1] Verify the assertion in `src/util/caller_token.py` -- `human_presence_reason` checks `human` and a 30-minute `human_iat`, refusing before any model call -- [x] T033 [P] Test that a request without the assertion is refused and makes **zero model calls**, counted on a patched graph rather than inferred from timing, in `tests/api/test_analysis_summary.py` (SC-004) - -## Phase 9: Polish - -- [x] T034 [P] Bound the summary in `src/api/analysis_summary.py` as the answer endpoint is, so a stuck upstream cannot hold a connection. **The bound existed since Phase 3 and was untested until now**; the test also pins that a truncated summary is never stored, which would otherwise be served forever -- [x] T035 [P] Log an abandoned summary stream in `src/api/analysis_summary.py`, as the answer endpoint does, so a caller that starts summaries it does not want is visible -- [x] T036 Run the [quickstart](./quickstart.md) scenarios against beta with a real analysis token and record the outcome, including first-token timing. **Seven of nine pass on image `cc4315c`; two cannot be constructed** (a null result, a ReactomeGSA analysis — the `gone` token turned out to be constructible from the timestamp the token encodes) and the quickstart says so rather than marking them skipped. First token 1.8s and 2.0s on two runs of a single token — two samples, not a distribution -- [x] T037 Told the website session on 2026-09-21, once it was live on beta (image `cc4315c`) rather than when it merged. Sent: the three terminal states they must handle (`gone` distinct from `not_found`, `unsupported` for the three ReactomeGSA types, `refused` with its `reason`), the three scenarios that could not be verified live and are unit-tested only, and that summaries are lost on deploy by design - -## Dependencies - -- **Phase 2 blocks everything.** T004 (the allow-list) blocks any task that sends result content anywhere. -- T006 blocks T010, T017, T021, T024 — nothing can be summarised before a result can be fetched. -- T008 blocks T027: the release is part of the storage key. -- T013 blocks T015, T029, T034, T035. -- **T031 is agreed (2026-09-19); T031a-c and T032-T033 follow from it, and block nothing else** — every other story can be built and tested behind a refusing gate. The website adding a summary route to its proxy is the only external dependency left. -- US1 is independent. US2, US3 and US4 each build on US1's prompt-input path but are separately testable. - -## Parallel opportunities - -- T005, T007 and T009 touch different files and can run together once T004 and T006 exist. -- T011, T016, T019, T022, T023, T026 are all tests in distinct files. -- T034 and T035 are independent polish on one file and should be done together. - -## Implementation strategy - -**MVP is Phase 1, 2 and 3** — an aggregate summary of what a result says, refusing -where human presence is not asserted. That is useful on its own: a reader with a -token gets a readable account, and nothing of theirs is disclosed. - -Then US2, which is the question users actually ask most, followed by stability -(Phase 7) before the remaining stories, because a summary that changes on reload -undermines trust faster than a missing type-specific reading. - -Phase 8 can be agreed in parallel with all of it and must land before the feature -is offered to anyone. +# Summarise Analysis Results: Task-ID Index + +Historical. The step-by-step task list for this work (T001–T037, with T031a–c) +is in git history up to b0ff6ec. Every task was marked done there; T031b was +only partly done (below). Kept here are the IDs that code comments and this +folder refer to, and one repository rule found along the way. + +| ID | Task | Outcome | +|---|---|---| +| T002 | Create `tests/analysis/` | Created without an `__init__.py`. No test directory in this repository has one, and adding one made `tests/analysis` shadow `src/analysis`. Test basenames must be unique for the same reason, which is why the client tests are `tests/analysis/test_analysis_client.py` (the task named `test_client.py`) | +| T019 | Test that the `identifiers` tier is never reached without an explicit request | `tests/api/test_analysis_summary.py:307-318`, by recording the outbound not-found call | +| T031 | Agree how human presence is asserted | Agreed with the website 2026-09-19 (research D6). The task text named the third claim `subject`; the agreed and implemented claim is `human_sub` | +| T031b | Rate limit per person, and per analysis token | Partly done: the limiter keys on `human_sub`, falling back to the caller identity (`src/api/analysis_summary.py:157-164`). The per-analysis-token limit was not built, though the task was marked done | +| T034 | Bound the summary so a stuck upstream cannot hold a connection | 120 s bound (`src/api/analysis_summary.py:47`); `tests/api/test_analysis_summary.py:636-667` also pins that a timed-out summary is not stored | +| T035 | Log an abandoned summary stream | `src/api/analysis_summary.py:311-315`; `tests/api/test_analysis_summary.py:670` | +| T036 | Run the quickstart scenarios against beta | Done 2026-09-21 on image `cc4315c`; results in [quickstart.md](./quickstart.md). Scenarios 2 and 8 could not be constructed | +| T037 | Tell the website once live | Done 2026-09-21: the terminal states, the scenarios verified only by unit tests, and that summaries are lost on deploy | diff --git a/specs/012-run-gene-analysis/spec.md b/specs/012-run-gene-analysis/spec.md index e7146012..8dde563e 100644 --- a/specs/012-run-gene-analysis/spec.md +++ b/specs/012-run-gene-analysis/spec.md @@ -4,26 +4,32 @@ **Created**: 2026-09-21 -**Status**: Implemented for uploads; public-dataset path deferred +**Status**: Historical record of running ReactomeGSA from the chat, 2026-09-21 to 2026-09-22 (1678026, 0a9c5c0, 980ea94, f51283a), with fixes through 2026-10-03 (6dee089, b31669d, 48d2a7d). The upload path (Story 2) is built. The public-dataset path (Story 1) and a model-written summary (Story 3's premise) are not. The requirements below were set for that work, not for later changes. Current behaviour is in `src/gsa/`, entered from `bin/chat-chainlit.py:693-713`. **Input**: User description: "Run a gene set analysis from an uploaded expression matrix in the chat, and return the pathway results as a downloadable file" -## Context +## Context (2026-09-21) Reactome runs gene set analysis at `gsa.reactome.org` — PADOG, Camera -("similar to the classical GSEA algorithm"), ssGSEA, terapadog. The chatbot -cannot start one. The gap was found the hard way: asked to "run a GSEA with -my list of genes", the chatbot replied that Reactome could not, and offered -`fgsea` and a YouTube tutorial. +("similar to the classical GSEA algorithm"), ssGSEA, terapadog. When this spec +was written the chatbot could not start one. The gap was found the hard way: +asked to "run a GSEA with my list of genes", the chatbot replied that Reactome +could not, and offered `fgsea` and a YouTube tutorial. -Five catalogue tools were added to reactome-mcp in September (methods, data -types, dataset search, examples, sources). They let the model *describe* the -service. None of them runs anything. +Since then the chat runs a ReactomeGSA analysis on an attached matrix (this +work), and a gene list typed into the chat is offered an over-representation +analysis instead (`src/analysis/gene_list.py`; 3b2c792, 2026-09-25), which this +spec does not cover. -**Everything below was measured against the live service on 2026-09-21**, not -read off the specification. +Five catalogue tools were added to reactome-mcp in September 2026 (methods, +data types, dataset search, examples, sources). They let the model *describe* +the service; none of them ran anything. -### What the service actually requires +**The figures in this section were measured against the live service on +2026-09-21**, not read off the specification. This repository records no +re-measurement. + +### What the service required (measured 2026-09-21) `POST /analysis` takes `datasets[].data`: the whole expression matrix, inline, as a tab-delimited string. It is a required field and there is no @@ -60,109 +66,118 @@ analysis page already accepts exactly these files. A user uploading their matrix here is doing what they can already do on the website. Users know what they are uploading. -**The boundary this feature moves is the results reaching OpenAI**, and only -if the user asks for a summary. That already has a mechanism and a warning, -and this feature reuses both rather than inventing either. +**The boundary the spec identified is the results reaching OpenAI**, which +would happen only if the user asked for a summary. The spec planned to reuse +the existing allow-list mechanism and the opt-in set for analysis summaries +(spec 011) rather than invent new ones. No summary was built, so no GSA +result reaches a model; the GSA allow-list was written separately (FR-003). + +## User stories as proposed (2026-09-21) -## User Scenarios & Testing *(mandatory)* +Priorities are as the spec set them; the build order differed (see "What +shipped"). Story numbers are kept because code cites them +(`src/gsa/chainlit_flow.py:283`). -### User Story 1 — Analyse a public dataset (Priority: P1) +### User Story 1 — Analyse a public dataset (proposed P1; not built) -A user says "run a gene set analysis on GSE12345 comparing treated with -control". The chatbot loads the dataset by identifier, shows the sample +Proposed: a user says "run a gene set analysis on GSE12345 comparing treated +with control"; the chatbot loads the dataset by identifier, shows the sample groups it found, runs the analysis, and returns the significant pathways plus -a downloadable table. - -**Why this priority**: it is the whole feature minus the upload, it needs no -file handling and no disk, and it is the case a user with no data of their own -can still reach. If only this ships, the chatbot can run real analyses. - -**Acceptance**: given a public dataset identifier and a named comparison, the -chat reports the pathway count, shows the top pathways with FDR, offers the -full table as a file, and links the Pathway Browser view. - -### User Story 2 — Analyse an uploaded matrix (Priority: P2) - -A user uploads their own expression matrix through the chat UI, names the -comparison, and gets the same output. - -**Why this priority**: it is what was asked for, and it is P2 only because it -depends on the submit-and-poll machinery that Story 1 already builds. - -**Acceptance**: a matrix uploaded through the UI reaches the service and -produces results, and the file is deleted from the container once submitted. - -### User Story 3 — Results without a model summary (Priority: P1) - -A user who declines the summary still gets the pathway table and the Pathway -Browser link. - -**Why this priority**: same priority as Story 1 deliberately. The analysis is -the product; the summary is a convenience. A user must never have to send -results to a third party in order to see their own results. - -**Acceptance**: with the summary declined, the file and the link are present -and nothing has been sent to the model. - -## Requirements *(mandatory)* - -- **FR-001** The matrix MUST NOT pass through the model's context or an MCP - tool call. At 1.2 MB it would bury the answer and cost more than the - analysis. The chatbot submits it server-side. -- **FR-002** The result table MUST NOT be sent to the model whole. 2,679 rows - is the measured size of one small run; the model receives a bounded top-N. -- **FR-003** Results sent to the model MUST go through the existing - `src/analysis/disclosure.py` allow-list, **extended for ReactomeGSA's own - free-text fields**. The current `NEVER_SENT` names `fileName`, `sampleName` - and `columnNames`, which are the Analysis Service's. GSA carries the same - hazard under different names — `datasets[].name` is chosen by the user, and - `design.samples` is their column headers, which in the measured example were - `patient.id` values. An allow-list is wrong only by omission; these must be - omitted deliberately, not by luck. -- **FR-004** The existing warning MUST be shown before any result reaches the - model, and declining MUST still yield the file (Story 3). -- **FR-005** A successful `POST /analysis` MUST be treated as *accepted*, never - as *succeeded*. Measured: a submission returned HTTP 200 and then failed with - `CONNECTION_FORCED - broker forced connection closure` while the service was - mid-upgrade. The failure was visible only through `GET /status`. -- **FR-006** Uploads MUST be capped well below Chainlit's default. The config - ships `max_size_mb = 500`; the host has **4.8 GB free of 88 GB (95% used)**, - and `~/update-beta-chat.sh` already needs 6 GB to deploy. A handful of - default-sized uploads takes down the chat and blocks the next deploy. ~20 MB - covers a real matrix. -- **FR-007** An uploaded file MUST be deleted once submitted, whether the - analysis succeeds or fails. -- **FR-008** The run takes minutes, so the flow MUST NOT be shaped like a - streamed answer. The user is told it started and told again when it finishes. - -## Out of scope - -- Hosting any of this in reactome-mcp. The MCP is public, stateless and was - deliberately narrowed this month; file handling and long-running jobs belong - in the chat application. The five catalogue tools stay where they are. -- Single-cell clustering parameters (`k` for Single Cell Expression Atlas). -- Report generation: `GET /report_status/{id}` returned 404 for a completed - analysis, so whatever produces reports is not reachable this way and needs - its own investigation. - -## Open questions - -- Which method to default to. PADOG is the service's own default and is what - was measured; Camera is faster. Needs a decision informed by run time on a - realistic dataset, not by preference. +a downloadable table. The spec ranked it first because it needs no file +handling and no disk, and a user with no data of their own can reach it. +Proposed acceptance: given a public dataset identifier and a named comparison, +the chat reports the pathway count, shows the top pathways with FDR, offers +the full table as a file, and links the Pathway Browser view. + +**Status**: not built as a chat feature. `submit_public_dataset` +(`src/gsa/job.py:86-140`) loads, checks and submits a public dataset and is +tested (`tests/gsa/test_gsa_job.py:103-135`), but nothing outside the tests +calls it. The blocker recorded on 2026-09-22: reaching it needs the model to +choose a dataset and a factor from the conversation, which is conversation +design, not plumbing. Whether it is still wanted is not recorded. + +### User Story 2 — Analyse an uploaded matrix (proposed P2; built 2026-09-22) + +A user uploads their own expression matrix through the chat UI, says which +group each sample is in, and gets the same output. It was what had been asked +for; the spec ranked it P2 only because it would reuse Story 1's +submit-and-poll machinery, and it was built first. Proposed acceptance: a +matrix uploaded through the UI reaches the service and produces results, and +the file is deleted once submitted. + +**How it works now**: the attachment is checked and its sample names echoed +(`src/gsa/chainlit_flow.py:144-164`, `src/gsa/upload.py:113`). The matrix then +waits, up to 30 minutes and one per session, for the user's next message that +reads as one label per sample (`bin/chat-chainlit.py:704-713`, +`src/gsa/pending.py:22`); other messages are answered as usual. Exactly two +groups, one comparison; the first label given is group 1 +(`src/gsa/chat.py:73-122`). The method is PADOG (`src/gsa/job.py:59`) and the +dataset type is always `rnaseq_counts` (`src/gsa/chainlit_flow.py:230`). + +### User Story 3 — Results without a model summary (proposed P1) + +Proposed: a user who declines the summary still gets the pathway table and the +Pathway Browser link, and nothing is sent to the model. The spec's reason, +which still describes the design: the analysis is the product and the summary +a convenience, and a user must never have to send results to a third party in +order to see their own results. + +**Status**: moot as written. No model summary was built, so every user gets +the table and the link and nothing is sent to a model +(`src/gsa/chainlit_flow.py:278-297`). An opt-in GSA summary is proposed and not +built; what blocks it and whether it is still wanted are not recorded. + +## Requirements of this work, and where they stand (checked 2026-10-21) + +| ID | Required by the work (2026-09-21) | Today | +|---|---|---| +| FR-001 | The matrix not to pass through the model's context or an MCP tool call (at 1.2 MB it would bury the answer and cost more than the analysis); the chatbot submits it server-side | Holds. An attached `.tsv`, `.csv` or `.txt` routes to the analysis flow instead of the graph (`bin/chat-chainlit.py:693-702`, `src/gsa/chainlit_flow.py:51-53, 94-107`), and the chat server posts the matrix inline to ReactomeGSA (`src/gsa/job.py:174-183`, `src/gsa/client.py:292-310`) | +| FR-002 | The result table not sent to the model whole (2,679 rows in one small measured run); the model to receive a bounded top-N | The bounded view exists: the 20 most significant rows plus exact counts (`src/gsa/results.py:50-52, 156-198`). No GSA result is sent to a model at all; the view is used only to write the chat's own description of the result (`src/gsa/chat.py:155-196`) | +| FR-003 | Results sent to the model to go through `src/analysis/disclosure.py`, extended for ReactomeGSA's own free-text fields: `datasets[].name` (chosen by the user) and `design.samples` (their column headers, which in the measured example were `patient.id` values) | Built differently. GSA has its own allow-list in `src/gsa/results.py`: `MODEL_COLUMNS`, and `NEVER_SENT` = `fold_changes`, `design`, `samples`, `mappings` (`results.py:22-48`), tested by name (`tests/gsa/test_gsa_results.py:85-108`). `disclosure.py` was not extended; its `NEVER_SENT` is still the Analysis Service's three fields (`disclosure.py:80`). The submitted dataset is named `uploaded`, never the user's filename (`src/gsa/job.py:176`), and the Pathway Browser link, which carries the analysis token, is kept out of the model view (`results.py:201-219`; 21acf75) | +| FR-004 | The existing warning shown before any result reaches the model, and declining still to yield the file (Story 3) | Nothing to apply to: no result reaches a model, and the GSA flow has no summary or warning step | +| FR-005 | A successful `POST /analysis` treated as accepted, never as succeeded. Measured 2026-09-21: a submission returned HTTP 200 and then failed with `CONNECTION_FORCED - broker forced connection closure` while the service was mid-upgrade, visible only through `GET /status` | Holds. `submit_*` returns an ID only; success or failure is decided in `await_result` (`src/gsa/job.py:9-12, 291-355`), and a failure after acceptance is reported to the user (`src/gsa/chainlit_flow.py:264-276`) | +| FR-006 | Uploads capped well below Chainlit's default (`max_size_mb = 500`), at about 20 MB. Reason given 2026-09-21: the host had 4.8 GB free of 88 GB (95% used), `~/update-beta-chat.sh` needed 6 GB to deploy, and a handful of default-sized uploads would take down the chat and block the next deploy | Holds, in two places: `.chainlit/config.toml:65` (`max_size_mb = 20`, enforced in the browser) and `src/gsa/upload.py:23-24, 118-126` (20 MiB by default, `GSA_MAX_UPLOAD_BYTES`, enforced on the server) | +| FR-007 | An uploaded file deleted once submitted, whether the analysis succeeds or fails | Holds, and on more paths than submission: in `submit_uploaded_matrix`'s `finally` (`src/gsa/job.py:166-185`), on a refused or unreadable upload (`src/gsa/chainlit_flow.py:152-161`), after a labelled run (`chainlit_flow.py:183-186`), when a newer upload replaces it or the 500-session bound evicts it (`src/gsa/pending.py:34-46`), when it is found expired after 30 minutes (`pending.py:48-57`), and at session cleanup (`bin/chat-chainlit.py:207`). Result tables are pruned before each write, at 24 hours or 200 MB in total (`src/gsa/job.py:200-259`) | +| FR-008 | The run takes minutes, so the flow not shaped like a streamed answer: the user told it started, and told again when it finishes | Holds: a "Started" message (`src/gsa/chainlit_flow.py:252-255`), a progress line edited in place (`bin/chat-chainlit.py:216-245`), then the result and the file (`chainlit_flow.py:294-297`) | + +## Out of scope for this work (set 2026-09-21) + +- Hosting any of this in reactome-mcp. The MCP was public and stateless and had + been deliberately narrowed in September 2026, so the spec put file handling + and long-running jobs in the chat application and left the five catalogue + tools where they were. +- Single-cell clustering parameters (`k` for Single Cell Expression Atlas). Not + supported: the submission carries no parameters (`src/gsa/client.py:292-306`). +- Report generation: on 2026-09-21 `GET /report_status/{id}` returned 404 for a + completed analysis, so reports were not reachable this way and were left for + a separate investigation. `src/gsa/` has no report support. + +## Open questions (recorded 2026-09-21; no decision recorded since) + +- Which method to default to. PADOG is the service's own default, was what was + measured, and is what the chat uses (`src/gsa/job.py:56-59`); Camera is + faster. The spec wanted the choice informed by run time on a realistic + dataset rather than by preference. - Whether an analysis should survive a chat session ending. Results live behind an analysis ID at the service, so resuming is possible; whether it is - wanted is a product question. + wanted is a product question. Nothing resumes today; if waiting fails, the + chat gives the user the analysis ID (`src/gsa/chainlit_flow.py:268-276`). --- ## What shipped, 2026-09-22 -**User Story 2 (upload) and Story 3 (results without a summary) are done.** -Story 1 (public datasets by identifier) is built underneath -- -`submit_public_dataset` works and is tested -- but has no chat route yet, +**User Story 2 (upload) shipped, and Story 3's outcome holds because no summary +was built.** Story 1 (public datasets by identifier) was built underneath -- +`submit_public_dataset` works and is tested -- but has no chat route, because reaching it needs the model to choose a dataset and a factor, which -is a conversation design rather than a plumbing job. +is a conversation design rather than a plumbing job. Still the case on +2026-10-21. + +**As deployed on 2026-09-22 the upload path could not complete.** Driving it +in a browser on 2026-09-25 found four bugs the tests had missed -- among them, +every submission failed because the service answers with a bare text ID, and +the chat UI crashed on delivering the TSV. They were fixed in 6dee089. Shipping Story 2 before Story 1 inverts the spec's priorities. The reason is that the upload path turned out to be the *simpler* one: a file arrives with @@ -172,11 +187,30 @@ disambiguate. **Nothing reaches OpenAI.** Story 3 was written as "a user who declines the summary still gets their results". What shipped is that with no summary at all: the result is described from the table and attached as a file, both of -which are the user's own data returning to the user. `Finished.for_model` -is computed and sent nowhere. When a summary is added it must be given the -bounded allow-listed view, behind the existing warning -- not the result. +which are the user's own data returning to the user. `Finished.for_model` is +used only to write that description (`src/gsa/chat.py:155-196`) and is never +sent to a model. The position recorded at the time: a summary, if one is +added, should be given the bounded allow-listed view behind an opt-in, not +the result. None has been added. **FR-006 landed in two places on purpose.** `.chainlit/config.toml` caps uploads at 20 MB, and `src/gsa/upload.py` caps them again. The first stops a browser sending the file; the second stops the server accepting it. Only the second is a guarantee. + +## Later changes (2026-09-25 to 2026-10-03) + +- **Asking in words** (f615bdd). A request to run a GSA, typed rather than + attached, gets instructions to attach a matrix, instead of a model answer + saying the chat cannot do it (`src/gsa/chat.py:199-253`, + `bin/chat-chainlit.py:753-760`). +- **Labels across messages** (48d2a7d). The first version asked for labels with + Chainlit's `AskUserMessage`, which a reconnect broke. The matrix now waits in + `src/gsa/pending.py`, and the next message that reads as labels runs it. + `run_analysis` (`src/gsa/chainlit_flow.py:110`) is the old single-step flow, + still exercised by tests but no longer called by the chat. +- **Correctness fixes from review** (b31669d): the first label given is group 1 + (it was the alphabetically first); R `write.table` headers keep their first + sample; CSV uploads are converted to tabs before submission; files are + decoded the same way at validation and at submit; polling retries transient + failures up to four times, and a failure gives the user the analysis ID. diff --git a/specs/013-continue-chat-hand/spec.md b/specs/013-continue-chat-hand/spec.md index 4e6ae49b..d0a4c4e2 100644 --- a/specs/013-continue-chat-hand/spec.md +++ b/specs/013-continue-chat-hand/spec.md @@ -4,132 +4,114 @@ **Created**: 2026-09-25 -**Status**: Stories 1 and 2 built and verified end to end; Story 3 needs a shared store +**Status**: Historical record of the Continue-in-chat work, 2026-09-25 (c036dd6, a8cda58, 2eac8ed), with review fixes on 2026-10-03. Stories 1 and 2 (an analysis summary or a search-page answer, into the chat served by the same process) are built, and were verified end to end on 2026-09-25. Story 3 (into the logged-in chat) is not built. The requirements below were set for that work, not for later changes. Current behaviour is in `src/api/handoff.py`, `src/handoff/`, `public/custom.js:95-146` and `bin/chat-chainlit.py:272-319, 502-543`. -**Input**: Adam, 2026-09-25: "with the chat on both the search page and the analysis results we want there to be a button to go to the [chat] interface … two options … either the logged in version or the guest version of the chat app and have the context already be set with either the search results or the analysis summary with the summary data." And: "the chat should open in a new tab." +**Input**: Request, 2026-09-25: "with the chat on both the search page and the analysis results we want there to be a button to go to the [chat] interface … two options … either the logged in version or the guest version of the chat app and have the context already be set with either the search results or the analysis summary with the summary data." And: "the chat should open in a new tab." -## Context +## Context (2026-09-25) -The website already shows a chatbot-written summary in two places, both -served by this repository: +The website shows a chatbot-written summary in two places, both served by this +repository: -| where | endpoint | cached? | +| where | endpoint | kept? | |---|---|---| -| search page | `POST /api/answer` (spec 010) | no | -| analysis results | `POST /api/analysis/summary` (spec 011) | yes — `SummaryStore`, keyed `(token, release, tier)` | +| search page | `POST /api/answer` (spec 010) | Not when this spec was written. Since 2eac8ed (2026-09-25) each answered stream is kept for one hour under an `answer_id`, for handoff only; answers are never reused (`src/api/answer_store.py:27`) | +| analysis results | `POST /api/analysis-summary` (spec 011) | yes — `SummaryStore`, keyed `(token, release, tier)` (`src/analysis/store.py:46-54`) | -The feature is a **"Continue in chat"** button beside each summary. It opens -the chat app in a new tab, guest or logged in, with the conversation already -holding what the user was looking at, so their first message can be a -follow-up rather than a restatement. +The feature is a **"Continue in chat"** button beside each summary, opening +the chat app in a new tab with the conversation already holding what the user +was looking at, so their first message can be a follow-up rather than a +restatement. Guest and logged-in versions were requested; only the chat served +by the process that minted the handoff can be opened (Story 3). -## User Scenarios & Testing *(mandatory)* +## User stories (set 2026-09-25) -### User Story 1 — Continue an analysis summary in the guest chat (Priority: P1) +Story numbers are kept because code and tests cite them +(`src/handoff/store.py:59`, `src/api/answer_store.py:3`, +`tests/api/test_handoff_endpoint.py:249`, `tests/api/test_answer_endpoint.py:547`). + +### User Story 1 — Continue an analysis summary in the guest chat (P1; built 2026-09-25, c036dd6) A user reads the summary of their enrichment analysis, clicks "Continue in chat (guest)", and a new tab opens on the chat with that summary already in the thread. They ask "which of these pathways involve TP53?" and get an answer grounded in their analysis. -**Why this priority**: analysis summaries are already cached and carry a -disclosure tier, so the handoff has everything it needs; and the guest chat -is the one path verifiable on beta. +**Why P1 at the time**: analysis summaries were already cached and carried a +disclosure tier, so the handoff had everything it needed; and the guest chat +was the one path verifiable on beta. -**Independent test**: from a real analysis token, the new tab's thread begins -with the same summary text the website showed, and a follow-up question is -answered using the analysis. +**Check used**: from a real analysis token, the new tab's thread begins with +the same summary text the website showed, and a follow-up question is +answered using the analysis. Result under "What was built" below. -### User Story 2 — Continue a search-page answer (Priority: P2) +### User Story 2 — Continue a search-page answer (P2; built 2026-09-25, 2eac8ed) The same, from the search page's AI answer and its results. -**Why P2**: `/api/answer` does not keep what it generated, so it needs a -store before the handoff can show the *same* answer (see FR-002). - -### User Story 3 — Continue in the logged-in chat (Priority: P2) - -The same handoff into `/chat/personal/`, so the conversation is kept in the -user's history. - -**Why P2, and why it cannot be verified on beta**: `/chat/personal` is not -deployed on beta — it needs a second container, Postgres, and OAuth whose -redirect URIs are bound to reactome.org. Verifiable in production only. - -## Requirements *(mandatory)* - -- **FR-001 Pass a reference, never the content.** No search result, summary - text or analysis data in a URL. URLs are written to nginx logs, leak to - other sites through `Referer`, and have length limits. The website asks the - chatbot for a short-lived handoff ID; the link carries only that ID. - -- **FR-002 The chat continues the summary the user saw, and does not - regenerate it.** Answers are not reproducible here — the same question on - the same build scores ~0.33 similarity run to run — so a regenerated summary - would greet the user with different text from the one they clicked from. - The handoff hands over the stored text. For analysis summaries it already - exists in `SummaryStore`; for search answers a store is required first. - -- **FR-003 The disclosure choice carries over.** A summary produced at the - `aggregate` tier continues at `aggregate`: the chat model receives the same - allow-listed view the summary was built from, and nothing wider. Continuing - in chat must not silently send OpenAI more than the user agreed to on the - website. The `identifiers` tier carries over only if it was chosen there. - -- **FR-004 The handoff ID travels in the URL fragment** - (`/chat/guest/#handoff=`). Browsers never send the fragment to a - server, so the ID does not reach nginx logs or `Referer` at all — FR-001 - enforced by the browser rather than by us remembering. - -- **FR-005 The handoff is bound to the tab, not the browser.** The link opens - in a new tab (FR-008), so a user can open several. A cookie is shared by - every tab: two handoffs in quick succession could overwrite each other - before the first tab connects, and that tab would open on the *wrong* - context. The ID is instead read by a script in the tab itself and sent over - that tab's own connection (FR-006). - -- **FR-006 Mechanism (to be prototyped).** Chainlit 2.11 does not give the app - the page URL on connect — its handler reads cookies only. It does offer - `custom_js` (a script injected into the chat page) and `@cl.on_window_message` - (a server hook for messages posted in the page). The script reads the - fragment and posts the ID; the hook redeems it and seeds the thread. The - risk to prove first: the post must arrive after the tab's socket is - connected, or it is lost. The script retries until the server acknowledges. - -- **FR-007 Valid for minutes, not once.** A single-use ID would give an empty - chat on a reload. The ID is redeemable for a short window (target: 15 - minutes) and then refused. It grants read access to one summary, so the - window is short and IDs are unguessable (≥128 random bits). - -- **FR-008 New tab.** The website opens the chat with `target="_blank"` and - `rel="noopener noreferrer"`, so the chat tab cannot reach back into the - website tab and receives no `Referer`. - -- **FR-009 An expired or unknown ID says so.** The chat opens normally and - tells the user the context could not be loaded, rather than silently - starting an empty conversation they will assume has the context. - -## Who builds what +**Why P2 at the time**: `/api/answer` did not keep what it generated, so it +needed a store before the handoff could show the *same* answer (FR-002). That +store is `src/api/answer_store.py`. + +### User Story 3 — Continue in the logged-in chat (proposed P2; not built) + +Proposed: the same handoff into `/chat/personal/`, so the conversation is kept +in the user's history. + +**Status**: not built. It is blocked by in-process state: the handoff store +and the summary cache live in each process's memory +(`src/handoff/store.py:22-24`, `src/api/analysis_summary.py:55`), and the guest +and logged-in chats are separate processes, so a handoff minted by one cannot +be claimed by the other. `POST /api/handoff` therefore returns only the +minting process's own chat path (`src/api/handoff.py:93-95, 169-181`). The two +ways forward recorded on 2026-09-25 are under "What was built" below; no +choice between them is recorded, nor whether Story 3 is still wanted. + +As recorded on 2026-09-25, it could not be verified on beta: `/chat/personal` +was not deployed there, needing a second container, Postgres, and OAuth whose +redirect URIs are bound to reactome.org. + +## Requirements of this work, and where they stand (checked 2026-10-21) + +| ID | Required by the work (2026-09-25) | Today | +|---|---|---| +| FR-001 | Pass a reference, never the content: no search result, summary text or analysis data in a URL, because URLs reach nginx logs, leak through `Referer` and have length limits. The website asks for a short-lived handoff ID and the link carries only that | Holds: `POST /api/handoff` returns `id`, `expires_in` and `path` (`src/api/handoff.py:168-182`); the content stays in the server-side store | +| FR-002 | Continue the summary the user saw, from stored text, not regenerated: the same question on the same build scores ~0.33 similarity run to run (measured 2026-09-18 for spec 010), so a regenerated summary would differ from the one clicked from | Holds. An analysis handoff copies the text stored for that token, release and tier, and is refused if none is stored (`src/api/handoff.py:149-166`); a search handoff copies the answer kept under `answer_id` (`handoff.py:129-140`). The copy is taken at mint (`src/handoff/store.py:3-14`), and the reader sees it verbatim (`src/handoff/seed.py:139-155`) | +| FR-003 | The disclosure tier carries over: an `aggregate` summary continues at `aggregate`, and the chat model receives the same allow-listed view the summary was built from, nothing wider; `identifiers` only if chosen on the website | Holds. The tier is part of the summary lookup, so a tier the reader did not choose finds nothing (`src/api/handoff.py:149-157`); it is stored on the handoff (`src/handoff/store.py:49-51`), and the data is rebuilt at that tier with the summary endpoint's own functions (`src/handoff/seed.py:64-82`). Search handoffs carry no tier (public pathway text). Web search is skipped on a seeded analysis thread (see "Later changes") | +| FR-004 | The handoff ID travels in the URL fragment (`/chat/guest/#handoff=`), which browsers never send to a server | Holds: the path is `/#handoff=` for the minting process's own chat (`src/api/handoff.py:93-95, 180`), read by `public/custom.js:126-128`. Added 2026-09-25 (a8cda58): a first-time visitor's captcha form POST dropped the fragment, so the captcha page stashes it in `sessionStorage` and `custom.js` restores it (`bin/chat-fastapi.py:246-257`, `public/custom.js:109-124`) | +| FR-005 | The handoff is bound to the tab, not the browser: a cookie is shared by every tab and two quick handoffs could seed the wrong one, so a script in the tab reads the ID and sends it over that tab's own connection | Holds (`src/handoff/window.py:7-13`, `public/custom.js:126-145`) | +| FR-006 | Mechanism, to be prototyped: Chainlit 2.11 gives the app no page URL on connect, so `custom_js` reads the fragment and posts the ID, and `@cl.on_window_message` redeems it and seeds the thread; the script retries until acknowledged, because a post before the socket connects is lost | Built as described. `custom_js` (`.chainlit/config.toml:94`) posts every 500 ms, up to 40 times (about 20 s), until acknowledged (`public/custom.js:130-145`); the hook claims and acknowledges (`bin/chat-chainlit.py:502-543`). Chainlit is still 2.11.0 (`poetry.lock`). Measurements under "What was built" | +| FR-007 | Valid for minutes, not once: a single-use ID would give an empty chat on reload. Redeemable for a short window (target 15 minutes), then refused; IDs unguessable (at least 128 random bits) | Holds: 15 minutes (`src/handoff/store.py:37`), claimable repeatedly until expiry (`store.py:96-104`), 192-bit IDs (`store.py:75-77`). The store holds at most 2,048 handoffs and is lost on restart (`store.py:22-24, 40`) | +| FR-008 | New tab: the website opens the chat with `target="_blank"` and `rel="noopener noreferrer"`, so the chat tab cannot reach back and receives no `Referer` | Website code; not verifiable from this repository | +| FR-009 | An expired or unknown ID says so, rather than starting an empty conversation the user will assume has the context | Holds: `seed.UNAVAILABLE` (`src/handoff/seed.py:158-162`) is sent when the handoff is missing or seeding fails (`bin/chat-chainlit.py:281-285, 307-309`) | + +## Who was to build what (2026-09-25) + +This repository's part is built (see the requirements table). The website's +part is not visible from here. | | repo | |---|---| | the two buttons, the new-tab link, requesting a handoff ID | WebsiteAngular | | `POST /api/handoff`, the store, the fragment script, the hook that seeds the thread | reactome_chatbot | -## Out of scope +## Out of scope for this work (set 2026-09-25) - Seeding the chat with the raw search results or the full analysis result. The chat receives what the summary was built from, bounded as it already - is, and can fetch more through its own tools. + was (`src/handoff/seed.py:64-136`). Its tools can add pathway detail, but none + fetches the analysis by token (`src/reactome_mcp/tools.py:54-97`). - Carrying a handoff across devices or accounts. -## Open questions +## Open questions (recorded 2026-09-25) -- Whether the button should be two buttons (guest / logged in, as asked) or - one that the chat resolves after login. Two is what was asked for; recorded - so the choice is visible, not reopened. -- The handoff window length (FR-007) — 15 minutes is a guess to be revisited - once real use shows how long people take. +- Whether the website shows two buttons (guest / logged in, as requested) or + one that the chat resolves after login. Two were requested. With Story 3 + unbuilt, a logged-in button from the guest deployment cannot work; what the + website shows is not visible from this repository. +- The handoff window length (FR-007). 15 minutes was a guess, to be revisited + once real use showed how long people take. It is unchanged + (`src/handoff/store.py:37`), and no measurement of real use is recorded. --- @@ -156,9 +138,9 @@ logged-in chats are separate processes. A handoff minted by one cannot be claimed by the other. The endpoint therefore returns only `path`, for the chat served by the process that minted it; the first version also returned a `personal_path` from the guest deployment, which could only ever have opened -on "couldn't load the summary". Two ways to do Story 3, to decide later: +on "couldn't load the summary". Two ways to do Story 3 were recorded; no choice between them is recorded: -- a store both processes share (production already has Postgres for the +- a store both processes share (production already had Postgres for the logged-in chat's LangGraph checkpoints), with the summary cache moved into it too, so a handoff can cross processes and still continue the *same* summary; or @@ -184,3 +166,21 @@ Verified end to end through the real Turnstile gate as a first-time visitor: a real answer, a handoff minted with no human claim, the tab opens on the question and the same answer, and "which protein kinase were we just discussing?" is answered "CDK5". The control, with no handoff, cannot say. + +## Later changes (2026-10-03 review) + +- **One claim per session** (5b28d55). Any page can post into the + window-message channel, so a session redeems only its first handoff ID; + later IDs are acknowledged and ignored (`bin/chat-chainlit.py:516-543`). +- **Messages wait for the seed** (5f5a7c0). A message sent while a handoff is + being seeded waits up to 45 seconds for it, so the seed is not lost beside a + running turn (`bin/chat-chainlit.py:617-629, 672`). +- **What the model is told** (d153fbf). The seeded analysis turn carries the + summary endpoint's data rules, and says whether the data could not be + fetched or is gone (`src/handoff/seed.py:64-136`). +- **No web search on an analysis thread** (1b8c939; read from the thread + itself since 14ca5cd). The rephrased question sent to web search is built + from history that holds the reader's analysis, so post-processing is skipped + on a seeded thread (`bin/chat-chainlit.py:561-566`, `src/agent/graph.py:425`). +- **Escaping** (1b8c939, 5f5a7c0). The question and summary shown to the + reader are escaped or rendered inert (`src/handoff/seed.py:139-155`).