Spec 004: answer in the user's language

specs/004-answer-in-user-language/checklists/requirements.md

@@ -0,0 +1,64 @@
+# Specification Quality Checklist: Answer in the User's Language
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-09-10
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Adversarial review of this specification
+
+Run before trusting it. Two load-bearing claims were checked against the corpus
+rather than reasoned about, and **one of them was wrong**.
+
+| claim as first written | verdict |
+|---|---|
+| "BM25, which is lexical, retrieves close to nothing for a French query" | **wrong.** It returns a full ten documents; named entities survive translation. The real cost is that 7 of 10 differ. Corrected, with the measurement. |
+| "#140's appended instruction pollutes retrieval" | **right, but first measured wrongly.** BM25 alone gives 0/10 — dramatic and irrelevant, because the query expander sits in front of it and four of five queries reach BM25 clean. Through the real retriever: **20/40 and 21/40 survive**, so about half the context changes. Corrected after Adam pushed back on the number looking too low. |
+| "#140 targets React-to-Me" | right, but it edits `call_model`, which main renamed to `generate_answer`. Added. |
+| "Plant Reactome's answer does not receive the language" | verified in `plantreactome.py`. |
+| "Cross-Database's summary does" | verified in `cross_database.py`. |
+| "#125 keeps the translate-to-English step" | verified in its diff. The obvious mistake, not made. |
+
+The wrong claim mattered: it would have justified FR-004 with a reason a reader could
+disprove in five minutes, which is worse than justifying it with the smaller, true
+one.
+
+And the review itself needed a second pass. It checked whether each claim was true
+but not whether the *test* measured the product — so it confirmed a component
+measurement and reported it as a pipeline one. That is the same failure
+`evaluator.py` had, four days apart. **An adversarial review has to attack the
+measurement as well as the claim.**
+
+## Notes
+
+Deviations from the template, deliberate:
+
+1. **Two contributed PRs are reviewed in the spec body.** They are the starting
+   material and the reason to reject each is inseparable from the reason to read it.
+2. **Measurements appear in a specification.** Constitution Article II says retrieval
+   claims come with numbers. The numbers are what make the choice between the two
+   approaches obvious rather than aesthetic.

specs/004-answer-in-user-language/spec.md

@@ -0,0 +1,292 @@
+# Feature Specification: Answer in the User's Language
+
+**Feature Branch**: `spec/answer-in-user-language`
+
+**Created**: 2026-09-10
+
+**Status**: Draft. Two contributed PRs to harvest; one decision (D1) for the team.
+
+**Input**: Issue #104, "RAG only responds in English". Two competing pull requests, #125 and #140.
+
+## The gap, precisely
+
+Language detection already works. `detect_language` runs on every message, concurrently
+with the safety check, and `detected_language` is on `BaseState` for every profile.
+
+It reaches some places and not others:
+
+| where | uses the detected language? |
+|---|---|
+| refusal messages (React-to-Me, Plant Reactome) | **yes** |
+| Cross-Database final summary | **yes** |
+| **React-to-Me's answer** | **no** |
+| **Plant Reactome's answer** | **no** |
+
+So a question in French is detected as French, rephrased into English for retrieval,
+answered in English — and if it had instead been *refused*, the refusal would have
+come back in French. The two profiles that serve real users are the two that answer
+in the wrong language; the one that gets it right is the prototype.
+
+That asymmetry is the whole feature. Nothing needs detecting, routing or storing that
+is not already detected, routed and stored.
+
+Reproduced against the Release95 bundle, running exactly what `generate_answer` does:
+
+| step | result |
+|---|---|
+| question | *Quel role joue TP53 dans l'apoptose ?* |
+| detected language | **French** |
+| rephrased for retrieval | *What role does TP53 play in apoptosis?* |
+| retrieval | good — the English rephrasing does its job |
+| **answer** | **English** |
+
+Four steps of five work. The detector is right, the translation is right, retrieval
+is right, and then the answer comes back in the wrong language.
+
+One detail makes it unambiguous: `chat_history` still holds the original French
+question, and the model answered in English anyway. It is not that the model cannot
+tell what language was used — nothing asks it to reply in that language.
+
+## The constraint that shapes the design
+
+**Retrieval must stay in English.** The Reactome corpus is English: the CSV rows BM25
+tokenises, and the vectors built from them.
+
+Measured on the `summations` collection rather than assumed, because the obvious
+version of this claim is wrong. BM25 does **not** collapse on a non-English query —
+it still returns a full ten documents, because named entities survive translation:
+
+| query | documents shared with the English form |
+|---|---|
+| *Quel role joue TP53 dans l'apoptose ?* | 3 of 10 |
+| *Wie wird die Glykolyse reguliert?* | 3 of 10 |
+
+So the cost of retrieving in French is that **seven of ten documents change**, not
+that retrieval fails. That is still a large enough change to be worth avoiding, and
+the rephrase step already translates to English for exactly this reason — but the
+reason is "materially different results", not "no results".
+
+So the language instruction has to reach the **answer** without reaching the
+**retrieval**. That is the entire design question, and it is where the two
+contributed PRs diverge.
+
+## Two contributed pull requests
+
+Both from GSoC applicants who will not update them. Two people arriving at the same
+issue independently is strong evidence it is worth fixing.
+
+### #140 — @bleedblack1
+
+Targets **React-to-Me**, which is the right profile: it is the one users use, and
+the one with the gap. The intent is exactly right.
+
+The mechanism is not. It appends the language instruction to `input`:
+
+```python
+query = f"{query}\n\n[CRITICAL INSTRUCTION: You MUST write your entire response in {detected_language}...]"
+result = await self.reactome_rag.ainvoke({"input": query, ...})
+```
+
+`create_retrieval_chain` passes `input` **straight to the retriever**. Verified in
+`langchain_classic`: *"it is expected that an `input` key be passed in — this is what
+will be used to pass into the retriever."* So that 61-word block of English prose
+about response languages becomes part of the BM25 query and the embedded vector, for
+every non-English question.
+
+Measured through the **whole retriever**, appending it to an English question:
+
+| query | fused documents surviving the appended instruction |
+|---|---|
+| *What role does TP53 play in apoptosis?* | 20 of 40 |
+| *Which complexes contain EGFR?* | 21 of 40 |
+
+**About half the context changes.** That is the number to argue from.
+
+It is worth saying what the first version of this section got wrong, because the
+mistake is instructive. Measuring BM25 *directly* on the polluted string gives 0 of
+10, 1 of 10, 0 of 10 — sixty-one words of instruction outweigh a six-word question,
+so lexical ranking collapses entirely. But BM25 never sees that string in production:
+`HybridRetriever` expands the query into four LLM-generated alternates first and
+appends the original last, so four of the five queries are clean rewrites and the
+fusion recovers most of the damage.
+
+The component number was dramatic and irrelevant; the pipeline number is half, and
+real. Constitution Article I, arrived at the hard way — twice in one week, after
+`evaluator.py` measured a retriever the product no longer used.
+
+Half the retrieved context silently differing for non-English users is still reason
+enough to reject the mechanism. The instruction belongs in the answer prompt, where
+it changes nothing about what is retrieved.
+
+A second, smaller problem: #140 edits a method called `call_model`, which `main`
+renamed to `generate_answer`. The patch does not apply as written.
+
+It also carries the answer to a real sub-problem, and gets it right: gene symbols,
+protein names, `R-HSA-*` identifiers and URLs must survive untranslated. That
+requirement is kept.
+
+### #125 — @bhavyakeerthi3
+
+Strengthens the **Cross-Database summarizer** prompt, where `{detected_language}` is
+already a template variable — the clean mechanism, applied to the profile that
+already worked. It also rewrites the rephrase prompt while **keeping** the
+translate-to-English step, which is correct and worth noting because removing it
+would have been the obvious mistake.
+
+Its problem is scope. The same PR adds a hallucination grader and web-search wiring
+to Cross-Database, overlapping #123, so the language change cannot be taken without
+taking a second feature that needs its own judgement.
+
+### What each contributes
+
+| | #140 | #125 |
+|---|---|---|
+| right profile | **yes** | no (prototype only) |
+| right mechanism | no — pollutes retrieval | **yes** — prompt variable |
+| protects nomenclature | **yes** | partly |
+| self-contained | **yes** | no — bundles #123's feature |
+
+Neither is mergeable as it stands. Between them they contain the whole answer:
+**#140's target and its nomenclature rule, #125's mechanism.**
+
+## User Scenarios & Testing *(mandatory)*
+
+### User Story 1 — A question in French is answered in French (Priority: P1)
+
+A researcher asks React-to-Me a question in French. Retrieval happens in English, and
+the answer comes back in French, with gene names and Reactome links untouched.
+
+**Why this priority**: It is the issue. Everything else here is a refinement of it.
+
+**Independent Test**: Ask the same question in English and in French. The retrieved
+documents should be substantially the same; the answers should differ only in
+language.
+
+**Acceptance Scenarios**:
+
+1. **Given** a question in French, **When** it is answered, **Then** the answer is in
+   French and retrieval used the English rephrasing, so the documents are the set the
+   English rendering retrieves.
+2. **Given** a question in English, **When** it is answered, **Then** nothing about
+   the behaviour differs from today — no extra instruction, no extra call.
+3. **Given** an answer in any language, **When** it names a gene, protein, pathway or
+   `R-HSA-*` identifier, **Then** that term appears in its original English form.
+4. **Given** an answer in any language, **When** it cites Reactome, **Then** the URL
+   is unchanged and still resolves.
+
+---
+
+### User Story 2 — Retrieval quality does not move (Priority: P1)
+
+A developer changes the language handling and can show that what reaches the model is
+unchanged for English questions.
+
+**Why this priority**: Equal to Story 1, because the plausible way to implement Story
+1 is the way #140 does it, and that quietly degrades retrieval for exactly the users
+the feature is meant to help. A feature that answers in French by retrieving worse
+has not helped anyone.
+
+**Independent Test**: `bin/retrieval_baseline` before and after. English questions
+must be unchanged beyond the known ANN noise floor.
+
+**Acceptance Scenarios**:
+
+1. **Given** the change, **When** the baseline is compared, **Then** English
+   retrieval differs only within the documented run-to-run variance.
+2. **Given** a non-English question, **When** retrieval runs, **Then** the text sent
+   to BM25 and the vector store contains no instruction prose — only the rephrased
+   question.
+
+---
+
+### User Story 3 — Plant Reactome behaves the same way (Priority: P2)
+
+The Plant Reactome deployment answers in the user's language too.
+
+**Why this priority**: The same gap, the same fix, a second profile. P2 only because
+the Reactome deployment is the larger audience.
+
+**Acceptance Scenarios**:
+
+1. **Given** a non-English question to Plant Reactome, **Then** the answer is in that
+   language, with nomenclature preserved.
+
+### Edge Cases
+
+- **The detector is wrong.** It returns a single label from one LLM call on the raw
+  input; a short question, or one mixing languages, can be misread. The cost is an
+  answer in the wrong language, which the user can see and rephrase — acceptable, but
+  it argues against building anything expensive on top of the label.
+- **A language the model answers poorly.** The instruction is best-effort; there is
+  no verification step and this specification does not add one.
+- **Nomenclature that is also an ordinary word.** Gene symbols like `SET`, `MAX` or
+  `CAT` are English words. A translator may render them as words rather than symbols.
+- **A question already in English.** Must take exactly today's path, with no added
+  instruction — otherwise every existing user pays for a feature they do not use.
+
+## Requirements *(mandatory)*
+
+### Functional Requirements
+
+- **FR-001**: React-to-Me MUST answer in the language its question was asked in.
+- **FR-002**: Plant Reactome MUST do the same.
+- **FR-003**: The text used for retrieval MUST contain only the rephrased question —
+  no language instruction, no prompt scaffolding.
+- **FR-004**: Retrieval MUST continue to happen in English.
+- **FR-005**: Scientific nomenclature — gene symbols, protein names, pathway names,
+  `R-HSA-*` identifiers — MUST NOT be translated.
+- **FR-006**: URLs and citation links MUST NOT be translated or altered.
+- **FR-007**: An English question MUST follow exactly the current path, with no
+  additional model call and no additional prompt content.
+- **FR-008**: The language MUST reach the answer prompt as its own input, not
+  concatenated into another field.
+
+### Key Entities
+
+- **Detected language**: one label per message, already on `BaseState`, produced
+  concurrently with the safety check.
+- **Rephrased question**: the English rendering, already produced, already used for
+  retrieval.
+- **Answer prompt**: per profile; the place the language belongs.
+
+## Success Criteria *(mandatory)*
+
+- **SC-001**: A question asked in French, German, Spanish or Japanese is answered in
+  that language by React-to-Me.
+- **SC-002**: `bin/retrieval_baseline` shows English retrieval unchanged beyond the
+  documented noise floor.
+- **SC-003**: For a non-English question, the retrieval query is byte-identical to
+  what the same question produces today.
+- **SC-004**: In a non-English answer, every gene symbol, pathway name and `R-HSA-*`
+  identifier appears in English, and every Reactome URL resolves.
+- **SC-005**: An English question costs the same number of model calls as today.
+
+## Decisions for the team
+
+### D1 — Instruct the answer prompt, or translate afterwards?
+
+| option | what it means |
+|---|---|
+| **A. A language variable in the answer prompt** (recommended) | The answer is generated in the target language directly. No extra call, no extra latency. It is the mechanism Cross-Database already uses successfully, so the pattern is proven in this codebase. Quality depends on the model answering well in that language. |
+| B. Answer in English, then translate | A separate translation step over the finished answer. Easier to protect nomenclature — a translator can be told to leave marked spans alone — but adds a model call and its latency to every non-English message, and translation of a scientific answer can introduce its own errors. |
+| C. Both, chosen per language | Honest about models being better at some languages than others, and a table nobody will maintain. |
+
+**Recommendation: A.** It is what already works for Cross-Database, it costs nothing
+extra, and B's advantage on nomenclature is reachable from A with prompt instructions —
+which is what both contributors did, independently.
+
+## Assumptions
+
+- The existing detector is good enough. It is already trusted for refusals, which are
+  the more sensitive case; nothing here makes it more load-bearing.
+- Users would rather have an answer in their own language than a marginally better
+  English one. If that is wrong, the feature is wrong, not the implementation.
+- The corpus stays English. Translating Reactome content is a different project.
+
+## Out of Scope
+
+- Translating the corpus, or multilingual embeddings.
+- Verifying that the answer really is in the requested language.
+- The web-search and hallucination-grading work bundled into #125 — that is #123's
+  decision to make.
+- The user interface: language is detected per message, not chosen in a setting.