Spec 004: answer in the user's language - #204
Merged
Merged
Conversation
Issue #104, and the two contributed PRs that address it. The gap is narrower than "no multi-language support". Language is already detected on every message and already used for refusal messages and the Cross-Database summary. It is the two profiles that serve real users -- React-to-Me and Plant Reactome -- whose ANSWERS ignore it. A question in French is detected as French, rephrased to English, answered in English; had it been refused instead, the refusal would have come back in French. Between them the two PRs contain the whole answer and neither is mergeable. #140 picks the right profile and the right nomenclature rule but appends the instruction to `input`, which create_retrieval_chain passes straight to the retriever. #125 uses the right mechanism -- a prompt variable, as Cross-Database already does -- on the wrong profile, and bundles a hallucination grader that overlaps #123. Adversarial review before trusting it, and it earned its keep: one load-bearing claim was wrong. "BM25 retrieves close to nothing for a French query" is false -- it returns a full ten documents, because named entities survive translation. The true cost is that seven of ten differ, which still justifies retrieving in English, but for a smaller reason than the one first written. The other claim was understated. Appending #140's 61-word block to an English question leaves 0/10, 1/10 and 0/10 documents surviving on summations: it does not degrade lexical retrieval, it replaces it, because sixty-one words of instruction outweigh a six-word question. Both measurements are in the spec, per Article II, and the review is recorded in the checklist so the corrected claim cannot quietly revert. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adam pushed back that 0/10 looked too low, and he was right. That number came from running BM25 directly on the polluted string. BM25 never sees that string: HybridRetriever expands the query into four LLM-generated alternates and appends the original last, so four of five queries reach BM25 clean and the fusion recovers most of the damage. Measured through the whole retriever, 20 of 40 and 21 of 40 documents survive. Half the retrieved context silently differing for non-English users is still reason enough to reject appending the instruction to `input`. But the honest number is half, not all, and the spec now argues from it. The review that produced the wrong number checked whether the claim was true without checking whether the test measured the product -- the same failure evaluator.py had four days ago. Recorded in the checklist, because an adversarial review has to attack the measurement as well as the claim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Shows what a non-English user actually gets today, rather than describing it. Detection, translation and retrieval are all correct; the answer comes back in English because detected_language never reaches the answer prompt. chat_history still holds the original French question and the model answered in English anyway, which rules out 'the model cannot tell' -- nothing asks it to reply in that language. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #104, and the two contributed PRs that address it.
The gap is narrower than "no multi-language support"
Language is already detected on every message, and already used for refusal messages and the Cross-Database summary. It is the two profiles that serve real users whose answers ignore it:
A question in French is detected as French, rephrased to English, answered in English. Had it been refused instead, the refusal would have come back in French. The profile that gets it right is the prototype.
Between them, the two PRs contain the whole answer
Neither is mergeable as it stands. #140's target and nomenclature rule, #125's mechanism.
The adversarial review earned its keep
I ran one before trusting this, and one load-bearing claim was wrong:
False. It returns a full ten documents — named entities survive translation. Measured: "Quel role joue TP53 dans l'apoptose ?" shares 3 of 10 documents with the English form. The true cost is that seven of ten differ, which still justifies retrieving in English, but for a smaller reason than I first wrote. That mattered: it would have justified FR-004 with a reason a reader could disprove in five minutes.
The other claim was understated. Appending #140's 61-word instruction block to an English question, on
summations:It does not degrade lexical retrieval — it replaces it. Sixty-one words of instruction outweigh a six-word question, so BM25 ranks on the instruction.
Both measurements are in the spec per Article II, and the review is recorded in the checklist so the corrected claim cannot quietly revert.
One decision for you (D1): instruct the answer prompt directly, or translate afterwards. Recommendation is the former — it is what already works for Cross-Database and costs no extra call.
🤖 Generated with Claude Code