Skip to content

Close the eight findings from the expanded-catalog review - #18

Merged
arst merged 1 commit into
mainfrom
fix/review-followups
Sep 1, 2026
Merged

Close the eight findings from the expanded-catalog review#18
arst merged 1 commit into
mainfrom
fix/review-followups

Conversation

@arst

@arst arst commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Applies every finding from the review of the expanded catalog, in the order it recommended. All eight were verified against the source before changing anything — each one was real.

The two that were actual bugs

1. ProactiveClarification never applied the clarification answer. stillMissing was computed from the original triage only, so a user who answered "Berlin, next Tuesday, 3 nights, max €150" was still told all four slots were unknown, and the booker "assumed" what it had just been told. The reply is now parsed per slot and merged through the gate.

One deviation from the review, and the live run is why. The suggested rule — an answer may only fill a slot that was asked about — also discards a volunteered budget after the budget question was cut by the three-question cap, and the booker then invents €90 despite being told €150. Discarding information the user gave is the same failure the pattern exists to avoid, one step later. So volunteered slots merge; what the gate refuses is a reply silently rewriting a slot the request had already settled.

2. DualLlm implied taint made the value safe. It does not. The injected 48000.00 is a well-formed decimal inside the range bound and would have filed — control flow intact, data flow corrupted. Added an unattended value limit that applies because a value is tainted, and the run now pushes the injected figure through both gates and prints the type check passing and the policy refusing:

=== If the quarantined model had returned the injected figure ===
  coerces to a valid decimal: True
  value policy: EUR 48,000.00 exceeds the EUR 10,000.00 unattended limit …

Taint stops untrusted data from becoming instructions. Taint does not turn untrusted data into true facts.

Modelling and taxonomy

3. EventDrivenAgents claimed a bounded channel and used CreateUnbounded. The bound is real but lives in the host counters, so the comment was what was wrong — a bounded channel bounds in-flight events (backpressure) and its overflow modes block or silently drop, neither of which is what this needs. Also split TerminalEvents from DeadLetters with typed Refusal reasons (NoSubscriber, GenerationLimit, RunBudgetExceeded).

4. MemoryPoisoningPrevention judged corroboration by trust class, which fails in both directions: a scraper re-reading its seed page counted as independent, and two unrelated publishers could not corroborate at all. Source is now (Id, Trust) and independence is counted over evidence identity. The demo now plants exactly that pair — same URL via a different mechanism (stays quarantined), then a contract record (promotes).

5. MemoryConsolidation deleted its source episodes, so a slightly wrong summary became canonical and unfalsifiable. Episodes are archived with their ids recorded on the semantic memory; retrieval and ripeness filter to Active.

6. GraphRAG lost provenance at the summariser: free-text summaries fed an answerer told to cite incident ids it might not have. Summaries are structured, and claimed source ids are checked against the graph rather than believed — a fabricated id is reported and dropped.

Overclaims

7. ChainOfVerification called a same-model re-ask an "independent measurement" and told the reviser "verification wins". It is a blind cross-check: disagreement is strong evidence, agreement is weak, and a shared misconception survives both. The verifier now signals CONFIDENT:/UNCERTAIN:, the reviser resolves into correct / contested / leave-alone, and coverage is reported so a partially checked answer says so instead of shipping under a "Verified answer" header.

8. LeastToMost carried answers forward as established facts with nothing checking them. Added a deterministic checkpoint where one exists — StepChecks.BillingTotal recomputes the schedule from the problem's own rules, not a hardcoded 144 — with one retry and a contested outcome, and [no verifier for this step] everywhere else. The lesson is now externalising state makes validation possible, not externalised state is safer.

Smaller

  • GraphOfThoughts enforces the six-sentence brief in host code (LengthPolicy) rather than trusting the scorer's instruction. Live run shows a 7-sentence draft capped 0.95 → 0.60 with its reason.
  • SpeculativeToolExecution drops the blanket "below 50%, don't" — break-even depends on latency value, call cost, rate limits and spare concurrency.
  • AgentRegistry documents that a verified card is discovery-time identity, not proof the peer controls it at connection time; transport identity still needs binding to the card.

Verification

  • 440 tests pass (30 new, covering every fix — including the two directions of the evidence-independence bug and the injected-decimal case).
  • Three pre-existing tests were updated because the API changes made them wrong, one of which had been asserting the dead-letter taxonomy this PR corrects.
  • All ten changed samples re-run against a live deployment. Pattern Explorer still serves 74 patterns.

Not done, per the review's own framing: no large refactor, and the observations on State Machine Agent, Control Plane, Mixture of Agents, Contrastive Explanation and Human-on-the-Loop needed no change.

🤖 Generated with Claude Code

https://claude.ai/code/session_0161UFxvL3zPhufoYaQh27Ss

ProactiveClarification never applied the clarification answer. `stillMissing`
was computed from the original triage only, so a user who supplied every slot
was still told none were known and the booker "assumed" what it had just been
told. The reply is now parsed per slot and merged through the gate.

The merge guard is narrower than the review proposed, and the live run is why:
rejecting every answer to a question that was not asked also discards a
volunteered budget after the budget question was cut by the three-question cap,
and the booker then invents a worse value. Volunteered slots merge; what the
gate refuses is a reply silently rewriting a slot the request already settled.

DualLlm implied taint made the extracted value safe. It does not. The injected
EUR 48,000 is a well-formed decimal inside the range bound and would have
filed - control flow intact, data flow corrupted. Added an unattended value
limit that applies because a value is tainted, and the run now pushes the
injected figure through both gates to show the type check passing and the
policy refusing. Taint stops data becoming instructions; it does not make data
true.

EventDrivenAgents claimed a bounded channel and used CreateUnbounded. The bound
is real but lives in the host counters, so the comment was the thing that was
wrong. Also split TerminalEvents from DeadLetters with typed Refusal reasons -
a workflow output filed as a delivery failure makes the queue useless as an
alarm, which matters when this composes with AgentCommunicationFaultTolerance.

MemoryPoisoningPrevention judged corroboration by trust class, which fails both
ways: a scraper re-reading its seed page counted as independent, and two
unrelated publishers could not corroborate at all. Source is now (Id, Trust) and
independence is counted over evidence identity.

MemoryConsolidation deleted its source episodes, so a slightly wrong summary
became canonical and unfalsifiable. Episodes are archived with their ids
recorded on the semantic memory; retrieval and ripeness filter to Active.

GraphRAG lost provenance at the summariser: free-text summaries fed an answerer
told to cite incident ids it might not have. Summaries are now structured, and
the claimed source ids are checked against the graph rather than believed.

ChainOfVerification called a same-model re-ask "independent measurement" and
"verification wins". It is a blind cross-check: disagreement is strong evidence,
agreement is weak. The verifier now signals CONFIDENT/UNCERTAIN, the reviser
resolves into correct/contested/leave, and coverage is reported so a partially
checked answer says so.

LeastToMost carried answers forward as established facts with nothing checking
them. Added a deterministic checkpoint where one exists - the billing schedule
recomputed from the problem's own rules - with one retry and a contested
outcome, and "[no verifier for this step]" everywhere else.

Smaller: GraphOfThoughts enforces the six-sentence brief in host code rather
than trusting the scorer's instruction; SpeculativeToolExecution drops the
blanket 50% break-even rule; AgentRegistry documents that a verified card is
discovery-time identity, not proof the peer controls it at connection time.

440 tests pass (30 new). All ten changed samples re-run against a live
deployment; Pattern Explorer still serves 74 patterns.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0161UFxvL3zPhufoYaQh27Ss
@arst
arst merged commit cde9f93 into main Sep 1, 2026
2 checks passed
@arst
arst deleted the fix/review-followups branch September 1, 2026 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant