Repository navigation
fix(agent): a run can report success for work it never did #3983
Description
The agent can finish a task by saying it finished. Nothing in the loop requires it to have observed the result of what it changed, so a run that never called a write tool can end with "Done. Wrote weekly_build_times.csv" — and report success.
This is measured twice, by different methods, with the same answer.
Session mining (docs/plans/agent-loop-gaps-from-session-mining.md, 345 sessions, 52,307 tool calls, no LLM in the measurement path): "unverified completion claim" is the second-largest cluster of human corrections, in 27.8% of interactive sessions. The largest is premature stop, at 40%.
Benchmark (4 tasks, flagship + kimi-k2p7-code against Claude Code Sonnet 5, identical fixtures and mechanical scoring, quality scored blind by an Opus judge):
| GAIA | Claude Code | |
|---|---|---|
| Passed | 3/4 | 4/4 |
| Judged quality (1–5) | 3.38 | 4.75 |
| Runs containing an unsupported claim | 3 of 4 | 0 of 4 |
Two of the three fabricating runs passed mechanically — one claimed a test passed without running it, one claimed cleanup it had not performed. That is the part that matters: pass rate overstates real performance, and the overstatement is invisible to every metric currently collected.
The agent already computes the correct verdict and then discards it. On the failing run above, GAIA's own verification line rendered directly beneath the false claim: "unverified — 4 tool calls ran, none of them a test, lint, or build."
Design and implementation plan: docs/plans/agent-verification-states.md.
🔍 Technical details
_note_verification_signal (src/gaia/agents/base/agent.py:5075) records every dispatched call as {tool, check_label, failed, ran}, and build_verification_scope reduces it to verified / partially verified / unverified. _with_verification_scope (agent.py:5104) then does exactly one thing with it:
return f"{answer.rstrip()}\n\n{self.verification_scope_statement()}"The answer is never compared against the signal. status (agent.py:7396) is "success" if has_valid_answer and not has_errors — verification is not an input.
The state machine already exists (agent.py:853-857): PLANNING, EXECUTING_PLAN, DIRECT_EXECUTION, ERROR_RECOVERY, COMPLETION. Nothing sits between execution and completion; STATE_COMPLETION is entered directly at agent.py:5473, 6612, 7282.
The fix reuses an existing pattern. The answer seam already carries three guards that reject a candidate answer and re-prompt — post-index query (6905), tool markup (7005), SD capability (7209) — all the same shape, capped by _MAX_UNFINISHED_ANSWER_REPROMPTS = 2. A fourth guard joining the answer's claim class to the execution log needs no new machinery, and no filename matching: check_label over the turn's executions already answers "did a check actually run".
Related: #3972 covers the file-existence slice of this (and should resolve paths against the project root rather than os.getcwd() — commented there). This issue covers the two cases a file check cannot see: a claimed test run, and claimed cleanup.
Activity
kovtcharov commentedon Sep 18, 2026
Repeated measurement changes the picture of this issue, and the fix it proposes turns out to target the wrong failure.
On head of
main(d14ea1e9), 3 repeats × 4 tasks, clean and broken environments, quality judged blind: the misreporting that remains is not the agent claiming an action it never took. It is the agent reporting results it never read back. A check that asks "did a mutating tool run?" can't see it, because the tool did run.From the judge's notes:
On the data-analysis task this happened in 7 of 12 GAIA runs. Claude Code, on the same tasks and scoring, misreported in about 1 of 24 runs.
A guard built along the lines proposed above — matching the answer's claims against whether a check or mutating tool ran — fired in 0 of 24 runs. So the durable fix isn't a better claim matcher. It is to require the answer's reported contents to come from a tool result: after the agent writes an artifact, it has to observe it (read it back, run it) before it can describe it. That is the gap
docs/plans/agent-loop-gaps-from-session-mining.mdleads with — the model announcing a result without observing what it changed.🔍 Technical details
Per task on GAIA (main and main + candidate changes pooled, 12 runs each):
b6-weekly-reportmisreported 7/12,b6-config-env9/12 (mostly minor — scratch files left behind, then a claim of cleanup),02-bugfix3/12,21-qa0/12."Broken" means the project's test runner was missing from
PATH. It barely moved GAIA's pass rate, and misreporting appears in both environments. This is baseline behaviour, not something a degraded environment causes.amd-pworfolk commentedon Sep 21, 2026
Carrying a finding over from #4010, which shipped the file-write slice of this as #4012 and auto-closed. Posting here because it is a datapoint against one specific design assumption in the plan above, and #4010 is closed.
The assumption
From the technical details:
That holds for the claimed-test and claimed-cleanup cases this issue covers. It does not hold for the file-write case once a workflow legitimately writes something earlier in the same turn.
What I measured
#4012's guard asks "did any disk-touching tool run this turn?" via
_DISK_TOUCHING_TOOLS. My repro — the one that motivated #4010 — callstranscribe_mediafirst, which correctly writes a transcript to disk and is correctly in that set. So:The claim detector still fires correctly (
_claims_file_write(...) == Trueon all three of my recorded fabricated answers). It is the second half of the gate that is satisfied by an unrelated write, so the guard stays silent on the exact case it was built for.This is not a bug in the predicate —
transcribe_mediagenuinely puts bytes on disk. It is the "no filename matching" simplification meeting a turn where two different files are in play: one written, one only claimed.Why it generalises
Any turn shaped
produce an artifact -> derive something from it -> save the resulthas a legitimate early write that blanket-licenses every later save claim. Transcribe-then-summarise is one instance; download-then-extract, screenshot-then-annotate, and render-then-export have the same shape. The guard's recall goes to zero on precisely the multi-step workflows where a dropped final save is most costly, because the user has already paid for the expensive earlier steps.What it suggests
For the file-write class specifically, the check likely has to be "was the claimed path written?" rather than "did any write happen?" — i.e. compare the path in the claim against the paths the turn's write calls actually targeted. That is the filename matching the plan hoped to avoid, and I think the trade is worth naming explicitly rather than discovering later: without it, the file slice of this issue is defended only on single-write turns.
The claimed-test and claimed-cleanup cases this issue centres on are unaffected —
check_labelis the right instrument there, and no path comparison is needed.Evidence
Full measurements, including the 4 live runs on merged
main@e260ef2fand the detector results against my three real fabricated answers: #4010 (comment)Environment: Gemma-4-E4B-it-GGUF via Lemonade (GPU, 65536 ctx), Windows,
gaia-tui run gaia --dev --bypass-permissions.(Separately, validating the above surfaced #4084 —
run_pythonpulling the model off the tool-call path — which blocked 2 of those 4 runs from reaching the save step at all.)amd-pworfolk commentedon Sep 22, 2026
One more datapoint for this issue's core claim — a run can report success for work it never did — from the same workflow as my earlier comment.
Filed #4141: asked to enumerate every item in a long document, the agent read all of it and reported 15 of ~40, with no signal that anything was missing. The verification line rendered
unverified — 13 tool calls ran, none of them a test, lint, or build, which is true and beside the point: every tool call succeeded. Nothing was claimed that a tool did not do. The answer was simply 37% of the answer.Raising it here because it is a completeness failure rather than a truthfulness one, and the guards discussed in this thread — claim-vs-execution-log — would not catch it. If the state machine grows a step between execution and completion, "did the answer draw on everything that was read?" may belong in it alongside "did a check actually run".
kovtcharov commentedon Sep 23, 2026
@amd-pworfolk — opened #4145 for review. It checks the specific output file and requires readback after its latest write, so a saved transcript cannot stand in for a missing summary. Missing saves and unsupported test/cleanup claims now return incomplete, including at the step limit.
1,286 tests and the Fireworks/local-model checks passed. The standard evaluation is still blocked by the judge's weekly limit; keeping this issue open pending validation and merge. Extraction completeness remains separate in #4141.
kovtcharov commentedon Sep 24, 2026
@amd-pworfolk — #4145 now handles unbroken transcripts and completed empty pages. Your repro led to another fix: a field cut off at a page boundary can now be recovered from the next page without losing known values or accepting conflicting evidence.
The stronger 86K-character, zero-newline fixture passed on Fireworks and local Gemma-4-E4B: 40/40 exercises, every field correct, complete save/readback, and a later memory lookup. Local extraction/save took 16.7 minutes. The specific-output guard you verified remains intact.
The PR is ready for review; please rerun your original workshop on the latest head before we close #4141. #4087 is already included; no other PR is a prerequisite. Standard judged evaluation remains deferred.
fix(agents): verify completion and preserve exhaustive inventories (a…