Repository navigation
Model claims a file was saved without ever calling write_file (independent of the confirmation gate) #4010
Description
Summary
A multi-step agent turn (transcribe → read → extract → save) reliably ends with the model composing its full final answer — including sentences like "I have saved the result to X" or "The routine has been saved to..." — without ever having called the save tool (write_file) at all. This happens even with --bypass-permissions set, so it isn't the confirmation gate on write_file (Agent.confirmation_required_tools()) blocking it — the tool call is simply never emitted.
Reproduction
Using the flagship gaia agent (Gemma-4-E4B-it-GGUF via Lemonade, local):
- Give the agent a task with a clear final "write the result to a file" step, reachable only after a few prior tool calls (in my case, a custom skill: transcribe a video → read the transcript → extract structured content →
write_filethe result → report both paths). - Across three separate reproductions (varying the skill's own wording each time, including one version that says explicitly, in bold: "Never say the file was saved before
write_filehas actually returned success in this conversation... you are not done: call it now"), the model:- Composes the full output content correctly.
- Ends its turn with a sentence asserting the file was saved (sometimes with a plausible-but-wrong path).
- Never emits a
write_filetool call — confirmed via--trace/--devoutput showing notool_call write_fileanywhere in the run, and by the target file genuinely not existing on disk afterward.
- This reproduces both with and without
--bypass-permissions(i.e., independent ofwrite_file's confirmation-required status — seeAgent.confirmation_required_tools()insrc/gaia/agents/base/agent.py).
Why this is worse than an ordinary missed tool call
GAIA already has gaia-voice (a bundled skill on the flagship agent) whose explicit purpose is "do not claim work you did not do... do not present empty output as a result." That instruction is present and active in every one of my repro runs, and did not stop the model from asserting a save that didn't happen. This is exactly the fabricated-completion failure mode gaia-voice exists to prevent, occurring at the framework/model-integration level rather than something a downstream skill's wording can reasonably be expected to fix — I tried three increasingly explicit versions of the same instruction with no change in outcome.
Impact
Any skill or task whose last meaningful step is "write/save the result" is at risk of silently completing with nothing written, while confidently telling the user it succeeded — a worse failure than an honest error, since the user has no signal to go check.
Possible directions (not prescriptive)
- A structural check: if the final assistant message asserts a file was written/saved (heuristically or via a required-tool marker) but the corresponding tool was never called this turn, refuse the turn or force a follow-up step rather than let it end.
- Consider whether
write_file(and similarly consequential terminal-step tools) should be excluded from whatever end-of-turn "I think I'm done" shortcut the loop is taking, forcing at least one more step when a plan explicitly named it as pending. - This may be related to the same class of issue as read_file truncation recovery (read_tool_output) stops after one page instead of reading to total_chars #4009 — both look like "the model treats having reasoned about a remaining step as equivalent to having executed it."
Environment
- GAIA main @
73cd240a - Model: Gemma-4-E4B-it-GGUF via Lemonade (local, GPU backend, 65536 ctx)
- Windows,
gaia-tui run gaia --query ... [--bypass-permissions] - Reproduced 3x across different wordings of the same downstream skill instruction
Activity
github-actions commentedon Sep 18, 2026
Confirmed, and it isn't something your skill's wording can fix — the loop never checks the final answer against what actually ran, so no instruction placed downstream can close it.
The agent does compute the right verdict: every run appends a line that should have read "unverified — N tool calls ran, none of them a test, lint, or build." That line is advisory text stapled to the answer, never a gate on ending the turn. The one place the loop genuinely blocks a false claim today is hardcoded to image generation, which is why your case sails through.
This is measured already in #3983 (unverified completion claims in 27.8% of mined interactive sessions; 3 of 4 benchmark runs contained one), and #3972 has the missing-output-file slice in flight. Your instinct about #4009 is right — same family.
@kovtcharov-amd — worth deciding whether this folds into #3983 or stays open as the
write_filecase; the repro here is cleaner than anything in that thread, and it's the "did a mutating tool run?" shape rather than the "reported contents it never read back" shape you measured.🔍 Technical details
_note_verification_signal/build_verification_scope(src/gaia/agents/base/agent.py:5167,src/gaia/agents/base/verification.py:176) already reduce the turn's executions to verified / partially verified / unverified._with_verification_scope(agent.py:5196) does exactly one thing with it — string-concatenates it onto the answer. The answer's claims are never compared to the signal, andstatusis"success" if has_valid_answer and not has_errors, with verification not an input.agent.py:7204), capped by_MAX_UNFINISHED_ANSWER_REPROMPTS = 2. The SD one is the exact shape you're asking for, with_CAPABILITY_CLAIM_PATTERNSand_SD_CAPABILITY_TOOLShardcoded togenerate_image— awrite_filesibling would need no new machinery.--bypass-permissionschanges nothing is consistent with this:Agent.confirmation_required_tools()sits on dispatch, and no call is ever dispatched.github-actions commentedon Sep 18, 2026
Opened #4012 with a fix.
What was fixed: the agent loop had no check tying "I saved the file" back to a tool call that would make it true, so the answer was accepted as final. It now rejects a final answer that claims a file was written when no disk-touching tool ran that turn, and pushes the model to either make the call or stop claiming it did.
How to verify:
git fetch origin autofix/issue-4010 && git checkout autofix/issue-4010pip install -e .write_file, with and without--bypass-permissions--trace/--dev, the turn should now either show a realwrite_filecall or end without asserting a saveThe PR is a draft: lint and the unit suite pass (no Python regression), but neither can exercise the model behaviour this targets, so it needs validation against a live Lemonade server before merge. Your repro is the check that matters here — if it still fabricates the save, the detection pattern is the thing to widen.
🔍 Technical details
The guard sits in the
process_queryanswer seam insrc/gaia/agents/base/agent.py, alongside the existing fabricated-completion guards (tool-markup-as-text, planning-only prose, raw-JSON tool output, unbacked capability claims), and uses the same mechanism: inject a correction and loop rather than setfinal_answer.Deliberately conservative, since a false positive costs a wasted round trip:
It does not touch any system prompt, so it is orthogonal to the
gaia-voiceinstruction you mentioned — this is the structural check that instruction could not enforce on its own. Your note about #4009 being the same class of failure looks right; this guard only covers the file-write case, so if that one needs the same treatment it is a second, narrower pattern at the same seam.fix(agents): judge a save claim by what a tool did, not what it tried
amd-pworfolk commentedon Sep 21, 2026
Re-tested on merged
main(e260ef2f) with the original #4010 repro. The guard's own fix looks good — mycreate_tablefalse-negative note was addressed (_DISK_TOUCHING_TOOLSis now an explicit allowlist read off the execution log withran/not failed, which is strictly better than the substring match I reviewed). But I could not get the workflow to reach the save step at all, for an unrelated reason that landed in the same pull.Blocker: model routes
transcribe_mediathroughrun_pythonand never recoversrun_python(#3994) merged alongside this. On 2 of 4 runs the model tried to invoke a GAIA tool inside it:…then looped — 5x
load_skill, 2xrun_python— burning all 8 steps and producing no summary and no file. Run 1 stalled similarly (5 steps, 1438 chars of plan narration, no output).Adding
Call transcribe_media as a tool directly. Do NOT use run_python for any step.to the query fixed it instantly (run 4:transcribe_mediacalled properly on step 1). So it is specificallyrun_python's presence pulling the model off the tool-call path.The docstring ("Run a Python snippet... to compute, transform data, or run a quick check") doesn't warn that GAIA tools aren't importable, and
from gaia import <tool>is the obvious wrong guess. A line to that effect would likely be enough. Happy to file this separately if you'd prefer it off this thread — flagging here since it arrived in the same merge and blocks validating #4010's fix end to end.Guard: not exercised on my repro, and structurally can't be
Separate from the blocker:
transcribe_mediaandrefine_transcriptare both in_DISK_TOUCHING_TOOLS, and my skill callstranscribe_mediaon every run. So_wrote_this_turnis alreadyTruebefore the save step is reached:The claim text still detects correctly (
_claims_file_write(...) == Trueon all three of my original fabricated answers), but the second half of the gate is satisfied by the transcript write, so the guard stays silent on exactly the #4010 case that motivated it.That is defensible as written —
transcribe_mediagenuinely does put bytes on disk, so it isn't a bug in the predicate. But it does mean #4010's own repro is outside the fix's reach, because any workflow whose earlier steps legitimately write a file gets a free pass on a later fabricated save. If the intent is "the claimed file was written," the check may need to compare the claimed path against what was actually written, rather than asking whether any disk write happened.Also seen (pre-existing, not from this PR)
Run 4, once unblocked, ignored the skill's procedure and went
index_document→query_specific_fileinstead ofread_file, returning two sentences of prose with no exercise list and no save — the RAG-instead-of-full-read path my skill explicitly forbids. Same family as #4009.Status
_DISK_TOUCHING_TOOLSallowlist — confirmedcreate_table/insert_datacorrectly excludedrun_pythonloop abovetranscribe_media, see aboveEnvironment: merged
main@e260ef2f, Gemma-4-E4B-it-GGUF via Lemonade (GPU, 65536 ctx), Windows,gaia-tui run gaia --dev --bypass-permissions, 4 runs.fix(agents): block final answers that claim a save no tool performed (a…
amd-pworfolk commentedon Sep 21, 2026
Follow-ups from the post-merge validation above, so this closed thread does not dead-end:
run_pythoninvitesfrom gaia import <tool>and the turn loops until the step budget is gone. This is what blocked 2 of my 4 runs from reaching the save step; unrelated to fix(agents): block final answers that claim a save no tool performed #4012 itself, arrived in the same pull via feat(tools): run_python runs a snippet without leaving a file in the repo #3994.transcribe_media-suppresses-the-guard finding, raised there as a datapoint against the "no filename matching" assumption in that plan, since it is the umbrella for this class.No objection to this staying closed — #4012 did what it set out to do, and my
create_tablenote was addressed in the merged version.