Skip to content

After a cancelled worker generation, subsequent local worker requests are never dispatched until the process restarts #490

Open

Description

@sosidudku1

atomic-agent 0.6.3 on macOS arm64 (MacBook Pro M5, 32 GB)
Component: local provider client / tools/fusion/worker-runner.ts (cancellation path)
Type: bug


What I see

One TUI process, one evening, four fusion.delegate calls containing the same task (bodies, worker model gemma-4-26b-a4b). Timeline from the session trace (times UTC):

18:32  bodies  max_steps  steps=0  45.0 min   (queued behind a wide fan-out — see #489)
19:26  bodies  max_steps  steps=0  43.4 min
20:11  bodies  max_steps  steps=0  45.0 min   <- only task in the fan-out, slots free
20:54  bodies  max_steps  steps=0  41.8 min   <- only task in the fan-out, slots free

llama-server.log tells the burns apart:

  • The 19:26 burn did reach the server: task 132072 decoded for ~40 min (n_decoded = 10,353 at 4.27 t/s), completed no tool call, and was cancelled at the limit:
1272.58.171.465 I slot print_timing: id  0 | task 132072 | n_decoded =  10340, tg = 4.27 t/s
1273.01.330.779 W srv          stop: cancel task, id_task = 132072
1273.01.571.529 I slot      release: id  0 | task 132072 | stop processing: n_tokens = 16476, truncated = 0
  • After that cancel the log is empty until the next day — the 20:11 and 20:54 delegations, run with free slots on an idle server, never produced a single server log line. The agent still reported two full ~45-minute worker runs with 0 tool calls each.

Next morning, after quitting and relaunching the TUI (same session resumed via /sessions), the identical work split into two tasks (physhalf, drawhalf) completed in 21.2 and 9.4 minutes on the first try. Same daemon config, same model, same specs; the only change besides the brief split was the fresh process.

So whatever the cancellation of a long, tool-call-less generation leaves behind, it appears to disable dispatch to the local leg for the rest of the process lifetime: requests are never sent, so no server-side or first-token timeout can fire, and each subsequent worker silently burns its full budget.

What I expect

  • A worker whose completion request was never dispatched should fail fast and say so, not report max_steps after 45 idle minutes.
  • Cancellation of an in-flight local generation should leave the client able to issue the next request.

Full artifacts

Complete llama-server.log (the silent gap is between uptime marks 1273.01 and 2218.18), full session trace with every delegate's briefs and outcomes, and the exact brief of the failing solo task: https://gist.github.com/sosidudku1/a9dc76c2cb34faaebd1c7363e6087e34

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions