Skip to content

fix(desktop): stop abandoning healthy mesh prefills at four minutes - #6115

Draft
micspiral wants to merge 1 commit into
mainfrom
micspiral/mesh-agent-llm-timeout
Draft

fix(desktop): stop abandoning healthy mesh prefills at four minutes#6115
micspiral wants to merge 1 commit into
mainfrom
micspiral/mesh-agent-llm-timeout

Conversation

@micspiral

Copy link
Copy Markdown
Collaborator

Problem

buzz-agent's BUZZ_AGENT_LLM_TIMEOUT_SECS defaults to 240 s
(crates/buzz-agent/src/config.rs:935). MeshLLM's OpenAI frontend gives a
backend call 600 s (OpenAiFrontendConfig::DEFAULT_BACKEND_TIMEOUT, raised
from 300 s in Mesh-LLM/mesh-llm#1355).

The two layers disagree, and the tighter one is the client. On shared compute
Buzz abandons requests the mesh is still legitimately serving.

Requests to a mesh are "stream": false (crates/buzz-agent/src/llm.rs:769), so a
cold prefill of a large agent prompt is one silent request with no bytes until
it finishes. Measured on an M5 Max serving unsloth/Qwen3.8-27B-GGUF:Q4_K_M:

prompt = 88,318 tokens (cold, no cache)
HTTP 200 after 503.17s   -> 176 tok/s prefill

At the 240 s default, the client gives up at four minutes on a request that
would have succeeded at 8.4 minutes, then retries with an escalated budget
(240 → 480 → 960, llm.rs:escalated_timeout) — piling a second and third
prefill onto a box already working on the first. Worst case is ~28 minutes for
a turn that a single 503 s wait would have completed.

Change

Seat the mesh client budget just above the server's, in the one place mesh
transport env is derived (apply_relay_mesh_env):

insert_default_if_unset(env, "BUZZ_AGENT_LLM_TIMEOUT_SECS", "660");

660 s > the frontend's 600 s, so the mesh's own error surfaces rather than a
client abort racing it.

Like the neighbouring BUZZ_AGENT_MAX_OUTPUT_TOKENS and
BUZZ_AGENT_REQUIRE_REPLY seeds this is a default, not policy: an explicit
user value survives, via insert_default_if_unset plus the copy-forward list in
relay_mesh_process_env (without which the spawn path re-defaults it).

Remote providers are untouched — Anthropic/OpenAI/OpenRouter agents keep
buzz-agent's own 240 s default. The gate is provider == "relay-mesh".

Tests

Four added, mirroring the existing reply-guard test shape:

  • native_provider_outlasts_the_mesh_backend_timeout — asserts the seeded value
    is > 600, i.e. the invariant, not the literal.
  • native_provider_preserves_explicit_llm_timeout
  • process_env_preserves_explicit_llm_timeout — the spawn-ordering hazard.
  • non_mesh_provider_leaves_llm_timeout_unset
cargo test --manifest-path desktop/src-tauri/Cargo.toml --features mesh-llm relay_mesh
  30 passed; 0 failed
cargo test --manifest-path desktop/src-tauri/Cargo.toml --features mesh-llm
  2544 passed; 1 failed

The one failure,
managed_agents::global_config::tests::inherited_shared_compute_translates_to_supported_agent_transport
(expects auto, gets mesh), is pre-existing on clean f956e6fe0
reproduced with this branch stashed. Unrelated to this change; not fixed here.

cargo clippy --features mesh-llm --all-targets -- -D warnings clean;
cargo fmt applied.

Caveat

Not exercised end-to-end through a live Desktop-spawned mesh agent — this is a
seeded-env change verified at the unit level plus the measured server-side
number above. Draft for that reason; happy to run the live path if wanted.

Originating conversation: buzz://message?channel=3861fcdd-61b9-424c-892a-00f062c15c35

buzz-agent's `BUZZ_AGENT_LLM_TIMEOUT_SECS` defaults to 240s. MeshLLM's
OpenAI frontend gives a backend call 600s
(`OpenAiFrontendConfig::DEFAULT_BACKEND_TIMEOUT`), so on shared compute the
client aborts work the server is still legitimately doing.

Requests to a local mesh are `stream: false`, so a cold multi-ten-thousand
token prefill is one silent request. Measured on an M5 Max serving
Qwen3.8-27B Q4_K_M: an 88,318-token cold prompt returns HTTP 200 after
503s at 176 tok/s prefill. At the 240s default the client gives up at
four minutes, then retries with an escalated budget (240 -> 480 -> 960),
piling load onto a box that is already prefilling the first attempt.

Seat the mesh client budget at 660s, just above the frontend's 600s, so
the mesh's own error surfaces rather than a client abort racing it.
Remote providers are untouched and keep buzz-agent's 240s default. Like
the other mesh seeds this is a default, not policy: an explicit user
value survives via `insert_default_if_unset` plus the copy-forward list
in `relay_mesh_process_env`.

Co-authored-by: Michael Neale <michael.neale@gmail.com>
Signed-off-by: Michael Neale <michael.neale@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants