fix(desktop): stop abandoning healthy mesh prefills at four minutes - #6115
Draft
micspiral wants to merge 1 commit into
Draft
fix(desktop): stop abandoning healthy mesh prefills at four minutes#6115micspiral wants to merge 1 commit into
micspiral wants to merge 1 commit into
Conversation
buzz-agent's `BUZZ_AGENT_LLM_TIMEOUT_SECS` defaults to 240s. MeshLLM's OpenAI frontend gives a backend call 600s (`OpenAiFrontendConfig::DEFAULT_BACKEND_TIMEOUT`), so on shared compute the client aborts work the server is still legitimately doing. Requests to a local mesh are `stream: false`, so a cold multi-ten-thousand token prefill is one silent request. Measured on an M5 Max serving Qwen3.8-27B Q4_K_M: an 88,318-token cold prompt returns HTTP 200 after 503s at 176 tok/s prefill. At the 240s default the client gives up at four minutes, then retries with an escalated budget (240 -> 480 -> 960), piling load onto a box that is already prefilling the first attempt. Seat the mesh client budget at 660s, just above the frontend's 600s, so the mesh's own error surfaces rather than a client abort racing it. Remote providers are untouched and keep buzz-agent's 240s default. Like the other mesh seeds this is a default, not policy: an explicit user value survives via `insert_default_if_unset` plus the copy-forward list in `relay_mesh_process_env`. Co-authored-by: Michael Neale <michael.neale@gmail.com> Signed-off-by: Michael Neale <michael.neale@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
buzz-agent'sBUZZ_AGENT_LLM_TIMEOUT_SECSdefaults to 240 s(
crates/buzz-agent/src/config.rs:935). MeshLLM's OpenAI frontend gives abackend call 600 s (
OpenAiFrontendConfig::DEFAULT_BACKEND_TIMEOUT, raisedfrom 300 s in Mesh-LLM/mesh-llm#1355).
The two layers disagree, and the tighter one is the client. On shared compute
Buzz abandons requests the mesh is still legitimately serving.
Requests to a mesh are
"stream": false(crates/buzz-agent/src/llm.rs:769), so acold prefill of a large agent prompt is one silent request with no bytes until
it finishes. Measured on an M5 Max serving
unsloth/Qwen3.8-27B-GGUF:Q4_K_M:At the 240 s default, the client gives up at four minutes on a request that
would have succeeded at 8.4 minutes, then retries with an escalated budget
(240 → 480 → 960,
llm.rs:escalated_timeout) — piling a second and thirdprefill onto a box already working on the first. Worst case is ~28 minutes for
a turn that a single 503 s wait would have completed.
Change
Seat the mesh client budget just above the server's, in the one place mesh
transport env is derived (
apply_relay_mesh_env):660 s > the frontend's 600 s, so the mesh's own error surfaces rather than a
client abort racing it.
Like the neighbouring
BUZZ_AGENT_MAX_OUTPUT_TOKENSandBUZZ_AGENT_REQUIRE_REPLYseeds this is a default, not policy: an explicituser value survives, via
insert_default_if_unsetplus the copy-forward list inrelay_mesh_process_env(without which the spawn path re-defaults it).Remote providers are untouched — Anthropic/OpenAI/OpenRouter agents keep
buzz-agent's own 240 s default. The gate is
provider == "relay-mesh".Tests
Four added, mirroring the existing reply-guard test shape:
native_provider_outlasts_the_mesh_backend_timeout— asserts the seeded valueis
> 600, i.e. the invariant, not the literal.native_provider_preserves_explicit_llm_timeoutprocess_env_preserves_explicit_llm_timeout— the spawn-ordering hazard.non_mesh_provider_leaves_llm_timeout_unsetThe one failure,
managed_agents::global_config::tests::inherited_shared_compute_translates_to_supported_agent_transport(expects
auto, getsmesh), is pre-existing on cleanf956e6fe0—reproduced with this branch stashed. Unrelated to this change; not fixed here.
cargo clippy --features mesh-llm --all-targets -- -D warningsclean;cargo fmtapplied.Caveat
Not exercised end-to-end through a live Desktop-spawned mesh agent — this is a
seeded-env change verified at the unit level plus the measured server-side
number above. Draft for that reason; happy to run the live path if wanted.
Originating conversation:
buzz://message?channel=3861fcdd-61b9-424c-892a-00f062c15c35