Orchestration and evaluation failure semantics (finding 4)

[arst]

Fourth and final PR remediating the second review (docs/reviews/2026-08-25-second-review.md). It covers orchestration failure semantics and evaluation fail-closed behaviour: what each of these samples does when work fails, when a judge cannot be parsed, and when a bound is reached.

What changed, per sample

Orchestrator-Workers — a failed worker no longer disappears. BuildSynthesisInput filtered failures out, so the synthesiser saw a gap it could not know about and confidently wrote over it. Failures now arrive with STATUS: FAILED, the error, and an explicit instruction not to invent an output. A new WorkerRegistry.Assess classifies the run Complete / Partial / Abstained; the sample skips the synthesiser entirely when every worker failed, and labels a partial run.

Exception handling — cancellation is not a transient fault. Retry.RunAsync's generic catch converted a cancelled turn into (null, lastError), so a caller who asked to stop got a fallback instead. It rethrows now. The Semantic Kernel twin had the identical defect — a cancelled call was swallowed, retried, and finally answered from a different kernel function — and is fixed too. The whole-turn-retry idempotency caveat now reaches the pattern doc and cross-links IdempotentToolCalls.

Tool authorization — a one-time capability now has a commit point. The nonce was burned during Authorize, before the tool ran, so any pre-effect failure permanently destroyed a valid capability. Authorize now reserves; Commit consumes after the effect is durable; Release returns a reservation after a verified pre-effect failure. Money amounts are validated whether or not a maximum is configured — previously a refund with no ceiling accepted a negative amount. And an approval request is a typed PendingApproval the host handles, not a string the model receives as tool output and paraphrases.

ReAct — the tool-call bound is enforced by the host. "Use at most 10 tool calls" was a sentence in a prompt. It is now an IAutoFunctionInvocationFilter that stops the loop at the boundary. See the note below: the first implementation of this did not work, and the reason is worth reading.

LLM-as-Judge — an unparseable verdict is Indeterminate, never a win. winner == "B" ? "B" : "A" meant every malformed reply counted as a victory for candidate A. JudgeParsing.Parse now returns Indeterminate for anything that is not exactly "A" or "B", indeterminates are excluded from the rate and counted, and a single position swap became five balanced randomized orderings.

Regression evals — a trace is a candidate, not a golden case. Extraction produced a GoldenCase directly from a recorded run, freezing the model's own historical answer into the suite as the thing to match. It now produces a CandidateCase awaiting review; only cases a human signed off with ReviewedBy are evaluated. The "exact" tier is renamed "contains", which is what it always did. A run that evaluated nothing now fails the gate instead of passing green.

Resource-aware optimization — the budget is soft, and the docs say so. The last-resort call after every tier failed was never recorded, so the printed spend under-reported the real spend. It is accounted for now. budget became softBudget in both flavors, and the copy stopped promising to "stop spending when the budget runs out" — routing decisions are made from spend observed after each call, which is what makes it soft. The docs point at BoundedExecution for a pre-call ceiling.

CI — actions pinned by commit SHA. Every action moved to its current major and is pinned to the commit its tag points at, with the version in a trailing comment. A mutable tag is a supply-chain hole in a workflow holding packages: write.

The finding worth reading

Task 4.4 was first implemented as a Semantic Kernel IFunctionInvocationFilter that threw when the budget was exceeded. It passed its unit tests. It did not work.

SK 1.79 wraps every auto-invoked function call in a catch-all that converts any exception into a tool-result error and continues the loop. Measured against real SK with a stubbed endpoint: the tool body was correctly stopped after 10 calls, but the loop made 129 model calls instead of 11, and the budget refusal was handed back to the model as tool output to paraphrase — the same failure mode this PR removes from ToolAuthorization two commits earlier. Meanwhile the pattern doc had been rewritten to claim the loop stopped.

The unit tests could not see any of it, because they drove a seam the runtime never calls.

The fix is IAutoFunctionInvocationFilter + context.Terminate = true, the shape GoalSettingsAndMonitoring.SemanticKernel already used. It is now pinned by ToolCallBudgetFilterRealLoopTests, which drives the real auto-invocation loop through a stub HttpMessageHandler — no network, no port, sub-second — and asserts 10 tool bodies, 11 model calls, and that the refusal text appears in no request sent to the model. That test goes red at 129 model calls under the original throwing shape.

Verification

• dotnet build "Agentic Patterns.slnx" -c Release — 85 projects, 0 errors, 0 warnings
• dotnet test "Agentic Patterns.slnx" -c Release --no-build — 300/300 (238 before this PR), 16.5s
• 21 mutations run by the whole-branch review and restored, zero survivors — each guarantee was broken in the source and the suite confirmed red. Further mutations by the fix-round re-review, also all red.
• The concurrency guarantee in ToolAuthorizationPolicy was measured rather than read: its first test caught its own reverted implementation 3 times in 500 runs, and 0 in 300 under load. The committed version catches it 25/25.
• Uniformity sweep across all 85 projects for the four defect classes this PR fixes. One further instance found and fixed (RubricJudgeEvaluator, below); the rest reported none, with the sites checked named.
• No test deleted. [Fact]/[Theory] 204 → 239. Exactly one assertion changed across the branch — Assert.DoesNotContain("simulated failure", …) inverted to Assert.Contains, because the behaviour it pinned is the behaviour this PR deliberately reverses.

Known gaps

• RubricJudgeEvaluator was a fifth instance of the coerced-verdict defect, in the same project Task 4.5 hardened — it threw on malformed JSON and scored {} as 0, below the rubric's own floor of 1. Fixed in this PR, but it was found by the whole-branch review, not by the task that touched the file next door.
• ToolAuthorization's MoneyMovingTools is a hand-maintained set inside the policy. Register a new money-moving tool and forget the line, and it gets no amount floor. Marked with its ceiling and upgrade path; moving the flag onto tool registration touches every sample that registers a tool and deserves its own change.
• HumanInTheLoop.SemanticKernel still returns APPROVAL_DENIED: … as function-result text — the shape ToolAuthorization.md now argues against. Real, out of this PR's scope, deliberately left for its own task.
• Commit/Release return void and no-op on an invalid transition, as specified. Every invalid transition was verified to fail safe (no path creates authority or resurrects a consumed capability), but a host that commits the wrong nonce gets no signal.
• .SemanticKernel twins sit outside the test assembly by design, so twin-only fixes — the retry cancellation rethrow, the narrowed fallback catch — rest on review rather than on a test.
• RegressionEvals writes candidates under bin/, so a candidate does not survive a clean to actually be reviewed. Deliberate for a sample; a real repo commits them beside the corpus.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LB4jjPp7i2pxV55Vpe6tpc