fix: enforce the integrity and isolation claims the samples were making (finding 3) - #14
Merged
Conversation
…nsition Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB4jjPp7i2pxV55Vpe6tpc
…entries Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB4jjPp7i2pxV55Vpe6tpc
…y in the test Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB4jjPp7i2pxV55Vpe6tpc
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB4jjPp7i2pxV55Vpe6tpc
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB4jjPp7i2pxV55Vpe6tpc
GuardRails' PII redaction and output truncation rebuilt a text-only response, dropping function calls, finish reason, usage, model id and everything else. A guardrail that destroys the response it is protecting is worse than no guardrail. Lift the redaction/truncation logic into a testable global-namespace GuardRails class built on a message-level core that rewrites only TextContent items, leaving every other content kind untouched, and copies response-level metadata onto the rewritten response. Wire it into both PiiGuardMiddleware and OutputGuardMiddleware's AgentResponse path (the one actually doing the flattening), and give the Semantic Kernel twin's OutputGuardFilter the equivalent fix via FunctionResult(FunctionResult, object?), which preserves the original result's metadata and culture instead of discarding them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB4jjPp7i2pxV55Vpe6tpc
RedactionPreservesFunctionCallsAndUsage only checked that a REDACTED tag was present in the output, not that the phone number itself was removed - a regression that stopped scrubbing while some other pattern still fired would have passed undetected. Add DoesNotContain for the raw phone number alongside the existing Contains check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB4jjPp7i2pxV55Vpe6tpc
…cache key SkillLifecycle.Approve/Activate/Retire now re-verify the on-disk digest before transitioning, so a file edited after MarkTested is refused at approval rather than deferred to load time. GuardRails.WithMessages now preserves ChatResponse.ConversationId so a stateful IChatClient doesn't silently lose its server-side thread on any turn where PII is detected. The semantic cache's partition key now folds every ChatOptions property that changes what a valid answer looks like (tools, tool mode, sampling/penalty knobs, stop sequences, reasoning, additional properties), not just ModelId/Temperature/ResponseFormat, matching the shape TraceReplay.CanonicalOptions already uses one sample over. Also: a six-row theory replaces two per-field cache-namespace tests so all six CacheNamespace dimensions are regression-protected; CodeActExecutionTests and StigmergicBuildGateTests share an xunit collection so they can't race on the process-global unsafe-execution env var; two stale comments (SkillLifecycle's "immutable" claim, GuardRails.md's Truncate table row) corrected; the cache's ponytail comment now notes expiry is reclaimed only on read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB4jjPp7i2pxV55Vpe6tpc
The final re-review found ChatResponse.ContinuationToken silently dropped by WithMessages, on the assumption it was [Experimental] like its AgentResponse namesake. It is not: it is the handle an IChatClient polls a background response with, so dropping it strands the caller exactly as dropping ConversationId did. Copy it, assert it, and bring the class summary's list of copied metadata back in line with what the code actually copies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB4jjPp7i2pxV55Vpe6tpc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third PR of the four-PR remediation of
docs/reviews/2026-08-25-second-review.md. It answers four findings that share one shape: a sample claimed a property its code did not enforce — and, in the fourth case, destroyed the data it claimed to protect.SkillLearning— approval bound nothing to the approved contentThe lifecycle was candidate → validated → tested → approved → active, with a reviewer's name recorded at approval. Nothing tied any of that to what
SKILL.mdactually said. Anyone who could write the file could edit an already-approved skill in place, and the agent would load it.SkillManifestnow carriesContentSha256, computed once when the candidate is written.ReadVerifiedrefuses content whose digest moved, and every content read goes through it —Validate,MarkTested,ReadActive, andApprove,Activate,Retire. Verification happens before the manifest is persisted, so a file edited betweenMarkTestedandApproveis refused at approval rather than at load time, andmanifest.jsonnever records a sign-off against content the reviewer never saw.The doc says what this buys and what it does not: the digest detects tampering, it does not prevent it — whoever can write
SKILL.mdcan writemanifest.jsontoo. Sign the manifest, or keep it outside the agent's write scope.SemanticCaching— a cache key that could not isolate anythingThe partition key covered the conversation shape and three
ChatOptionsfields. No tenant, no authorization scope, no tool policy, no data revision. Entries never expired and never evicted.CacheNamespace(TenantId, PrincipalScopeHash, SystemPromptHash, ToolSchemaHash, ModelVersion, DataRevision)is a required constructor argument — the isolation dimensions cannot be forgotten, only chosen.TenantIdcontaining the delimiter cannot collide with a different tenant's key.AIContentkinds — a history differing only in a tool call no longer shares a partition.ChatOptions' 20 properties to 16, includingTools(name + schema),ToolMode,MaxOutputTokens,SeedandStopSequences. A runtime tool set that diverges from the declaredToolSchemaHashcan no longer collide.EvaluationAndMonitoring— traces defaulted to full plaintext captureThe safe modes existed; you had to know to ask for them.
RunTracenow defaults toRedactedContent. Full capture is its own mode word,record-full, gated onAGENTIC_PATTERNS_ACKNOWLEDGE_FULL_TRACE_CAPTURE=I_UNDERSTAND_THIS_WRITES_PROMPTS_AND_OUTPUTS_IN_PLAINTEXT— exact ordinal match, so a wrong case or a trailing space is rejected — with a banner in the same shape as CodeAct's unsafe-execution warning, printed before anything is recorded.The doc is explicit that redaction is best-effort pattern matching, not a guarantee: it recognises a limited set of shapes and misses whatever it was not taught. Traces are plaintext JSON on disk; treat the directory as production data.
GuardRails— the guard destroyed the response it was guardingPII redaction and output truncation concatenated every
TextContent, redacted the string, and returned a brand-new single-message response. Function calls, function results, finish reason, usage, model id: gone, on every turn where a regex happened to match.Redaction now maps each message's
Contents, rewriting onlyTextContent; truncation trims the last text item rather than collapsing the message. Both theIChatClientpath and the agent path route through the same core, so neither flattens.ConversationIdandContinuationTokenare carried through — a stateful client would otherwise lose its server-side thread, and a background response its polling handle, intermittently and only when PII was detected.RawRepresentationis deliberately not copied: it would hand the caller a route back to the un-redacted payload.The SemanticKernel twin gets the same semantic fix in its own shape —
new FunctionResult(context.Result, text)preserves the original's metadata and culture instead of discarding them.Verification
dotnet build "Agentic Patterns.slnx" -c Release→ 85 projects, 0 errors, 0 warningsdotnet test "Agentic Patterns.slnx" -c Release --no-build→ 238/238.Text. There is no fifth sample.[Fact]/[Theory]184 → 203).Known gaps, stated rather than papered over
internaland expose nothing to assert on. Its fix rests on the documentedFunctionResult(FunctionResult, object?)contract.FinishReason = null, and expired entries in a never-queried partition are reclaimed only on the next read — both bounded bymaxEntriesPerPartition, both noted in// ponytail:comments with their upgrade paths.🤖 Generated with Claude Code
https://claude.ai/code/session_01LB4jjPp7i2pxV55Vpe6tpc