Skip to content

docs: explain reviewer tool evidence in task artifacts - #584

Merged
zzwong merged 1 commit into
mainfrom
zzwong/issue-551/document-reviewer-tool-evidence
Sep 7, 2026
Merged

docs: explain reviewer tool evidence in task artifacts#584
zzwong merged 1 commit into
mainfrom
zzwong/issue-551/document-reviewer-tool-evidence

Conversation

@zzwong

@zzwong zzwong commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #551.

A reviewer task can finish successfully while its review still cannot approve: task success means the returned structured output is valid, while review coverage also depends on the required diff-tool evidence. The artifact guide explained task statuses and telemetry but omitted reviewer_tool_evidence, leaving that combination unexplained for anyone inspecting a saved or resumed run.

This adds the missing field to the guide's behaviorally significant metadata and documents its existing contract in one place.

What the guide now explains

  • The four diff_status values: not_invoked, incomplete, succeeded, and failed, plus the optional diff_diagnostic string.
  • Why a recorded result with unsuccessful tool evidence receives incomplete_tool coverage and turns an otherwise approving review into a comment.
  • How metadata preserves the evidence when a cached task is restored, so reusing successful output retains the tool state used to assess coverage.
  • How schema version 1 handles an absent field: no adapter-provided evidence is available, normal coverage checks still apply, and the existing schema, fingerprint, and payload requirements still govern reuse.

The example makes the distinction concrete:

{
  "status": "succeeded",
  "reviewer_tool_evidence": {
    "diff_status": "failed",
    "diff_diagnostic": "fixed diff unavailable"
  }
}

This task has valid output, but its failed tool evidence prevents complete review coverage. The guide also distinguishes an absent field from explicit not_invoked evidence and explains that successful tool evidence still requires the normal assigned-file coverage checks.

Scope and verification

The change is confined to docs/llm-task-artifacts.md and describes the implemented behavior. The companion reporting correction is in #585.

  • Cross-checked field names, status values, persistence, cached restoration, coverage classification, and approval handling against the adapter, lifecycle, and pipeline source.
  • Seven existing lifecycle and pipeline regression tests passed; the JSON example, internal heading link, and whitespace checks passed.
  • CI passed, including tests, the 1Password opt-out configuration, static smoke tests, lint, and Linux/macOS/Windows builds.

@monit-reviewer monit-reviewer left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated PR Review

Reviewed commit: 28640c468d13
Profile: open-cli-collective-review - Posting as: monit-reviewer

Summary

Reviewer Findings
documentation:docs 0
policies:conventions 0

Reviewer Coverage

  • documentation:docs — complete (broad); skipped: none; constraints: Review scope was limited to the assigned documentation change and its directly referenced lifecycle, adapter, pipeline, and coverage sources.
  • policies:conventions — complete (broad); skipped: none; constraints: Reviewed only the assigned documentation change; shared CLI standards were not present in the provided repository context.
Inspected files (1)
  • docs/llm-task-artifacts.md

0 PR discussion threads considered. 0 summarized; 0 resolved.


Completed in 1m 35s | $0.03 | openai-codex/gpt-5.6-terra | cr 0.10.299
Field Value
Model openai-codex/gpt-5.6-terra
Reviewers documentation:docs, policies:conventions
Engine pi_rpc · openai-codex/gpt-5.6-terra
Reviewed by cr · monit-reviewer
Duration 1m 35s wall · 2m 07s compute
Cost $0.03
Tokens 6.0k in / 632 out

Per-workstream usage

  • orchestrator-selection — openai-codex/gpt-5.6-terra
    • In: 818
    • Out: 143
    • Cache read: 4.9k
    • Cache create: 0
    • Cost: $0.00
    • Duration: 17s
  • documentation:docs — openai-codex/gpt-5.6-terra
    • In: 1.9k
    • Out: 299
    • Cache read: 17.9k
    • Cache create: 0
    • Cost: $0.01
    • Duration: 1m 05s
  • policies:conventions — openai-codex/gpt-5.6-terra
    • In: 2.2k
    • Out: 153
    • Cache read: 3.6k
    • Cache create: 0
    • Cost: $0.01
    • Duration: 41s
  • orchestrator-rollup — openai-codex/gpt-5.6-terra
    • In: 1.1k
    • Out: 37
    • Cache read: 2.8k
    • Cache create: 0
    • Cost: $0.00
    • Duration: 2s

@zzwong
zzwong marked this pull request as ready for review September 7, 2026 23:40
@zzwong
zzwong merged commit f95b44b into main Sep 7, 2026
10 checks passed
@zzwong
zzwong deleted the zzwong/issue-551/document-reviewer-tool-evidence branch September 7, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: Explain how reviewer tool evidence affects resumed reviews

2 participants