diff --git a/CHANGELOG.md b/CHANGELOG.md index c989180cb..de35cb229 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,7 @@ - `diff --application` reads bounded local factories returning OpenAI Agents SDK and Google ADK tool lists. It keeps the returned members and their construction locations, and reports unread constructor identity, import effects, escaping values, or unresolved callers as coverage gaps rather than treating them as established wiring. This is static source analysis; it does not execute a factory or establish runtime enforcement. (#874, #964) - Constructor ownership still withholds canonical SDK Runner execution consumers and anonymous callable or generator carriers. These are static coverage limits; readable candidate tools and handoffs remain visible, and incomplete scans do not report them as removed. The direct `from . import agent` ADK package layout is supported. When a binding graph is incomplete, a capability comparison now withholds only what depends on absence (removed and narrowed capabilities, resolved findings) and still reports added, changed and unknown-direction changes, saying so in its notes; a remote Google ADK `McpToolset` whose tools cannot be listed is not itself incomplete binding evidence, so a changed endpoint, `os.environ[...]` credential reference or tool filter on it, nested stdio parameters included, is reported again. An ADK sub-agent behind a constructor gap stays a candidate with an incomplete edge instead of being reported as unmatched. A module that re-exports `eval` or `exec` from `builtins` is no longer treated as an unused import while another module imports it. The proof that no repository file provides a standard-library module reads the directory listing, so it is the same on case-sensitive and case-insensitive filesystems. (#964) - **An MCP row names the `env_vars` entries that changed.** Replaying openai/codex-security#1281, a `.mcp.json` that added `CODEX_SECURITY_PLUGIN_ROOT` to a server's `env_vars` array and changed nothing else was a `changed` row reading `no difference in the command name launch_codex_security_mcp, launch arguments, env key names or header key names; the change is in a detail this output does not show, such as the command's path or another setting`, which pointed at the command's path for a change to a variable name. An MCP server grant now publishes `env_var_names` (host-grants `0.9`, extended in place): the plain names its `env_vars` list declares, in declared order, never a `NAME=value` entry, an object, a token-shaped string or an entry the digest's redaction rewrites. The row reads `codex-security: env_vars names +CODEX_SECURITY_PLUGIN_ROOT` on `diff` (text and `--json`), `verify`, the PR comment and `check`, beside `env keys` and `header keys` (`env keys +API_BASE -DEBUG; env_vars names +X -Y`); the same names in another order read `env_vars names in a different order`, and an added or removed server's cell lists them. The label is the key the list is declared under, so no claim is made that a variable is set or read. No value is published or printed, and rotating an `env` value is still no row, because `config_sha256`'s input redacts every `env` value. The field is display-only, left out of grant equality, the inventory digests and saved baselines, so no row, direction, `expands`, severity, expansion signal or `check` decision moves; a change confined to an unpublished entry still says it is not shown. See [STABILITY.md](STABILITY.md#mcp-env-var-names-795). (#795) +- **The five MCP tools describe themselves in `tools/list`.** Each `shipgate.*` tool was listed with an empty description; each now states its task, its inputs and that it runs no git, Git-backed host comparison, scan or verify, with names, input schemas and annotations unchanged. (#800) - **An OpenShell policy row names the declarations that changed.** Replaying openshift-kni/ai-sandbox#29, a selected policy that removed `/sandbox/.uv/python` from `filesystem_policy.read_only`, changed binary selectors and added GitHub release destinations printed `authored, OpenShell 0.1.2, 6 endpoint(s), facts → …7 endpoint(s), facts `, which names none of them. `diff` (text and `--json`), `verify`, `check` and the PR comment now print the difference: `filesystem read_only -/sandbox/.uv/python`, `destination aiplatform.googleapis.com:443 (l4) binaries +/tmp/**/node_modules/…/claude -/usr/local/bin/claude`, `removed destination oauth2.googleapis.com:443 (l4) via /usr/local/bin/claude` and `added destinations objects.githubusercontent.com:443, release-assets.githubusercontent.com:443 (l4) via …`, with changed request declarations, `include_workdir`, Landlock compatibility and process identity named the same way. Each list names its first entries and counts the rest. Only values the OpenShell inventory already publishes are named, and a value that would change under redaction prints as ``. Query and parameter constraints and endpoint options appear by field name, not value. A policy selected or removed whole lists what it declares and which sections and inspected endpoints took an upstream default, as a description and not a proof that authority grew. Display only: the published `rows[]`, direction, `expands`, severity, expansion signals and `check` decisions are unchanged, the existing unknown-direction and runtime-limit wording stays in each row's reason, and no schema changes. (#968) - **`diff` classifies a large base tree about eight times faster.** Translating each boundary glob into a regular expression was repeated for every path in the tree and was 74–86% of first-run `diff` time on two large repositories (#698's measurement). The compiled pattern is now cached per pattern; which paths match is unchanged, and a test checks the cached matcher against the previous implementation on over 10,000 random pattern and path pairs. (#698) - **`diff --application` leads with one finding per changed agent.** The text opens with a `Findings:` line and, for each agent that has a row, a finding whose first line stands alone: the tools the change added (`+`), removed (`-`) or changed (`~`), or only candidates (`?`); what they newly reach (the outbound call or library effect, its host, the credentials it sends by name, the parameters the model controls) or, for a changed tool, what it still reaches; and where reading stopped (`not read beyond memory_bank.py:531`), with the first unread hop and a count of the rest per tool. A finding with a binding or a reach it did not read is `partial`, though `comparison_status` still speaks of bindings only. A question specific to the change follows ("should `agent` reach `POST …` through `act`, sending `env WEATHER_TOKEN`, with the model controlling `city`?"; the same call newly taking a model-controlled argument; a removed tool; a changed one and what changed). Rows that are `not_established` for the same reason are one line with a count and one example, however many tools and agents share it, and limits no row states are one line per agent; effect evidence is counted once instead of `write (provisional: unknown effect)` per row. Per-row text follows under `Detail:` for up to 10 rows; above that the text says how many it did not print and `--json` carries every row, gap and reach limit. The JSON gains a top-level `summary` with the same content (`counts`, `findings`, `effects`, `causes`, `not_read`), each entry naming the rows it summarizes by index; it states nothing a row does not, and no row, status, direction, gap, scope or exit code changes. `application_comparison_schema_version` stays `0.4`, unreleased, with `summary` added. `comparison_id` digests it with the rows; `benchmark/application-q2/summarize.py` leaves it out of `answer_id`, so a hand score keeps judging the answer it read. On the 49-PR development corpus (`benchmark/application-q2/`) against `main` at `e703b41b`, `summarize.py` counts the same statuses and rows and every answer has the same `answer_id`. VidulaWickramasinghe/O.R.I.O.N#126's text goes from 11,926 lines to 24, jpka/attest#3's findings fit in 7 lines before its rows and Tiendat2703/MIS_TALENT#7's in 10; the committed goldens in `benchmark/application-q2/goldens/` hold six real answers. (#914) diff --git a/docs/distribution-surfaces.md b/docs/distribution-surfaces.md index 4cc730047..ae3adf24b 100644 --- a/docs/distribution-surfaces.md +++ b/docs/distribution-surfaces.md @@ -86,7 +86,7 @@ and this document are checked against each other by | `examples` | `examples/` | `executable_pin`, `merge_verdict_vocabulary` | `test_executable_pin_resolves_in_a_published_channel`, `test_surface_enumerations_match_the_engine_vocabulary`, `tests/test_adopter_pins_resolve.py::test_every_pin_init_writes_into_an_adopter_repo_names_the_published_release` | Illustrative CI wiring. It gates on the engine's merge verdict rather than restating the release-decision set, so only the verdict claim is registered. | | `policies` | `policies/` | — | — | Manifest fragments only. Answers no question the CLI answers: they are *inputs* the engine evaluates, not restatements of its output. | | `harness` | `harness/` | `merge_verdict_vocabulary`, `release_decision_vocabulary` | `test_harness_holds_no_drifted_copy_of_the_engine_vocabularies` | Internal adoption-measurement harness, not an adopter-facing surface. It grades artifacts from whichever build a cell ran, so it carries literal vocabularies — each one derived from the engine's and checked against it. | -| `mcp_server` | `src/agents_shipgate/mcp_server/` | — | — | Transport only. Every answer it returns is produced by calling the CLI in-process, so it has nothing of its own to drift. | +| `mcp_server` | `src/agents_shipgate/mcp_server/` | — | — | Transport only. Every answer it returns is produced by calling the CLI in-process. Its only words of its own are each tool's `tools/list` description, which enumerates no verdict set and names no pin or contract floor, and reads the schema IDs it names from the engine's constants, so it adds no claim; `tests/test_mcp_server.py` holds each description to its tool's inputs and boundary, in-process and over stdio, and CI's `mcp extra` job checks the installed command sends them (#800). No tool runs git, the Git-backed host comparison (`diff --base`), `scan` or `verify`. | | `design_partner_runbook` | `docs/design-partner-verifier-pilot.md` | `contract_floor`, `executable_pin`, `placeholder_ownership` | `test_executable_pin_resolves_in_a_published_channel`, `test_runbook_channel_table_states_the_released_contract_correctly`, `test_surface_routes_human_owned_placeholders_to_a_human`, `tests/test_adopter_pins_resolve.py::test_every_pin_init_writes_into_an_adopter_repo_names_the_published_release` | Version-specific by construction: it names one channel per partner and states what the released build does *not* emit. | | `human_entry_path` | `README.md`, `docs/quickstart.md` | `contract_floor`, `executable_pin`, `merge_verdict_vocabulary`, `placeholder_ownership`, `release_decision_vocabulary`, `report_schema_pin` | `test_executable_pin_resolves_in_a_published_channel`, `test_the_human_entry_path_states_what_the_published_build_provides`, `test_surface_enumerations_match_the_engine_vocabulary`, `test_surface_routes_human_owned_placeholders_to_a_human`, `test_surface_names_the_current_report_schema` | States its vocabularies as Markdown reference tables rather than braced sets; the vocabulary reader was taught that shape by [#498](https://github.com/ThreeMoonsLab/agents-shipgate/issues/498) rather than exempted from it. The quoted `diff` answers, the auto-detected `origin/HEAD` base, the documented missing-base recovery and the channel statements are held to this tree's output and to `.github/release-channels.json` by `tests/test_host_diff_entry_docs.py` (#779), which the claims vocabulary has no row for. The same module holds the build the pages say their quotes came from to a record of the answers the newest published release prints, taken from that build at release-runbook step 8 (#778). The quickstart's detector continue rule names `suggested_sources`, `codex_plugin_candidates` and `host_boundary_candidates`, so a host-only repository is not stopped where `detect` routes it forward; `tests/test_public_surface_contract.py::test_detect_continue_rules_do_not_stop_a_host_only_repository` holds it and `llms.txt`'s rule to those fields (#792). | | `agent_instructions` | `AGENTS.md`, `docs/agent-recipes.md`, `docs/agents/`, `docs/target-repo-agent-snippets.md` | `executable_pin`, `placeholder_ownership`, `report_schema_pin` | `test_executable_pin_resolves_in_a_published_channel`, `test_surface_routes_human_owned_placeholders_to_a_human`, `test_surface_names_the_current_report_schema` | States no contract floor of its own — it links [`agent-contract-current.md`](agent-contract-current.md) for the versions. What it does answer is who may fill a manifest placeholder, on the copy a coding agent actually reads, and it carries an Action ref, a `shipgate_version:` input and an install floor a reader runs. The `AGENTS.md` host-configuration review bullets name the manifest-free `diff` route and quote none of its output, so they add no claim; the quoted `diff` answers belong to the `human_entry_path` row (#792). | diff --git a/docs/mcp-server.md b/docs/mcp-server.md index ca766024b..c14c2a801 100644 --- a/docs/mcp-server.md +++ b/docs/mcp-server.md @@ -36,7 +36,15 @@ Claude Code registration (`.mcp.json`): | `shipgate.preflight` | `{workspace?, config?, plan?, changed_files?, diff_text?, capability_request?, base_preflight?}` | exact `PreflightResultV6` (preflight `0.6`) | | `shipgate.explain` | `{check_id}` or `{fingerprint, report_path}` | deterministic check/finding explanation JSON | | `shipgate.capabilities` | `{config}` or `{base_lock, head_lock}` | capability lock or capability lock diff JSON | -| `shipgate.handoff` | `{verifier_path, report_path?, verify_run_path?}` | exact `shipgate.agent_handoff/v8` | +| `shipgate.handoff` | `{verifier_path, report_path?, verify_run_path?}` | exact `shipgate.agent_handoff/v9` | + +Each tool's `tools/list` entry carries a description that a connected agent +reads when it picks a tool: the task first, then every input the tool accepts, +what it returns and what it cannot do. No tool runs git or the CLI's Git-backed +host comparison: `shipgate.check` takes diff text the caller already has and +obtains no base or head ref, and `shipgate.capabilities` compares two lock +files that already exist. For a comparison against a base ref, run +`agents-shipgate diff --base ` from the CLI. `shipgate.check` is the same protocol surface documented in [`agents/protocol.md`](agents/protocol.md). `shipgate.preflight` is proactive @@ -64,8 +72,10 @@ following a verification route or reporting completion. `shipgate.handoff` is a read-only projection over existing verifier artifacts. It never runs `verify`, shells out to git, or writes `agent-handoff.json`; it -returns the same `shipgate.agent_handoff/v8` shape that `verify` writes for -agents that need a compact control/release-readiness object. +returns the same `shipgate.agent_handoff/v9` shape that `verify` writes for +agents that need a compact control/release-readiness object. It does not check +that those artifacts still match the working tree: they describe only the tree +that `verify` run checked, so re-run `verify` after any change. ## Trust model diff --git a/scripts/mcp_extra_requirement.py b/scripts/mcp_extra_requirement.py index b02cdc910..944c1ac6a 100644 --- a/scripts/mcp_extra_requirement.py +++ b/scripts/mcp_extra_requirement.py @@ -7,7 +7,8 @@ `--stdio-smoke` starts the installed `agents-shipgate mcp-serve` as a real stdio MCP server, lists its tools through the SDK client, and exits non-zero unless -exactly the five read-only, closed-world tools answer. +exactly the five read-only, closed-world tools answer, each with its own +non-empty description. """ from __future__ import annotations @@ -69,7 +70,16 @@ def stdio_smoke() -> None: or annotations.open_world_hint is not False ): raise SystemExit(f"{tool.name} is not advertised read-only and closed-world") + # #800: a bare name tells a connected agent neither when to call the + # tool nor what it needs. The wording itself is held by + # tests/test_mcp_server.py; this proves the installed command sends it. + if not (tool.description or "").strip(): + raise SystemExit(f"{tool.name} is listed with an empty description") + if len({tool.description for tool in tools}) != len(tools): + raise SystemExit("mcp-serve listed two tools with the same description") print(f"mcp-serve answered over stdio with {len(tools)} read-only tools") + for tool in sorted(tools, key=lambda item: item.name): + print(f"{tool.name}: {tool.description}") def main(argv: list[str]) -> None: diff --git a/src/agents_shipgate/mcp_server/server.py b/src/agents_shipgate/mcp_server/server.py index 7b4a9564c..3ffcaf8d6 100644 --- a/src/agents_shipgate/mcp_server/server.py +++ b/src/agents_shipgate/mcp_server/server.py @@ -22,7 +22,9 @@ ) from agents_shipgate.core.errors import ConfigError from agents_shipgate.core.preflight import build_preflight_result -from agents_shipgate.schemas.preflight import CapabilityRequestV1 +from agents_shipgate.schemas.agent_boundary import AGENT_BOUNDARY_RESULT_SCHEMA_VERSION +from agents_shipgate.schemas.agent_handoff import AGENT_HANDOFF_SCHEMA_VERSION +from agents_shipgate.schemas.preflight import PREFLIGHT_SCHEMA_VERSION, CapabilityRequestV1 def shipgate_check( @@ -251,6 +253,95 @@ def _load_mcp_sdk() -> tuple[Any, Any]: return MCPServer, ToolAnnotations +# What an agent host shows a model when it chooses among these tools: the task +# first, then the inputs the handler accepts, what it returns and what it cannot +# do. Without them ``tools/list`` advertised five bare names (#800). Each one +# restates its handler above and docs/mcp-server.md; none may promise a Git +# comparison, a scan or verify run, or an authority the tool does not have. +_TOOL_DESCRIPTIONS = { + "shipgate.check": ( + "Check a coding agent's change against the agent-boundary policy for " + "host configuration and other boundary files (for example " + "`.codex/config.toml`, `.claude/settings.json`, `.mcp.json`), from a " + "unified diff you already have. Pass the change as `diff_text`; " + "optional `agent` (codex, claude-code or cursor; default codex), " + "`workspace` (holding the tree before or after the diff), `config` and " + "`policy`. The diff is detached: this tool runs no git and does not " + "obtain a base or head ref, so it is not the CLI's Git-backed host " + "comparison (`agents-shipgate diff --base `). Returns " + f"`{AGENT_BOUNDARY_RESULT_SCHEMA_VERSION}`: decision, affected hosts, " + "violations and advisory rows for only the host files the diff " + "changes. Because the diff is bound to no checkout, every result stops " + "for human routing (`control.state` `human_review_required`), " + "authorizes no verify command and grants no permission, even when " + "`decision` is `allow`. It runs no scan or verify and is not a release " + "or merge verdict." + ), + "shipgate.preflight": ( + "Before editing, route a planned change: does it touch a protected " + "release surface, need a human, need capability evidence, or need " + "verify? Pass either `plan` (preferred: a `PreflightPlanV1` object " + "with `changed_files`, `diff_text`, `capability_requests` and " + "`host_permission_requests`) or the direct inputs `changed_files`, " + "`diff_text`, `capability_request` " + "and `base_preflight` (an earlier preflight result to compare policy " + "and trust-root hashes against), never both: a mixed request is " + "rejected. Optional `workspace` and `config`. Returns a preflight " + f"result (`preflight_schema_version` `{PREFLIGHT_SCHEMA_VERSION}`): " + "protected-surface touches, signals, required evidence, " + "`requires_human_review`, `requires_verify` and a routing `control`. " + "Projection only: it is not a release verdict, every permission in its " + "`control` is false and its `control.state` is never `complete`; when " + "verification is owed it names a CLI `verify` command but runs " + "nothing. It runs no git, host comparison, scan or verify." + ), + "shipgate.explain": ( + "Explain an Agents Shipgate check, or one finding in an existing " + "report. Pass `check_id` (for example `SHIP-POLICY-APPROVAL-MISSING`) " + "for that check's catalog entry: what it checks, when it fires, its " + "default severity and recommendation. Or pass `fingerprint` with " + "`report_path` (a `report.json` that `scan` or `verify` already " + "wrote) for that finding's fields plus a deterministic explanation; " + "`fingerprint` requires `report_path` and takes precedence over " + "`check_id`. `no_plugins` keeps third-party check plugins out even " + "when AGENTS_SHIPGATE_ENABLE_PLUGINS is set. An unknown check ID or " + "fingerprint is an error. Lookup only: it produces no findings and " + "runs no scan, verify, git or host comparison." + ), + "shipgate.capabilities": ( + "Export or compare static capability locks: the tools a Shipgate " + "manifest declares for an agent, with their typed actions. Pass " + "`config` (manifest path, default shipgate.yaml) to build a lock from " + "the manifest's declared tool sources, or both `base_lock` and " + "`head_lock` (existing capability lock JSON files, as `verify` or " + "`capability export` write them) to diff them; one lock alone is " + "rejected, and a lock pair ignores `config`. `no_plugins` keeps " + "third-party adapters out even when AGENTS_SHIPGATE_ENABLE_PLUGINS is " + "set. Returns capability lock JSON or capability lock diff JSON; " + "neither is a release gate. It writes no lock, produces no findings " + "and runs no scan or verify. The diff reads only the two files given: " + "this tool " + "runs no git and checks out no base ref, so it is not the CLI's " + "Git-backed host comparison (`agents-shipgate diff --base `)." + ), + "shipgate.handoff": ( + "Read the compact agent handoff of a `verify` run that already " + "happened. Pass `verifier_path` (default " + "agents-shipgate-reports/verifier.json, which must exist); " + "`report_path` and `verify_run_path` default to `report.json` and " + "`verify-run.json` beside it and are read when present. Returns " + f"`{AGENT_HANDOFF_SCHEMA_VERSION}`, the projection `verify` writes: " + "release gate, control, blockers, remediation plan and " + "reproducibility identities. It is not a new verdict and grants " + "nothing the verifier did not; artifacts that disagree with each " + "other are refused. It runs no scan, verify, git or host comparison, " + "and does not check that the artifacts still match the working tree: " + "they describe only the tree that run verified, so re-run " + "`agents-shipgate verify` after any change." + ), +} + + def create_server(): MCPServer, ToolAnnotations = _load_mcp_sdk() @@ -273,7 +364,11 @@ def create_server(): ), ) - @server.tool(name="shipgate.check", annotations=read_only) + @server.tool( + name="shipgate.check", + description=_TOOL_DESCRIPTIONS["shipgate.check"], + annotations=read_only, + ) def _shipgate_check( agent: str = "codex", workspace: str = ".", @@ -289,7 +384,11 @@ def _shipgate_check( policy=policy, ) - @server.tool(name="shipgate.preflight", annotations=read_only) + @server.tool( + name="shipgate.preflight", + description=_TOOL_DESCRIPTIONS["shipgate.preflight"], + annotations=read_only, + ) def _shipgate_preflight( workspace: str = ".", config: str = "shipgate.yaml", @@ -309,7 +408,11 @@ def _shipgate_preflight( base_preflight=base_preflight, ) - @server.tool(name="shipgate.explain", annotations=read_only) + @server.tool( + name="shipgate.explain", + description=_TOOL_DESCRIPTIONS["shipgate.explain"], + annotations=read_only, + ) def _shipgate_explain( check_id: str | None = None, fingerprint: str | None = None, @@ -323,7 +426,11 @@ def _shipgate_explain( no_plugins=no_plugins, ) - @server.tool(name="shipgate.capabilities", annotations=read_only) + @server.tool( + name="shipgate.capabilities", + description=_TOOL_DESCRIPTIONS["shipgate.capabilities"], + annotations=read_only, + ) def _shipgate_capabilities( config: str = "shipgate.yaml", base_lock: str | None = None, @@ -337,7 +444,11 @@ def _shipgate_capabilities( no_plugins=no_plugins, ) - @server.tool(name="shipgate.handoff", annotations=read_only) + @server.tool( + name="shipgate.handoff", + description=_TOOL_DESCRIPTIONS["shipgate.handoff"], + annotations=read_only, + ) def _shipgate_handoff( verifier_path: str = "agents-shipgate-reports/verifier.json", report_path: str | None = None, diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py index 6dd1f87bf..c8f417d6c 100644 --- a/tests/test_mcp_server.py +++ b/tests/test_mcp_server.py @@ -4,6 +4,8 @@ import importlib.util import json +import os +import re import shutil import sys import tomllib @@ -431,5 +433,232 @@ def test_build_server_registers_read_only_tools() -> None: assert tool.annotations.open_world_hint is False, tool.name +#: Each tool's inputs as registered before #800 gave the tools descriptions: +#: JSON types and default per input, and none required. #800 changed what +#: ``tools/list`` says about a tool, never what it accepts. The SDK's cosmetic +#: ``title`` keys are left out so an SDK release that retitles them does not +#: read as an interface change. +_REGISTERED_INPUTS = { + "shipgate.check": { + "agent": (("string",), "codex"), + "workspace": (("string",), "."), + "diff_text": (("string",), ""), + "config": (("string",), "shipgate.yaml"), + "policy": (("null", "string"), None), + }, + "shipgate.preflight": { + "workspace": (("string",), "."), + "config": (("string",), "shipgate.yaml"), + "changed_files": (("array[string]", "null"), None), + "diff_text": (("null", "string"), None), + "capability_request": (("null", "object"), None), + "plan": (("null", "object"), None), + "base_preflight": (("null", "object"), None), + }, + "shipgate.explain": { + "check_id": (("null", "string"), None), + "fingerprint": (("null", "string"), None), + "report_path": (("null", "string"), None), + "no_plugins": (("boolean",), False), + }, + "shipgate.capabilities": { + "config": (("string",), "shipgate.yaml"), + "base_lock": (("null", "string"), None), + "head_lock": (("null", "string"), None), + "no_plugins": (("boolean",), False), + }, + "shipgate.handoff": { + "verifier_path": (("string",), "agents-shipgate-reports/verifier.json"), + "report_path": (("null", "string"), None), + "verify_run_path": (("null", "string"), None), + }, +} + +#: What each description must say besides naming every input (#800). The first +#: entry is the task it opens with, so a host that shows only the start still +#: shows when to call it; the rest are the input alternatives and boundaries its +#: handler actually has. ``check`` and ``capabilities`` both compare something, +#: so both say they are not the CLI's Git-backed host comparison. +_REQUIRED_STATEMENTS = { + "shipgate.check": ( + "Check a coding agent's change", + "from a unified diff you already have", + "this tool runs no git and does not obtain a base or head ref", + "not the CLI's Git-backed host comparison (`agents-shipgate diff --base `)", + "every result stops for human routing", + "`human_review_required`", + "authorizes no verify command and grants no permission", + "runs no scan or verify and is not a release or merge verdict", + ), + "shipgate.preflight": ( + "Before editing, route a planned change", + "Pass either `plan`", + "never both: a mixed request is rejected", + "Projection only: it is not a release verdict", + "every permission in its `control` is false", + "names a CLI `verify` command but runs nothing", + "runs no git, host comparison, scan or verify", + ), + "shipgate.explain": ( + "Explain an Agents Shipgate check, or one finding in an existing report", + "Pass `check_id`", + "Or pass `fingerprint` with `report_path`", + "`fingerprint` requires `report_path`", + "produces no findings and runs no scan, verify, git or host comparison", + ), + "shipgate.capabilities": ( + "Export or compare static capability locks", + "Pass `config`", + "or both `base_lock` and `head_lock` (existing capability lock JSON files", + "one lock alone is rejected", + "neither is a release gate", + "this tool runs no git and checks out no base ref", + "not the CLI's Git-backed host comparison (`agents-shipgate diff --base `)", + ), + "shipgate.handoff": ( + "Read the compact agent handoff of a `verify` run that already happened", + "which must exist", + "grants nothing the verifier did not", + "runs no scan, verify, git or host comparison", + "does not check that the artifacts still match the working tree", + "re-run `agents-shipgate verify` after any change", + ), +} + + +def _listed_tools() -> dict: + import asyncio + + return {tool.name: tool for tool in asyncio.run(build_server().list_tools())} + + +def _input_contract(schema: dict) -> tuple[list[str], dict]: + """Each input's JSON types and default, without the SDK's titles.""" + + def types(prop: dict) -> tuple[str, ...]: + return tuple( + sorted( + branch["type"] + + (f"[{branch['items']['type']}]" if "items" in branch else "") + for branch in prop.get("anyOf", [prop]) + ) + ) + + return sorted(schema.get("required", [])), { + name: (types(prop), prop.get("default")) + for name, prop in schema["properties"].items() + } + + +@pytest.mark.skipif( + not _HAS_MCP_SERVER_API, reason="requires the optional mcp extra (SDK 2.x)" +) +def test_every_tool_tells_an_agent_its_task_inputs_and_boundary() -> None: + """#800: ``tools/list`` advertised five tools with empty descriptions. + + A connected agent choosing between a provided-diff check, a plan + projection, an explanation, a lock comparison and an artifact read saw + only names. Each description now opens with the task, names every input + the tool accepts, and states what the tool cannot do: none runs git or the + CLI's Git-backed host comparison, scan or verify, or grants an authority. + """ + + listed = _listed_tools() + assert set(listed) == set(_REQUIRED_STATEMENTS) + descriptions = {name: tool.description or "" for name, tool in listed.items()} + for name, description in descriptions.items(): + assert description.strip(), f"{name} is listed with an empty description" + # Some model APIs cap a function description at 1024 characters, and a + # host forwarding these tools to one would cut off or refuse a longer + # one, losing the boundary statements that close each description. + assert len(description) <= 1024, (name, len(description)) + statements = _REQUIRED_STATEMENTS[name] + assert description.startswith(statements[0]), (name, description) + missing = [statement for statement in statements if statement not in description] + assert not missing, f"{name} no longer says {missing}" + unnamed = [ + argument + for argument in listed[name].input_schema["properties"] + if f"`{argument}`" not in description + ] + assert not unnamed, f"{name} does not tell an agent about inputs {unnamed}" + assert re.search(r"runs no [^.:;]*\bgit\b", description), ( + f"{name} must say it runs no git" + ) + assert len(set(descriptions.values())) == len(descriptions), ( + "two tools share a description, so an agent cannot tell their jobs apart" + ) + + +@pytest.mark.skipif( + not _HAS_MCP_SERVER_API, reason="requires the optional mcp extra (SDK 2.x)" +) +def test_descriptions_left_every_tool_name_and_input_unchanged() -> None: + listed = _listed_tools() + + assert set(listed) == set(_REGISTERED_INPUTS) + for name, tool in listed.items(): + required, inputs = _input_contract(tool.input_schema) + assert required == [], name + assert inputs == _REGISTERED_INPUTS[name], name + assert tool.annotations is not None, name + assert tool.annotations.read_only_hint is True, name + assert tool.annotations.open_world_hint is False, name + + +@pytest.mark.skipif( + not _HAS_MCP_SERVER_API, reason="requires the optional mcp extra (SDK 2.x)" +) +def test_stdio_tools_list_carries_the_descriptions_and_a_tool_still_answers( + tmp_path: Path, +) -> None: + """The protocol response, not just the in-process registry (#800). + + ``mcp-serve`` runs as a real stdio server in a scratch directory and is + asked over JSON-RPC what ``tools/list`` returns, then one existing tool is + called, so the descriptions an agent host receives are the registered ones + and a call behaves as before. + """ + + import asyncio + + from mcp import ClientSession + from mcp.client.stdio import StdioServerParameters, stdio_client + + async def over_stdio(errlog): + parameters = StdioServerParameters( + command=sys.executable, + args=["-m", "agents_shipgate", "mcp-serve"], + # The whole environment, so a run against a source tree on + # PYTHONPATH serves that tree rather than an installed copy. + env=dict(os.environ), + cwd=str(tmp_path), + ) + async with stdio_client(parameters, errlog=errlog) as (read, write): + async with ClientSession(read, write) as session: + await session.initialize() + tools = (await session.list_tools()).tools + called = await session.call_tool( + "shipgate.explain", + {"check_id": "SHIP-DOC-MISSING-DESCRIPTION", "no_plugins": True}, + ) + return tools, called + + with (tmp_path / "server-stderr.log").open("w", encoding="utf-8") as errlog: + tools, called = asyncio.run(asyncio.wait_for(over_stdio(errlog), timeout=120)) + + registered = _listed_tools() + served = {tool.name: tool for tool in tools} + assert set(served) == set(registered) + for name, tool in served.items(): + assert tool.description, f"{name} reached the client without a description" + assert tool.description == registered[name].description, name + assert tool.input_schema == registered[name].input_schema, name + assert tool.annotations == registered[name].annotations, name + assert called.is_error is False + assert (called.structured_content or {}).get("id") == "SHIP-DOC-MISSING-DESCRIPTION" + assert sorted(path.name for path in tmp_path.iterdir()) == ["server-stderr.log"] + + def _write_json(path: Path, payload: dict) -> None: path.write_text(json.dumps(payload), encoding="utf-8")