Skip to content

fix(mcp): describe each tool's task, inputs and boundary in tools/list (#800) - #1114

Open
pengfei-threemoonslab wants to merge 2 commits into
mainfrom
claude/issue-800-mcp-tool-descriptions
Open

pengfei-threemoonslab wants to merge 2 commits into
mainfrom
claude/issue-800-mcp-tool-descriptions

Conversation

@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor

Summary

Closes #800.

create_server() registered all five MCP tools with @server.tool(name=..., annotations=read_only) on inner wrappers that had no description= and no docstring. The docstrings were on the outer shipgate_* functions, which are not the registered objects. So tools/list listed shipgate.check, shipgate.preflight, shipgate.explain, shipgate.capabilities and shipgate.handoff with description: "". That is the defect SHIP-DOC-MISSING-DESCRIPTION reports in other projects. I reproduced it on origin/main (4809636) at the extra's floor (mcp==2.0.0) and newest (mcp==2.3.0), in-process and over stdio.

  • src/agents_shipgate/mcp_server/server.py: each tool now passes description= through MCPServer.tool, which accepts description at both 2.0.0 and 2.3.0. I wrote each description from its handler and docs/mcp-server.md. Each one opens with the task, names every input the tool accepts, says what it returns and states what it cannot do:
    • shipgate.check takes caller-provided, detached diff text. It runs no git, obtains no base or head ref, and is not the CLI's Git-backed agents-shipgate diff --base <ref>. Every result stops at human_review_required, authorizes no verify command and grants no permission, even when decision is allow.
    • shipgate.preflight takes a plan or the direct inputs, never both, and is projection only. It is not a release verdict, every control permission is false and its control.state is never complete. It may name a CLI verify command but runs nothing.
    • shipgate.explain takes check_id, or fingerprint with report_path (an existing report.json). fingerprint requires report_path and takes precedence over check_id. It produces no findings.
    • shipgate.capabilities takes config, or both base_lock and head_lock (existing lock files). One lock alone is rejected. It runs no git, checks out no base ref and is not the Git-backed host comparison.
    • shipgate.handoff reads an existing verifier.json, plus the report.json and verify-run.json beside it when present. It runs no verify, does not check that those artifacts still match the working tree, and grants nothing the verifier did not.
    • The schema IDs a description names (shipgate.agent_boundary_result/v3, preflight 0.6, shipgate.agent_handoff/v9) are read from the engine's constants, not copied. Wrapper names and signatures are unchanged, so input schemas, output schemas and annotations are byte-identical to origin/main's.
  • scripts/mcp_extra_requirement.py --stdio-smoke (CI's mcp extra jobs) now also fails when the installed agents-shipgate mcp-serve lists an empty or duplicated description. It prints the five descriptions to the job log.
  • docs/mcp-server.md: one paragraph on the descriptions and the Git-backed-comparison boundary. The handoff output row and paragraph said shipgate.agent_handoff/v8; the server returns v9, so they now say v9. The handoff paragraph also gains the currentness limit.
  • docs/distribution-surfaces.md: the mcp_server row said the server "has nothing of its own to drift". It now says the descriptions are its own words, carry no claim, and are held by tests/test_mcp_server.py and the stdio smoke. Claims are unchanged (none), so tests/test_distribution_surface_parity.py needs no change, and it passes.
  • CHANGELOG.md: one line under ### Changes, after the env_vars entry.

Type

  • Check or risk-model change
  • Input adapter change
  • CLI or GitHub Action behavior
  • Report, schema, or SARIF output
  • Documentation only

Authorship

  • A coding agent wrote some or all of this change. Agent: Claude Code (Claude Opus 5.5)
  • A person read the whole diff before opening this pull request.

Acceptance criteria

  • All five registered tools have nonempty, accurate, task-specific descriptions in the actual SDK and stdio tools/list responses on the supported installed MCP extra.
    • SDK registry: tests/test_mcp_server.py::test_every_tool_tells_an_agent_its_task_inputs_and_boundary reads build_server().list_tools(). For each tool it asserts a description that is non-empty, at most 1024 characters and opens with that tool's task. It must contain that tool's boundary statements and name every input in the tool's input schema in backticks. All five descriptions must be distinct.
    • Protocol: test_stdio_tools_list_carries_the_descriptions_and_a_tool_still_answers starts python -m agents_shipgate mcp-serve as a real stdio server in a scratch cwd. It runs initialize and tools/list over JSON-RPC and asserts each served description, input schema and annotation equals the registered one. Then it calls shipgate.explain.
    • Installed command: CI's stdio smoke (mcp extra (floor), mcp extra (newest)) now fails on an empty or duplicate description.
    • Fails without the fix (server.py reverted to origin/main, mcp 2.0.0): both tests fail. The smoke exits 1 with shipgate.check is listed with an empty description.
  • Descriptions state valid input alternatives and the comparison/evidence boundary; none implies Git-backed host diff, scan/verify execution or merge authority that the tool does not provide.
    • Each description must say it runs no git, checked by a regex over all five. check and capabilities must say they are not agents-shipgate diff --base <ref>. check, preflight, capabilities and handoff carry their authority limits as required statements (see the description text below).
    • I recorded every subprocess the five handlers start, in a real git checkout (subprocess.Popen hooked): none for check, preflight (diff and empty plan), capabilities (export and diff) or explain. I also checked the stated behaviours against the handlers: empty-diff check gives decision=allow, control.state=human_review_required, allowed_next_commands=[] and all permissions false. Empty-plan preflight gives planning_only. The error cases (one lock, fingerprint without report, missing verifier) raise the errors the descriptions name.
  • Regression tests would fail on the five empty descriptions and distinguish the tools' actual jobs; existing names, input schemas, result semantics and read-only annotations remain compatible.
    • test_descriptions_left_every_tool_name_and_input_unchanged pins each input's JSON types and default, with nothing required, to origin/main's registration. It leaves out the SDK's cosmetic title keys, so an SDK release that retitles them does not fail CI. It also pins the read-only and open-world annotations. Perturbation check: changing one capabilities default fails it.
    • Full catalogs dumped from origin/main and this branch over stdio, at mcp 2.0.0 and 2.3.0, are identical in name, inputSchema, outputSchema and annotations. Only description differs (0 characters → 701–989). The branch catalog is identical at both SDK versions.
    • The existing handler tests (result semantics) pass unchanged.
  • Installed-build evidence records package/SDK versions and a successful minimal existing-tool call. See below. I claim no agent-selection improvement; Measure coding-agent discovery and voluntary selection separately from explicit execution #799 owns that measurement.

Installed-build evidence

I built a wheel from this branch (uv build --wheel, agents_shipgate-1.2.0-py3-none-any.whl: the source version, not a published release). I installed it with mcp<3 into a fresh venv and ran it outside the checkout:

  • agents_shipgate imported from …/installed/lib/python3.12/site-packages/agents_shipgate/, version 1.2.0; mcp 2.3.0.
  • scripts/mcp_extra_requirement.py --stdio-smoke against the installed agents-shipgate mcp-serve: exit 0, mcp-serve answered over stdio with 5 read-only tools.
  • Minimal call over stdio: shipgate.explain {"check_id": "SHIP-DOC-MISSING-DESCRIPTION", "no_plugins": true} returned is_error=False with structured_content.id == "SHIP-DOC-MISSING-DESCRIPTION".
  • The installed catalog equals the source-tree catalog field for field. Before the fix, every description was '' at both 2.0.0 and 2.3.0.
The five descriptions as stdio tools/list returns them (installed wheel, mcp 2.3.0)
  • shipgate.capabilities (845 chars): Export or compare static capability locks: the tools a Shipgate manifest declares for an agent, with their typed actions. Pass config (manifest path, default shipgate.yaml) to build a lock from the manifest's declared tool sources, or both base_lock and head_lock (existing capability lock JSON files, as verify or capability export write them) to diff them; one lock alone is rejected, and a lock pair ignores config. no_plugins keeps third-party adapters out even when AGENTS_SHIPGATE_ENABLE_PLUGINS is set. Returns capability lock JSON or capability lock diff JSON; neither is a release gate. It writes no lock and runs no findings, scan or verify. The diff reads only the two files given: this tool runs no git and checks out no base ref, so it is not the CLI's Git-backed host comparison (agents-shipgate diff --base <ref>).
  • shipgate.check (989 chars): Check a coding agent's change against the agent-boundary policy for host configuration and other boundary files (for example .codex/config.toml, .claude/settings.json, .mcp.json), from a unified diff you already have. Pass the change as diff_text; optional agent (codex, claude-code or cursor; default codex), workspace (holding the tree before or after the diff), config and policy. The diff is detached: this tool runs no git and does not obtain a base or head ref, so it is not the CLI's Git-backed host comparison (agents-shipgate diff --base <ref>). Returns shipgate.agent_boundary_result/v3: decision, affected hosts, violations and advisory rows for only the host files the diff changes. Because the diff is bound to no checkout, every result stops for human routing (control.state human_review_required), authorizes no verify command and grants no permission, even when decision is allow. It runs no scan or verify and is not a release or merge verdict.
  • shipgate.explain (701 chars): Explain an Agents Shipgate check, or one finding in an existing report. Pass check_id (for example SHIP-POLICY-APPROVAL-MISSING) for that check's catalog entry: what it checks, when it fires, its default severity and recommendation. Or pass fingerprint with report_path (a report.json that scan or verify already wrote) for that finding's fields plus a deterministic explanation; fingerprint requires report_path and takes precedence over check_id. no_plugins keeps third-party check plugins out even when AGENTS_SHIPGATE_ENABLE_PLUGINS is set. An unknown check ID or fingerprint is an error. Lookup only: it produces no findings and runs no scan, verify, git or host comparison.
  • shipgate.handoff (765 chars): Read the compact agent handoff of a verify run that already happened. Pass verifier_path (default agents-shipgate-reports/verifier.json, which must exist); report_path and verify_run_path default to report.json and verify-run.json beside it and are read when present. Returns shipgate.agent_handoff/v9, the projection verify writes: release gate, control, blockers, remediation plan and reproducibility identities. It is not a new verdict and grants nothing the verifier did not; artifacts that disagree with each other are refused. It runs no scan, verify, git or host comparison, and does not check that the artifacts still match the working tree: they describe only the tree that run verified, so re-run agents-shipgate verify after any change.
  • shipgate.preflight (978 chars): Before editing, route a planned change: does it touch a protected release surface, need a human, need capability evidence, or need verify? Pass either plan (preferred: a PreflightPlanV1 object with changed_files, diff_text, capability_requests and host_permission_requests) or the direct inputs changed_files, diff_text, capability_request and base_preflight (an earlier preflight result to compare policy and trust-root hashes against), never both: a mixed request is rejected. Optional workspace and config. Returns a preflight result (preflight_schema_version 0.6): protected-surface touches, signals, required evidence, requires_human_review, requires_verify and a routing control. Projection only: it is not a release verdict, every permission in its control is false and its control.state is never complete; when verification is owed it names a CLI verify command but runs nothing. It runs no git, host comparison, scan or verify.

Verification

The local .venv holds mcp 1.27.2, where the SDK tests skip. So I ran tests/test_mcp_server.py in two scratch venvs built like CI's mcp-extra job: constraints/dev.txt and constraints/build-backend.txt with --require-hashes, an editable install of this worktree, then the SDK at the floor or the newest admitted release.

Command Exit
python -m pytest tests/test_mcp_server.py -rs (mcp 2.0.0) 0 (17 passed, 1 skipped: the no-SDK error path)
python -m pytest tests/test_mcp_server.py -rs (mcp 2.3.0) 0 (17 passed, 1 skipped)
same, server.py reverted to origin/main (mcp 2.0.0) 1 (the description test and the stdio test fail)
python scripts/mcp_extra_requirement.py --stdio-smoke (mcp 2.0.0 and 2.3.0, editable) 0, 0
same, server.py reverted to origin/main 1 (shipgate.check is listed with an empty description)
same, installed wheel outside the checkout (mcp 2.3.0) 0
PYTHONPATH=src .venv/bin/python -m pytest tests/test_mcp_server.py tests/test_distribution_surface_parity.py tests/test_docs_links.py tests/test_public_surface_contract.py tests/test_release_pipeline.py tests/test_local_contract.py tests/test_agent_instructions_renderers.py -n 4 0 (1118 passed, 6 skipped)
python -m ruff check . 0

Release-readiness notes

  • No user-code import added to default scan paths
  • No network access added to default scan paths
  • New or changed check IDs are documented in docs/checks.md (n/a: no check changes)
  • Report/schema changes are additive or documented in STABILITY.md (n/a: no report or schema change. Only the MCP tools/list description field changes, from empty to text.)

Out of scope

  • Other copies that list the MCP tools, left alone for follow-up. AGENTS.md:889 and llms-full.txt:914 name the five tools with no descriptions; another v1.3.0 PR edits those files. docs/agents/protocol.md still says the handoff returns shipgate.agent_handoff/v7. docs/integrations.md:367 and :390 and docs/INDEX.md:26 list four tools and omit shipgate.handoff.
  • Agent-selection effect: Measure coding-agent discovery and voluntary selection separately from explicit execution #799 measures it, and this PR claims none.
  • Dogfooding SHIP-DOC-MISSING-DESCRIPTION on this server is not possible yet. The repository's static MCP reader cannot follow MCPServer to a server because _load_mcp_sdk() resolves it at runtime. A scan of server.py reports server_binding_not_proven for each registration and SHIP-INVENTORY-NOT-ENUMERABLE, before and after this change.
  • Input-schema redesign, such as marking diff_text required or making agent an enum: the issue defers it. Schemas are unchanged here.

🤖 Generated with Claude Code

#800)

The five MCP tools were registered on wrappers with no description and no
docstring, so tools/list advertised shipgate.check, shipgate.preflight,
shipgate.explain, shipgate.capabilities and shipgate.handoff with empty
descriptions: the defect SHIP-DOC-MISSING-DESCRIPTION reports in other
projects.

Each tool now passes description= through MCPServer.tool, which exists at
the extra's floor (mcp 2.0.0) and newest (2.3.0). Each description opens
with the task, names every input the tool accepts, says what it returns,
and states its boundary as implemented: check takes a caller-provided
detached diff and obtains no base or head ref; capabilities builds a lock
from a config or diffs two existing locks; preflight is projection only;
explain takes a check ID, or a fingerprint with a report path; handoff
reads existing verify artifacts and does not check they still match the
working tree. None runs git, the Git-backed host comparison
(agents-shipgate diff --base), scan or verify. Schema IDs are read from
the engine's constants. Names, input and output schemas and annotations
are unchanged.

Tests hold the descriptions over the SDK's list_tools() and over a real
stdio tools/list, plus one tool call; the CI stdio smoke now fails on an
empty or duplicated description.
"runs no findings, scan or verify" read as if findings were something a
tool runs. The description now says the tool writes no lock, produces no
findings and runs no scan or verify, matching the explain tool's wording.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Published MCP tools have empty descriptions: expose task and input boundaries in tools/list

1 participant