Conversation
Most pi_rpc tests use a fake Pi, so they stay green when Pi's real completion and event contracts drift. Add tests that run the adapter against the installed Pi with an isolated agent directory and home, offline mode, and a scripted loopback OpenAI-compatible provider with a dummy key. Reviewer tool calls go through the real __pi-review-tool helper, which TestMain dispatches the way the cr binary does. The tests cover structured JSON without tools, a 503 followed by a successful retry, permanent provider failures, a multi-turn reviewer run with the diff-first gate, a path escape, and exact usage, cache, and cost totals, a denied shell tool, and caller cancellation and task deadlines with process-group cleanup. Hostile global and project instructions, extensions, MCP servers, and skills must not load. Ordinary go test skips them. make test-pi-runtime sets CR_PI_RUNTIME_VERSION from the Makefile pin, so a missing or different Pi fails. A new non-required pi-runtime CI job installs that version with npm --ignore-scripts on pinned Node 22.22.3 and runs the target.
A blank PI_RUNTIME_VERSION made make test-pi-runtime export an empty CR_PI_RUNTIME_VERSION, so every runtime test skipped and the target reported ok. The same value made pi-runtime-version print nothing, which the CI install step would turn into an unpinned, latest Pi install. Both Make targets now require exactly one version word before doing anything. The runtime tests also fail, rather than skip, when the opt-in variable is set but blank. The CI step reads the version in its own assignment so a failed lookup stops the step before npm runs. A focused test runs both targets against a fake go and checks that blank, whitespace, and multi-word pins fail without dispatching go test. The development guide now states that the runtime fixture's isolation does not extend to production agent directories (#642).
zzwong
added this pull request to stack #647
October 3, 2026 17:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #639. Stacked on #641; merge #641 first.
Summary
Most
pi_rpctests use a fake Pi, so they can miss changes in Pi's real contracts. Add credential-free installed-Pi tests and a pinned CI check. No production code changed.HOME, offline flags, a dummy key, and loopback. Reviewer calls dispatch the real__pi-review-toolhelper throughTestMain.pi-runtimeCI with pinned Node 22.22.3/action revisions andnpm install --ignore-scripts. A failed version lookup stops before installation. Existing check identities remain unchanged.Test plan
make test-pi-runtimepasses on macOS and in a Linux arm64/root container; five repeated race runs pass.GOFLAGS=-tags=keyring_nopassage go test ./..., Go 1.26.3 lint,actionlint, andgit diff --checkpass.pi-runtimeon x64/non-root.Limits
Evidence is local/mock, not deployed. Process-group assertions are Unix-only; Windows has no runtime coverage.
Hostile instructions at the default HOME location and in project resources are tested under a fresh selected agent directory. This does not establish production ambient-instruction isolation: selected-agent
AGENTS.mdaffects non-reviewers, andAPPEND_SYSTEM.mdaffects both modes. Follow-up #642 tracks that fix. Codemode remains separate in #640.