fix(evals): honor disabled harness observations

[shrey150]

Summary

• honor EVAL_HARNESS_OBSERVATIONS=none when the Claude Code adapter mounts a browser tool
• centralize the recorder-construction gate
• add a regression test for the disabled-observation path

Why

This surfaced while comparing the Eve package in #2822 with the claude_code + stagehand_code Online-Mind2Web baseline on live Browserbase sessions. The harness already exposes and tests EVAL_HARNESS_OBSERVATIONS=none, but the Claude Code adapter constructed ObservationRecorder unconditionally whenever the tool exposed captureEvidence.

That made the setting ineffective. On pages with stale or detached frames, every completed run-tool call started another accessibility-tree capture. The timeout returned control to the harness without cancelling the underlying capture, so the captures continued emitting DOM.getFrameOwner and Accessibility.getFullAXTree failures and prevented a useful baseline run.

Minimal reproduction before this change:

EVAL_HARNESS_OBSERVATIONS=none \
EVAL_ONLINEMIND2WEB_IDS=<task-id> \
node packages/evals/dist/cli/cli.js run b:onlineMind2Web \
  --harness claude_code --tool stagehand_code \
  -e browserbase -m anthropic/claude-haiku-4-5 -t 1 -c 1

Despite none, the adapter still invoked per-step evidence capture. After this change, the same five-case live run emitted no per-step stale-frame flood and all owned Browserbase sessions completed. A later normal-budget run exposed a separate unhandled page.goto RPC-timeout failure; that is not addressed here.

This is a generic eval-harness fix rather than an Eve adapter. Eve's native eval runner already provides the package/tool-call evidence needed for #2822.

Validation

• 35 focused framework tests
• eval package typecheck
• eval CLI build
• live claude_code + stagehand_code Browserbase run with observations disabled

No changeset is included because the eval package is private.

---

Summary by cubic

Honors EVAL_HARNESS_OBSERVATIONS=none in the Claude Code adapter so the setting actually suppresses per-step evidence capture instead of always recording when the tool exposes captureEvidence.

• The adapter previously constructed an ObservationRecorder unconditionally on captureEvidence, so the none setting had no effect and per-step accessibility-tree captures kept running.
• Centralizes the recorder-construction gate in a new createObservationRecorder() that returns undefined when observations are disabled or no capture is provided.
• Adds a regression test covering the disabled-observation path; behavior with observations enabled is unchanged.

^{Written for commit 74ba0a9. Summary will update on new commits.}

[changeset-bot]

🦋 Changeset detected

Latest commit: 74ba0a9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 11 packages

Name	Type
@browserbasehq/stagehand-python	Patch
@browserbasehq/stagehand-extension	Patch
@browserbasehq/stagehand-go	Patch
@browserbasehq/stagehand	Patch
@browserbasehq/stagehand-integrations	Patch
@browserbasehq/stagehand-integrations-example-eve-facade	Patch
@browserbasehq/stagehand-integrations-example-pi-facade	Patch
@browserbasehq/stagehand-integrations-example-claude-code-facade	Patch
@browserbasehq/stagehand-integrations-example-codex-facade	Patch
@browserbasehq/stagehand-integrations-example-mastra-facade	Patch
@browserbasehq/stagehand-integrations-example-vercel-ai-facade	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[cubic-dev-ai]

No issues found across 3 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant CLI as Eval CLI
    participant Harness as External Harness
    participant Adapter as Claude Code Adapter
    participant Recorder as ObservationRecorder
    participant Browser as Browserbase Session

    Note over CLI,Browser: Claude Code Adapter Runtime Flow

    CLI->>Harness: Launch external harness (claude_code)
    Harness->>Adapter: prepareAgentMountAdapter(input)
    Adapter->>Adapter: read env EVAL_HARNESS_OBSERVATIONS

    alt EVAL_HARNESS_OBSERVATIONS=none
        Note over Adapter: Observations disabled
        Adapter->>Adapter: createObservationRecorder() returns undefined
        Adapter-->>Harness: Adapter ready without recorder
        Harness->>Browser: Run agent tool calls
        Note over Harness,Browser: No per-step evidence capture - no accessibility tree scans
        Browser-->>Harness: Clean tool results
    else Observations enabled (default/all)
        Note over Adapter: Observations enabled
        Adapter->>Adapter: check running.captureEvidence exists
        Adapter->>Recorder: new ObservationRecorder(captureEvidence)
        Recorder-->>Adapter: Recorder instance
        Adapter-->>Harness: Adapter ready with recorder
        Harness->>Browser: Run agent tool calls
        Browser-->>Harness: Tool result
        Harness->>Recorder: record(step observation)
        Recorder->>Browser: Capture accessibility tree evidence
        Browser-->>Recorder: ProbeEvidence
        Recorder-->>Harness: Indexed observation
    end

    Note over CLI,Harness: Fallback path when no capture provided
    alt No captureEvidence available
        Adapter->>Adapter: createObservationRecorder(undefined) returns undefined
        Adapter-->>Harness: Adapter ready without recorder
    end

Loading

_{Re-trigger cubic}