ocAdvisor model-aware gate and refreshable Artificial Analysis snapshot — implementation plan #8

Plan file: /home/ubuntu/.opencode/plan/2026-09-19-ocadvisor-artificial-analysis-benchmarks.md

Tracking issue: #8
Plan file: /home/ubuntu/.opencode/plan/2026-09-19-ocadvisor-artificial-analysis-benchmarks.md
Repository: pfoundation/ocAdvisorTool

Objective

For the implementing agent: Switch to Build mode and use the executing-plans skill to execute this plan task by task. Use GitHub Helper for progress and bot-authored commits. The user requested no further advisor-tool calls in this session.

Goal: Give the TypeSafe gate accurate requester/advisor model identities and locally cached Artificial Analysis capability scores, with a packaged CLI that refreshes the data between plugin releases and makes it available to the next consultation without restarting OpenCode.

Architecture: A shared benchmark module validates a versioned JSON snapshot and maps exact OpenCode model/effort references to stable Artificial Analysis model IDs. A manual CLI fetches and atomically publishes the user snapshot. The plugin loads a consistent snapshot into memory, detects replacement before consultations, and adds compact profiles and comparable score differences to the existing gate state. One TypeSafe request continues to judge consultation value and, when available, choose advisor effort.

Tech stack: TypeScript, Bun, existing OpenCode V2 plugin hooks/tools, Bun SQLite session metadata reads, native fetch and filesystem APIs, @typesafe-ai/sdk, Bun tests, existing TypeScript build and npm packaging.

Confirmed requirements

1. Supply the identity of the model making the advisor request and the model selected as advisor, including reasoning-effort information when known.
2. Use Artificial Analysis coding and reasoning-related results as measured capability evidence.
3. Keep a local database snapshot for fast lookups; benchmark data is read locally on the consultation path.
4. Add a quick CLI refresh command usable between plugin releases.
5. Refreshed snapshots survive package upgrades and become visible to an already-running plugin on its next consultation.
6. Model aliases/effort mappings must also be maintainable locally between releases.
7. Preserve the existing consultation-value policy: benchmark differences inform the likely benefit of advice; a stronger requester can still benefit from independent review.

Discovery baseline and source map

Planning baseline: clean checkout /home/ubuntu/dev/ocAdvisor, branch fix/release-versioning, commit acc9cc5; origin https://github.com/pfoundation/ocAdvisorTool.git. Recheck the checkout and default branch at implementation start and use a task-specific feature branch while preserving unrelated work.

Related work: the TypeSafe gate plan is issue #3 (closed). Issue #6 is an open, separate caller-model opt-out plan that overlaps src/ocAdvisor.ts, its tests, and README. It is not a prerequisite. Inspect its state before editing, coordinate overlapping changes, and preserve its guard ordering if it has landed. Do not fold that feature into this issue.

Current implementation anchors:

• src/ocAdvisor.ts:415-427,602-640: SessionModel, callerLabel, and session-model lookup; the label currently discards variant.
• src/ocAdvisor.ts:1141-1245: catalog lookup, advisor variant validation, and supported effort intersection.
• src/ocAdvisor.ts:1648-1923: runAdvisor, gate input construction, generation, and metrics.
• src/ocAdvisor.ts:1932-2021: direct tool registration and context hook; the hook has the outgoing model, but the local type currently omits variant/session identifiers.
• src/typesafeState.ts:181-267,379-442: decision-state contract, shell builder, transcript collection, and budget accounting.
• src/typesafeGate.ts:37-45,236-298,402-567: judgment instructions, final wire-payload projection, trimming, and gate decisions. Both the builder and final projection must carry new fields.
• src/advisorGate.test.ts, src/typesafeState.test.ts, src/typesafeGate.test.ts: fixture DB, runtime and transport seams.
• src/typesafeGate.eval.ts and src/fixtures/typesafeGate.cases.json: optional live gate evaluation.
• package.json has no CLI bin; only dist, README and LICENSE are published. tsconfig.build.json explicitly includes four entrypoints. A new CLI needs build inclusion and an executable packaged entrypoint.
• .github/workflows/ci.yml runs tests/typecheck/build and verifies the root shim plus tarball allowlist.

Authoritative references checked during planning:

• Artificial Analysis API: https://artificialanalysis.ai/documentation ; Context7 /websites/artificialanalysis_ai_api-reference.
• Intelligence methodology: https://artificialanalysis.ai/methodology/intelligence-benchmarking . The page currently identifies Intelligence Index v4.3.2; the documented API response does not guarantee that version metadata is attached to each record.
• OpenCode V2 plugins: https://opencode.ai/v2/docs/build/plugins . Verify request/tool metadata against installed V2 types/source before implementing caller identity propagation.
• TypeSafe state: https://docs.typesafe.ai/concepts/state.md ; supporting facts belong in named state fields, and the need/effort judgments remain independent questions over that state.

The official AA endpoint is GET https://artificialanalysis.ai/api/v2/data/llms/models, authenticated with x-api-key. Its documented response contains stable id and model_creator.id, mutable name/slug, and evaluations. Example metric keys are artificial_analysis_coding_index, artificial_analysis_intelligence_index, artificial_analysis_math_index, gpqa, hle, livecodebench, and scicode. There is no documented dedicated reasoning_index, structured effort field, evaluation timestamp, or methodology version in that response. The authenticated response and exact coverage of the configured models remain to be checked in implementation. Documentation example scores are not seed data.

Design contract

1. Snapshot schema and metric semantics

Use a compact JSON database loaded into maps in memory. Shared runtime validation applies to downloads, the bundled seed, local snapshots, mappings, CLI status, and plugin reads. Introduce a dedicated schema version independent of npm package version.

Suggested normalized contract (final exported names may follow repository conventions):

type MetricDefinition = {
  key: string;
  label: string;
  category: "coding" | "general" | "reasoning" | "math";
  unit: "index_points" | "fraction";
  higherIsBetter: boolean;
  benchmarkVersion: string | null;
};

type BenchmarkModel = {
  id: string;                 // Stable Artificial Analysis ID.
  creatorID: string;
  name: string;
  slug: string;
  evaluatedEffort: string | null;
  evaluatedAt: string | null;
  evaluations: Record<string, number | null>;
};

type BenchmarkSnapshot = {
  schemaVersion: 1;
  source: "artificial-analysis";
  sourceURL: "https://artificialanalysis.ai/";
  endpoint: string;
  fetchedAt: string;          // Download time, never evaluation time.
  contentHash: string;        // Digest of canonical normalized content.
  methodologyVersion: string | null;
  metricDefinitions: MetricDefinition[];
  models: BenchmarkModel[];
};

Preserve finite published scores, including zero; retain missing values as null. Enforce metric-specific units/ranges only where established by the source contract. Treat source index scores and fractional benchmark scores separately. Preserve unknown finite evaluation fields in storage for forward compatibility but send only recognized metric definitions to the gate. Reject empty/unusable datasets, duplicate stable IDs, malformed identities, unsupported local schema versions, and invalid known scores before activation. Ignore unrelated additive upstream fields.

Primary compact metrics: Coding Index and Intelligence Index, plus at most two documented reasoning/math results that are available and relevant (prefer HLE and GPQA when supplied; Math Index may be included where appropriate). Keep AA's published labels. Intelligence Index is broad capability evidence, not a renamed reasoning score. The metric selection order is deterministic, shared by both model profiles, and tested. Do not manufacture an aggregate capability score or infer one benchmark from another.

Record evaluation dates, effort and methodology version only from verified source metadata or explicit curated metadata with provenance. Null is the correct value when unavailable. Download time and snapshotAgeDays describe snapshot freshness, not when a model was evaluated. Include AA attribution in the snapshot, README and CLI output.

2. Paths, precedence and compatible releases

Default user snapshot: $XDG_DATA_HOME/opencode/ocadvisor/artificial-analysis.json, or ~/.local/share/opencode/ocadvisor/artificial-analysis.json when XDG_DATA_HOME is absent. Use an absolute XDG path; fall back to the home default if the environment value is relative. Keep the refreshed snapshot outside the package installation directory.

Add plugin options:

{
  "benchmarks": {
    "path": "/absolute/path/artificial-analysis.json",
    "mappingsPath": "/absolute/path/model-mappings.json"
  }
}

Environment fallbacks: OCADVISOR_BENCHMARKS_PATH and OCADVISOR_BENCHMARK_MAPPINGS_PATH. Resolution is explicit plugin option, then environment, then shared default. Validate explicit paths; document absolute paths and avoid implicit cwd-dependent resolution. The CLI uses --path, then the same environment variable, then the same default. It does not attempt to discover every OpenCode project's plugin options: when a project sets a custom path, the documented CLI command must pass that path or use the shared environment variable. Default mappings file is model-mappings.json beside the selected user snapshot; --mappings-path/the mapping environment variable selects another location.

Ship a genuine validated seed as src/data/artificialAnalysis.snapshot.json, copied or emitted as dist/data/artificialAnalysis.snapshot.json. The build must explicitly guarantee inclusion. Keep a small curated baseline mapping file src/data/artificialAnalysis.mappings.json, likewise shipped under dist. A compatible valid user snapshot takes precedence over the bundled seed and is never overwritten by installation/build/release scripts. Each file is a complete snapshot; never fill missing model scores from a different vintage. Unsupported user schema versions or corrupt replacement data keep an already-loaded valid snapshot, otherwise use a valid bundled seed, otherwise supply explicit unavailable benchmark evidence. Surface the active source and rejected-candidate reason in status/metrics without breaking advisor consultation.

3. Exact model and effort mapping

Preserve the actual serving provider and complete model ID in the gate identity, including IDs with additional slashes. Map that exact identity to AA using explicit curated bindings to stable AA IDs. Do not fuzzy-match names, strip arbitrary provider prefixes, choose the highest-scoring sibling, or assume two effort settings are equivalent.

Suggested local mapping format:

{
  "schemaVersion": 1,
  "bindings": [
    {
      "providerID": "provider-id",
      "modelID": "complete/model-id",
      "variant": "xhigh",
      "aaModelID": "verified-stable-aa-id",
      "evaluatedEffort": "verified-source-effort-or-null",
      "evidenceURL": "https://artificialanalysis.ai/"
    }
  ]
}

The example contains placeholders, not real bindings. In actual JSON, unknown evaluatedEffort is null. Binding keys are exact (providerID, modelID, variant) tuples; variant null means specifically unknown/unset, never a wildcard. Local bindings override bundled bindings for the same tuple; duplicate/conflicting local keys are rejected. Source effort equivalence requires explicit verified mapping. A missing current AA ID produces an unavailable match, not substitution of another model. A binding can supply scores with effort_unknown when evaluation effort is unreported, but those scores must be labeled and excluded from strict numeric capability comparisons.

Expose compact statuses such as matched, unmapped, missing_record, effort_unknown, and unavailable. Include the match source (bundled or local) and stable AA ID when present. Keep mapping overrides in their own user file so benchmark refreshes never erase them. Reload mappings using the same file-change rules as snapshots. The status command should report unresolved bindings, and an optional exact --model provider/model#variant filter should show the resolved record or why it is unmatched. This lets a user add newly released model bindings without a plugin release.

Compare scores only for the same metric definition, units and compatible evaluated configurations within the same active snapshot. Code computes an oriented numeric delta (advisorMinusCaller, positive means better advisor score) and a comparability status; Jev need not do arithmetic. Avoid arbitrary strength categories or significance thresholds. Differences are descriptive evidence, not a deterministic consult/skip rule.

4. Manual update command

Publish a Bun CLI bin named ocadvisor pointing to dist/cli.js, with #!/usr/bin/env bun and executable permissions verified from the tarball.

ocadvisor benchmarks update
ocadvisor benchmarks status
ocadvisor benchmarks update --path /absolute/path/artificial-analysis.json
ocadvisor benchmarks status --model provider/model#variant

Document a one-shot package-runner invocation as well as the installed bin, and verify its syntax from the packed package. The CLI must import a small CLI/benchmark module graph rather than importing and initializing the plugin or TypeSafe client.

update reads ARTIFICIAL_ANALYSIS_API_KEY from its environment, trims it, and sends it only as the AA x-api-key header. The gate/plugin reader needs no AA key. Fetch the official endpoint with a bounded timeout (proposed 30 seconds), a bounded response size (proposed 16 MiB), and clear handling for 401/403, 429, transport, parse, and validation errors. No retry loop is required for this manual command. Reject redirects rather than forwarding credentials to another origin. Never store/log keys or raw credential-bearing exception details.

Update algorithm:

1. Resolve the target path and acquire an exclusive per-target update lock. Concurrent updater invocations fail promptly with a clear busy result. Use a tested lock implementation; for a PID-bearing filesystem lock, check process liveness and never steal a live lock solely because it is old. Clean up on normal failure and document dead-lock recovery after forced termination.
2. Fetch and normalize one complete response. Validate schema and score semantics before touching the active file.
3. Serialize a complete version-1 snapshot into a uniquely named temporary file in the target directory, flush/close it, and atomically rename it over the target. Clean up temporary files and release the lock in finally. A failed download/parse/validation/write before rename leaves the old file byte-for-byte intact.
4. Print active path, fetchedAt, usable model count, recognized metric coverage, and content hash. Return exit 0 only after successful publication. No-op identical data may still update the download timestamp if a successful fresh fetch occurred; do not rewrite scores or label that time as an evaluation date.

status is offline and reports active source/path, schema version, download time/age, model count, metric coverage, mapping resolution, and fallback/rejection diagnostics. It shares selection/validation with the plugin; if the running plugin retained an older in-memory copy after an invalid replacement, make clear that offline status describes on-disk candidates rather than pretending to inspect process memory. Return nonzero when no usable snapshot exists or a requested exact model cannot be resolved, and document exit codes.

5. Reader, consistency and reloads

Create a reader/cache instance per plugin setup, keyed by resolved snapshot/mapping paths, with an injectable filesystem/runtime seam for tests. Before each gate-enabled consultation, stat the candidate files; reload when their identity changes (device/inode, size, high-resolution modification time where supported). Atomic replacement must be detected even for equal-size content and coarse/equal timestamps. Re-stat around read/validation if needed to avoid caching bytes under the wrong file identity. An unchanged version uses the cached parsed map.

Capture a validated immutable snapshot/mapping view once per consultation, then resolve both participants from that view. File updates during a consultation affect the following call. Rejected versions retain the last good in-memory data; record the rejected signature to avoid reparsing/log flooding on every call, but retry when the file changes. Missing initial local data falls back to the seed. If an active user file disappears, retain the last good in-memory data for that process and report it; a newly started process follows the ordinary disk/seed precedence.

Use async filesystem operations and bound local reads. Runtime benchmark lookup performs no network I/O. Refresh works while OpenCode stays running. Disabled/keyless gates and earlier model exclusions bypass benchmark loading as well as TypeSafe work.

6. Requester/advisor identity and effort

Requester identity should be tied to the outgoing request that produced the tool call, not inferred from the parent transcript or whatever model is selected after a concurrent switch. Confirm the installed OpenCode V2 invocation identifiers and metadata in T2. Prefer a request/tool-invocation model reference when provided; otherwise locate the originating assistant record using the tool call/message identifier in session_message. Only use a context-hook cache if its correlation to the same request is demonstrably sound; a single process-global latest-model value is not acceptable. A session-row fallback is explicitly labeled session_fallback; missing fields remain unknown.

Planning evidence: in the local V2 DB, assistant session_message.data.model stores {id, providerID, variant}. The legacy message attribution query did not find this session. Do not implement the V2 resolver by assuming only the legacy message table exists. Tests must cover subagent identity, sequential and in-flight model switches, unknown variant, and fallback provenance.

Advisor identity comes from resolved plugin configuration and the same catalog-supported variant resolver used for generation. Keep three effort states explicit: caller-pinned effort; gate-selectable candidates plus resolved fallback effort; fixed/default effort when automatic selection is unavailable. Use the actual validated default, not an unsupported configured variant. Preserve existing explicit effort validation errors. Read current live model variants using the installed supported V2 catalog APIs, with narrowly scoped compatibility for the repository's existing catalog.* access when needed.

The need and effort questions run independently in the same TypeSafe request. Therefore there is no final gate-selected advisor effort yet. Represent default/fixed profile plus supported candidates and their benchmark matches; do not present one candidate's score as already selected. Describe that the need question assesses consultation with this advisor under the allowed effort policy; the effort question selects among those candidates. The final selected variant and associated match are recorded after the decision. Do not add a second gate request merely to resolve this ordering.

7. Gate state, budget and instructions

Add structured models.requester, models.advisor, benchmarks provenance, and comparable deltas to both DecisionState and buildStatePayload. Keep the legacy caller label for existing metrics/history consumers if useful. Put identity/coverage fields in the durable header, and use compact bounded score records (roughly 3–4 recognized metrics per model/effort). Reference shared metric definitions and provenance once rather than repeating URLs/labels per score.

Count the final serialized UTF-8 payload, including profiles, in the existing byte budget. Trim verbose benchmark details before losing essential task context, retain model identity and explicit benchmark-omission coverage, then apply existing oldest-message trimming. Preserve the consultation question and latest user intent. Extremely small budgets should use the existing ordinary-advisor fallback rather than silently send invalid/oversize state. Do not pass the full database, raw API response, or arbitrary local mapping notes to Jev. Cover multi-byte strings, many effort candidates, and long model IDs in budget tests.

Update both NEEDED_INSTRUCTIONS and EFFORT_INSTRUCTIONS to reference model profiles and available task-relevant benchmark evidence. Rules: scores estimate comparative capabilities, not certainty on this task; independent review/stuck debugging may help a high-scoring requester; weaker requesters still do not need advice for every trivial action; unknown/mismatched/old data is uncertainty; scores do not override explicit caller effort or justify inventing a model capability. Include source snapshot age but do not choose an arbitrary age-based automatic skip rule.

Add compact metrics: requester model/variant/provenance, advisor model and resolved effort policy, active snapshot hash/fetchedAt/source, mapping coverage, and final advisor variant match. Distinguish these from gate.model, which is the TypeSafe model. Do not log full benchmark responses or duplicate the full gate state in metrics. Update report summaries for benchmark coverage and active data versions while retaining old JSONL compatibility.

Execution policy and critical path

This plan is intentionally sequential. Shared configuration, types, budget code, test fixtures and packaging make serial integration clearer than overlapping edits. Critical path: T1 → T2 → T3 → T4 → T5 → T6 → T7 → T8 → T9 → T10 → T11. There are no scheduled subagents, background tasks, parallel groups, or async joins. Each task's targeted checks must pass before the next wave. If work is later delegated or reordered, revise this plan and the issue first. Preserve issue #6 work through normal branch/file coordination.

Detailed wave checklists

Wave 0 — T1: Planning and tracking

Prerequisites: none. Execution: sequential. Deliverable: agreed design, exact source map, local plan and linked issue assigned to judsd. Files: this plan file only.

• T1.1 Confirm origin, clean baseline, related issues and applicable instructions.
• T1.2 Record the official AA contract and documented metadata gaps without inventing live data.
• T1.3 Publish both records, verify attribution/assignee and bidirectional links, and post a planning checkpoint.

Entry: user requested a full plan. Exit: the two records contain the full contract, schedule and acceptance criteria. T2 is ready once an implementation-capable agent is active. No implementation/test completion is implied by this wave.

Wave 1 — T2: Verify runtime identity and live-source contract

Depends on: T1. Execution: sequential. Files/components: inspect src/ocAdvisor.ts, installed OpenCode V2 tool/context types, fixture message shapes, official AA API; update this plan/issue with confirmed findings.

• T2.1 Recheck git status/default branch and issue Caller-model advisor opt-out — implementation plan #6; create a task-specific branch using approved tooling without discarding release/user work.
• T2.2 Establish the precise requester-model correlation path and session/message schema for the installed V2 runtime. Record the identifiers available to the tool executor and the fallback chain.
• T2.3 With an existing AA environment key, fetch the official response without printing secrets; verify real metric fields, null behavior, exact model/effort coverage and useful payload bounds. Store any temporary evidence only under /tmp/opencode/ during implementation. If the key is unavailable, explicitly retain this as a T11 live-data blocker while progressing with deterministic synthetic fixtures.
• T2.4 Freeze version-1 schema and mapping semantics using the contract above; identify any source mismatch and update both records before dependent edits.

Entry: published plan and Build mode. Exit: request identity strategy is supported by primary-source types/evidence, schema semantics are frozen, and any credential-dependent live verification is explicit. No invented model-ID mappings or scores.

Wave 2 — T3: Shared schema, paths and normalization

Depends on: T2. Execution: sequential. Create: src/benchmarkTypes.ts, src/benchmarkConfig.ts, src/artificialAnalysis.ts, src/benchmarkTypes.test.ts, src/benchmarkConfig.test.ts, src/artificialAnalysis.test.ts. Modify: src/ocAdvisor.ts configuration pipeline and src/ocAdvisor.test.ts; keep explicit runtime fixture configs compiling.

• T3.1 Add tests for complete/partial model results, zero/null/invalid scores, duplicate IDs, unknown additive fields, unsupported versions, deterministic hashing, and preserved metric units.
• T3.2 Add path/precedence tests for XDG defaults, env/plugin/CLI selection, custom mappings path, invalid explicit paths, and omitted options.
• T3.3 Run bun test src/benchmarkTypes.test.ts src/benchmarkConfig.test.ts src/artificialAnalysis.test.ts; observe meaningful failures before implementing normalization/configuration.
• T3.4 Implement pure types, validators, metric definitions, AA-response normalization and shared absolute-path resolution. Export helpers consumed by both plugin and CLI. Add optional benchmarks config without changing existing advisor defaults.
• T3.5 Run the targeted tests plus bun test src/ocAdvisor.test.ts and bun run typecheck.

Entry: T2 contract frozen. Exit: validated versioned data and one shared path/precedence implementation; tests cover failure behavior and additive upstream compatibility.

Wave 3 — T4: Safe refresh transaction

Depends on: T3. Execution: sequential. Create: src/benchmarkUpdate.ts, src/benchmarkUpdate.test.ts.

• T4.1 Add temp-directory tests using injected fetch/clock/filesystem failures: success, missing/blank key, timeout, oversized response, HTTP errors, malformed/empty/duplicate-ID data, failed write, interrupted staging, and concurrent updates.
• T4.2 Assert failed updates preserve the existing file's exact bytes and never replace local model mappings. Assert published files are always complete parseable snapshots.
• T4.3 Run bun test src/benchmarkUpdate.test.ts to observe the absent behavior, then implement the bounded fetch, validated staging, per-target lock, atomic rename and cleanup contract.
• T4.4 Test live-lock refusal and dead-lock recovery, and ensure staged files are created in the destination directory for same-filesystem rename.
• T4.5 Re-run bun test src/benchmarkUpdate.test.ts and typecheck. Include sanitized error codes/messages that the CLI can present directly.

Entry: T3 passes. Exit: transactional refresh succeeds deterministically and failures preserve valid active data; the updater has no plugin/TypeSafe dependency.

Wave 4 — T5: Exact identity mapping and comparisons

Depends on: T4. Execution: sequential. Create: src/benchmarkMatch.ts, src/benchmarkMatch.test.ts, src/data/artificialAnalysis.mappings.json.

• T5.1 Add fixtures for direct and gateway model IDs, explicit local overrides, unknown model/version, missing stable AA record, unset effort versus named effort, equivalent verified efforts, conflicting bindings, null scores and multiple advisor candidates.
• T5.2 Write tests that compare only compatible metric units/configurations in one snapshot, retain source scores, and calculate oriented deltas without strength thresholds.
• T5.3 Run bun test src/benchmarkMatch.test.ts; implement exact tuple indexing, validated local overrides, statuses and comparisons.
• T5.4 Populate only verified real bundled mappings from T2 evidence; keep unverified coverage visibly unmapped. Every curated equivalence carries a source link.
• T5.5 Verify local bindings can cover a new model with no source-code change, and data refresh does not change those bindings.

Entry: T4 passes. Exit: deterministic mapping/coverage and comparison tests pass; configured caller/advisor coverage is known or explicitly missing.

Wave 5 — T6: Snapshot cache, fallback and live reload

Depends on: T5. Execution: sequential. Create: src/benchmarkStore.ts, src/benchmarkStore.test.ts, src/data/artificialAnalysis.snapshot.json (genuine seed when the API key is available).

• T6.1 Add tests for user-over-seed precedence, unchanged-file cache hits, equal-size/equal-mtime atomic replacement, mapping-only changes, per-instance isolation, replacement during a read, and a consistent immutable view during one consultation.
• T6.2 Cover initial absent/corrupt/unsupported-version files, invalid replacements after successful load, file deletion, reappearance, and recovery to a subsequent valid file.
• T6.3 Run bun test src/benchmarkStore.test.ts; implement the reader/cache and bounded change detection with last-good retention and explicit diagnostics.
• T6.4 Generate a real normalized seed through the same normalization/validation path; do not use documentation example scores or synthetic tests as the shipped database. If live access is blocked, retain the seed/live acceptance gate as blocked and do not claim release readiness.
• T6.5 Assert benchmark reader calls cannot perform network I/O and that a changed snapshot is observed on the next load with no process restart.

Entry: T5 passes. Exit: reader/cache tests pass, genuine seed provenance is recorded or the remaining credential blocker is explicit, and hot reload works independently of package version.

Wave 6 — T7: Requester and advisor model profiles

Depends on: T6. Execution: sequential. Create: src/modelProfiles.ts, src/modelProfiles.test.ts. Modify: src/ocAdvisor.ts, src/ocAdvisor.test.ts, src/advisorGate.test.ts.

• T7.1 Add requester tests for the exact originating turn, in-flight model switch, sequential switch, parent/child differences, gateway IDs, unknown variant and labeled session fallback.
• T7.2 Add advisor tests for explicit valid/invalid effort, configured supported/unsupported variant, unknown catalog, fixed effort and all allowed candidate efforts.
• T7.3 Implement the verified T2 correlation path; carry provider/model/variant and provenance through runAdvisor without trusting tool-input self-reports. Preserve complete nested model IDs.
• T7.4 Resolve advisor profiles using the generation variant validation rules. Narrowly support the live V2 model-list API if existing catalog discovery needs adaptation.
• T7.5 Run bun test src/modelProfiles.test.ts src/ocAdvisor.test.ts src/advisorGate.test.ts and typecheck.

Entry: T6 passes. Exit: model profiles reflect the actual requester and valid advisor effort policy; isolation/switch tests pass and existing model-exclusion/Fable guards retain precedence.

Wave 7 — T8: Gate state, instructions and metrics integration

Depends on: T7. Execution: sequential. Modify: src/typesafeState.ts, src/typesafeGate.ts, src/ocAdvisor.ts, src/typesafeState.test.ts, src/typesafeGate.test.ts, src/advisorGate.test.ts, src/usageReport.ts. Create if needed: src/benchmarkReport.test.ts for genuine report behavior.

• T8.1 Add wire-payload assertions at the injected GateClient boundary: requester/advisor profiles, all relevant candidate efforts, sourced scores, comparable deltas, missing coverage and one shared snapshot hash actually reach TypeSafe.
• T8.2 Add budget tests with multi-byte text, many efforts/metrics, long IDs and tiny budgets; assert essential task intent/model identities and explicit omission coverage survive, or the ordinary fallback is selected.
• T8.3 Wire one immutable benchmark view into state creation after early guards and before the gate. Extend the final payload projection as well as the state builder. Keep benchmark enrichment optional on failure.
• T8.4 Update need/effort question wording according to the contract. Retain explicit caller effort precedence and one independent TypeSafe request.
• T8.5 Add compact version/match/provenance metrics, resolve the final advisor-effort match after selection, and summarize benchmark coverage in the usage report without breaking old records.
• T8.6 Test keyless/disabled/early-excluded calls do no benchmark loading; malformed/unavailable benchmarks preserve consultation; TypeSafe failures retain existing fallback behavior; refresh affects the next invocation in the same plugin instance.
• T8.7 Run bun test src/typesafeState.test.ts src/typesafeGate.test.ts src/advisorGate.test.ts src/ocAdvisor.test.ts plus any new report test and bun run typecheck.

Entry: T7 passes. Exit: actual serialized gate input includes bounded profiles/evidence, snapshot reload is observable end-to-end, and effort/bypass/fallback/metrics compatibility checks pass.

Wave 8 — T9: Packaged refresh/status CLI

Depends on: T8. Execution: sequential. Create: src/cli.ts, src/cli.test.ts, src/benchmarkPackage.test.ts, a small scripts/copyBenchmarkData.ts only if the compiler cannot emit the data assets. Modify: package.json, tsconfig.build.json, src/entrypoint.test.ts, and .github/workflows/ci.yml only as necessary for the packaged CLI/data smoke gate.

• T9.1 Add CLI tests for help/unknown command/flags, update success/failure exit codes, status without credentials/network, snapshot/custom-path selection, --model coverage and --mappings-path.
• T9.2 Implement benchmarks update and benchmarks status as thin adapters over shared modules; print concise stable summaries and documented errors.
• T9.3 Add bin.ocadvisor = ./dist/cli.js, the Bun shebang, executable handling and explicit build/data asset inclusion. Leave package.json version at the last published release per repository policy.
• T9.4 Build and pack into an isolated directory under /tmp/opencode/. Verify the tarball contains the executable CLI plus baseline snapshot/mappings, and excludes tests/eval fixtures/live user snapshots/credentials. Test from extracted or clean-installed package paths, not the source checkout.
• T9.5 Smoke-test the packaged plugin import, CLI --help, offline status using the bundled seed, and a fixture-backed refresh followed by a same-process plugin reload. Verify an actually runnable one-shot invocation (for example bunx --package @pfoundation/ocadvisor ocadvisor benchmarks update) before documenting it.
• T9.6 Run bun test src/cli.test.ts src/benchmarkPackage.test.ts src/entrypoint.test.ts, typecheck and build. Keep the existing npm tarball allowlist valid.

Entry: T8 passes. Exit: the published artifact can execute refresh/status and load the seed with no source-checkout dependency; packaging tests verify paths, permissions and runtime behavior.

Wave 9 — T10: Labeled evaluation and user documentation

Depends on: T9. Execution: sequential. Modify: src/typesafeGate.eval.ts, src/fixtures/typesafeGate.cases.json, README.md.

• T10.1 Extend the eval harness to accept structured model/benchmark profiles and record snapshot provenance. Synthetic benchmark fixtures must be clearly labeled as synthetic, isolated from real shipped data, and reproducible.
• T10.2 Add labeled cases: stronger requester needing independent review; weaker requester doing a trivial lookup; unresolved debugging; unknown scores; effort mismatch; different available advisor efforts; same task with comparable capability advantage; stale snapshot; and missing identity. Retain existing must-consult cases.
• T10.3 Run the existing and enriched cases with the same pinned TypeSafe model/settings when a TYPESAFE_API_KEY is available. Compare false skips, unnecessary proceeds, fallbacks, payload bytes/tokens and latency. Do not assert success from a mock gate's hard-coded decisions or tune a new threshold to a single stochastic run. Investigate every new false skip in a must-consult case.
• T10.4 Document AA source/attribution, account API-key setup, metric meanings and metadata gaps, default/XDG/custom paths, mapping overrides, update/status/one-shot commands, data-only hot reload, fallback behavior, supported schema versions, errors and recovery. Distinguish the one-time plugin-code deployment restart from subsequent data-only refreshes.
• T10.5 Include an end-to-end example: initial installation uses seed; manual refresh publishes user data; next consultation logs the new hash; later package upgrade preserves that file. Document unsupported/unmapped model behavior explicitly.

Entry: T9 passes. Exit: deterministic eval fixtures/docs are complete, commands match packaged behavior, and live eval evidence is recorded or explicitly blocked by credentials.

Wave 10 — T11: Final verification and implementation handoff

Depends on: T10. Execution: sequential. Files/components: all changed code, tests, README, npm tarball, this plan and tracking issue.

• T11.1 Run bun test, bun run typecheck, bun run build, npm pack --dry-run --json, and git diff --check. Run the clean packed-install smoke test from T9 and verify the CLI binary permissions and data asset resolution.
• T11.2 Perform one authenticated AA refresh to an isolated custom path and verify the real snapshot/coverage. Run the pinned live gate evaluation against it where exact matches exist. Never print API keys. If required credentials are absent, report the specific remaining live verification and do not check off full release readiness.
• T11.3 Confirm in a single long-running test runtime that snapshot A supplies both profiles, a CLI refresh publishes B, and the next consultation uses B while a consultation already using A remains internally consistent. Confirm local mappings and user data survive a simulated package upgrade.
• T11.4 Review the diff against this contract, resolve issue Caller-model advisor opt-out — implementation plan #6 overlap, and commit through GitHub Helper with Refs: pfoundation/ocAdvisorTool#<tracking-issue>. Verify judagent[bot] author/committer identity.
• T11.5 Update plan/issue checkboxes and verification results, link commits/PR, state any missing model coverage or credential blocker, and mark implementation complete/awaiting merge only when acceptance gates pass. PR description uses Closes pfoundation/ocAdvisorTool#<tracking-issue> and the Models line. Keep the issue open until merged to the default branch.

Entry: T10 complete and known verification blockers resolved or explicitly reported. Exit: all deterministic and required live checks pass, published artifact behavior is verified, and the work is ready for merge with reproducible evidence.

Verification matrix

Models used — full planning attribution

Models: openai/gpt-6-astra:xhigh, anthropic/claude-fable-5-1:xhigh

The completed advisor consultation contributed to the initial design before the user requested no further advisor calls. The later interrupted consultation supplied no additional advice. The primary session's model/effort was confirmed from V2 assistant session_message.data.model. Reuse this full contributor line in planning checkpoints; append actual implementation models when implementation begins.

Scope

• Accurate requester/advisor model profiles and effort policy in the existing TypeSafe gate.
• Versioned Artificial Analysis JSON database, real bundled baseline, exact stable-ID mappings, and user overrides.
• Fast in-memory lookup with next-consultation disk-change detection, consistent per-call views, and last-good fallback.
• Packaged ocadvisor benchmarks update and offline ocadvisor benchmarks status commands using shared paths and validation.
• Compact task-relevant metric evidence/comparisons, source attribution, coverage metrics, labeled evaluation, and user documentation.
• Sequential implementation with explicit coordination for the separate caller-model opt-out issue Caller-model advisor opt-out — implementation plan #6.

Execution schedule

Wave 0

• T1 Publish agreed contract and issue-backed plan

Wave 1

• T2 Verify V2 requester identity and AA source contract

Wave 2

• T3 Implement schema, normalization and shared paths

Wave 3

• T4 Implement transactional AA snapshot refresh

Wave 4

• T5 Implement exact model mappings and comparisons

Wave 5

• T6 Implement local reader, reloads and genuine seed

Wave 6

• T7 Resolve requester and advisor effort profiles

Wave 7

• T8 Integrate profiles into gate payload and metrics

Wave 8

• T9 Ship refresh/status CLI and data assets

Wave 9

• T10 Evaluate benchmark-aware decisions and document usage

Wave 10

• T11 Complete integration checks and merge handoff

Acceptance

• The actual TypeSafe wire payload identifies requester and advisor provider/model/effort policy and includes matched AA benchmark evidence with explicit coverage and provenance.
• Requester identity is correlated to the calling turn, handles subagents and model switches, and clearly labels fallback/unknown information.
• Published benchmark names, units, model/version/effort distinctions and unknown metadata are preserved; no invented scores, benchmark index or evaluated-at dates.
• AA data is read from a bounded local snapshot/cache on the consultation path; ordinary benchmark lookups perform no network request.
• A published ocadvisor benchmarks update command refreshes user data independently of npm releases, preserves existing valid data on failure, and coordinates simultaneous updaters.
• A published offline ocadvisor benchmarks status command shows source/path/schema/age/coverage and exact-model mapping diagnostics.
• The next consultation in an already-running plugin observes a valid CLI refresh; a call in progress uses one internally consistent snapshot for both models.
• User snapshots and mapping overrides survive plugin upgrades; exact new model mappings can be supplied locally between releases.
• Missing/corrupt/unsupported/unknown benchmark evidence degrades to last-good/seed/unavailable state without preventing ordinary advisor consultation.
• Explicit caller effort, supported effort constraints, early exclusions and existing disabled/keyless/failure behavior remain correct.
• The serialized state respects UTF-8 byte limits and preserves critical task/model context with explicit omission coverage.
• Tarball contains a runnable Bun CLI and genuine baseline assets; plugin and CLI work in a clean package install.
• All required deterministic checks and authenticated refresh/live evaluation are evidenced; any missing key/model coverage remains explicitly reported rather than claimed complete.
• Plan/issue progress uses stable task IDs; commits use bot identity and Refs; issue remains open until work reaches the default branch.

Models used

Models: openai/gpt-6-astra:xhigh