Skip to content

feat(robotics): add paired recovery campaign scorer - #2862

Draft
Tristan-Stoltz-ERC wants to merge 2 commits into
research/paired-robotics-recovery-benchmark-v0-1from
research/paired-robotics-recovery-scorer-v0-1
Draft

Tristan-Stoltz-ERC wants to merge 2 commits into
research/paired-robotics-recovery-benchmark-v0-1from
research/paired-robotics-recovery-scorer-v0-1

Conversation

@Tristan-Stoltz-ERC

@Tristan-Stoltz-ERC Tristan-Stoltz-ERC commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Stacked on #2860. Adds a Rust-first, dependency-light scorer for the preregistered paired recovery campaign.

Implementation

  • Adds recovery_benchmark.rs and exports its public types from the robotics bridge core.
  • Serializes/deserializes arm IDs, expected scenarios, trial records, error categories, and summaries so a future runner can exchange structured evidence records without adding a new dependency.
  • Requires an explicit expected-scenario manifest so omitted whole scenarios do not vanish from the denominator.
  • Fails closed on empty/invalid scenario identities, missing or unexpected scenarios, missing/duplicate arm records, scenario/config digest mismatches, and incomplete/invalid recovery fields.
  • Enforces four matched arms per scenario: native-controller reference, Symthaea intent, guarded adapter, and guarded outcome feedback.
  • Verifies plant/native-controller/profile identity across paired arms so a claimed policy improvement cannot silently be explained by changing the comparison platform.
  • Reports per-arm task/recovery rates, false-veto numerator/denominator, p50/p95/p99/max available safe-state latency samples, deadline misses, and hard-boundary/lifecycle violations.
  • Reports paired B3-vs-B2 recovery wins/losses/ties and point-estimate percentage-point delta. Hard-boundary violations remain independent: a positive recovery score never clears a failed hard gate.
  • Adds nine focused Rust unit tests for paired metrics, missing/duplicate records, configuration mismatch, safety violations, missing/unexpected scenarios, false-veto and latency validation, deterministic percentiles, and malformed metadata.
  • Inherits the CI branch allowlist from the current benchmark base and is rebased to that exact base.

Claim ceiling

This is a deterministic scoring core, not a simulation rollout runner, statistical-inference engine, durable event ledger, physical controller, or safety certification. It computes point estimates only; paired confidence intervals must be computed in the separate analysis step. No results or SOTA gains are claimed.

Validation

Related: #2828, #2836, #2847, #2852, #2860.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant