Skip to content

research(robotics): preregister paired fault-recovery benchmark - #2860

Draft
Tristan-Stoltz-ERC wants to merge 2 commits into
research/outcome-lifecycle-ledger-v0-1from
research/paired-robotics-recovery-benchmark-v0-1
Draft

Tristan-Stoltz-ERC wants to merge 2 commits into
research/outcome-lifecycle-ledger-v0-1from
research/paired-robotics-recovery-benchmark-v0-1

Conversation

@Tristan-Stoltz-ERC

@Tristan-Stoltz-ERC Tristan-Stoltz-ERC commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Stacked on #2852. Adds a preregistered, simulation-first experiment to measure whether validated controller-outcome feedback improves recovery without weakening the controller authority boundary.

Added

  • docs/research/ROBOTICS_PAIRED_RECOVERY_BENCHMARK_V0_1.md: four-arm design (native-controller reference, Symthaea intent, guarded adapter, guarded feedback loop); six fault families; non-compensatory lifecycle/dispatch gates; paired statistics; evidence tiers and claim limits.
  • docs/research/robotics-paired-recovery-scenarios-v0.1.json: frozen 30 nominal seed cases and 6 × 3 × 30 = 540 fault cases, for 570 paired scenario identities and 2,280 arm-episode records across four arms. Numeric values/timing that depend on the controller profile must be frozen before scored runs.
  • CI trigger allowlist expanded for the stacked outcome-ledger, paired benchmark, and scorer branches.
  • Rebased onto the latest outcome-ledger head, so the PR stack no longer diverges from its base.

Research hypothesis (not a result)

On the same scenario/seed, B3 should improve fault-recovery success against B2 while keeping nominal success within a preregistered 1 percentage-point non-inferiority margin. No weighted reward can compensate for a hard boundary violation. A proposed +10 percentage-point recovery improvement is a target, not a result or industry standard.

Claim ceiling

This PR defines the experiment; it does not implement a rollout runner, restart-safe persistence, a physical robot adapter, or a safety certification. The current in-memory ledger does not by itself provide durable restart fencing. Any missing, crashed, timed-out, or unresolved episode must be represented, not discarded.

Validation

Related: #2828, #2836, #2847, #2852.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant