ruby: Recover retried worker history

[dpashk-figma]

Describe

• Preserve coverage-producing work when a distributed Minitest worker is retried.
• Add an opt-in worker-history retry mode that expands suite chunks, replays the worker’s durable reservation history locally, resumes the shared Redis queue, and writes recovery evidence only after a complete run.
• Keep the existing failure-only retry behavior as the default and fail closed when reservation or chunk metadata is missing.

Implementation design

The existing Redis worker log records every reservation made by a worker, but normal retry behavior intersects that log with the build’s failed-test set. That is appropriate for test failure recovery, but not for process-local artifacts such as SimpleCov output: tests that passed before a worker was terminated disappear from the retried process’s coverage data and are not selected by a failure-only retry.

This change introduces an opt-in recovery path:

--retry-mode worker-history
--recovery-manifest <path>

--retry-mode defaults to failures, so existing callers retain their current behavior.

flowchart LR
  A["Buildkite retries worker N"] --> B["Read worker N reservation history"]
  B --> C["Expand suite chunks and deduplicate test IDs"]
  C --> D["Replay tests locally without Redis mutations"]
  D -->|"Replay completes"| E["Resume the shared Redis queue"]
  E --> F["Write recovery manifest after the run completes"]
  D -->|"Interrupted or incomplete"| G["Fail without a recovery manifest"]

Loading

Durable history reconstruction

• Each Redis worker already has a build-scoped reservation list keyed by its stable build ID and worker ID.
• On a Buildkite retry, Worker#worker_history reads that list and restores chronological reservation order.
• Suite-bin-packing reservations are chunk IDs rather than individual test IDs. Recovery loads each chunk’s existing Redis metadata and expands it to the original ordered test IDs.
• Test IDs are deduplicated while preserving their first reservation order. This handles requeues and repeated reservations without running the same test repeatedly during the replay phase.
• Missing reservation history, missing chunk metadata, invalid chunk JSON, or an empty chunk causes recovery to fail closed.

Two-phase recovery queue

WorkerHistoryRecovery wraps the original Redis worker and presents the same queue interface to Minitest:

1. During the replay phase it yields the reconstructed test IDs from a local list. Acknowledgements and requeues do not mutate Redis, because this work exists only to reconstruct process-local outputs.
2. It still honors shutdown, circuit-breaker, and maximum-failure state. If replay stops early, the queue never transitions to the shared phase.
3. After every historical test has been replayed, the wrapper switches to the original Redis worker and resumes normal shared-queue polling in the same Ruby process.

Keeping both phases in one process is important for coverage: SimpleCov accumulates the replayed history and any newly claimed shared work into one result set. It also supports repeated Buildkite retries. The worker’s durable history is retained, and reservations made after the shared queue resumes are appended for the next retry.

Recovery evidence

When --recovery-manifest is supplied, a Minitest reporter writes JSON only after the queue is exhausted and any required replay completed. A terminated worker or incomplete replay leaves no manifest.

The manifest records:

• schema version, worker ID, and Buildkite retry count;
• raw reservation count and deduplicated replayed-test count;
• whether replay completed and whether the worker resumed the shared queue.

A successful first attempt records zero replay counts, replay_completed: true, and resumed_shared_queue: false. A successful worker-history retry records positive history/replay counts and both completion booleans as true. Downstream artifact producers can therefore reject incomplete or failure-only retries instead of silently publishing partial data.

Compatibility and operating assumptions

• Failure-only retry remains the default and is unchanged.
• Worker-history recovery requires a distributed Redis queue.
• The retry must retain the original worker ID and queue build ID; Buildkite parallel retries satisfy this contract.
• The existing release command may expire the worker’s active lease before retry, but it must retain the worker reservation list and chunk metadata until their normal Redis TTL expires.

Fixes: https://app.asana.com/1/10497086658021/project/1217018759338815/task/1217018759338827

Test Plan

• Run focused configuration and Redis worker/chunk tests covering ordered expansion, stable deduplication, and missing metadata.
• Run recovery queue tests covering local replay, no-op replay acknowledgements/requeues, shared-queue resume, failure cutoffs, and missing test IDs.
• Run reporter tests verifying incomplete runs do not write recovery evidence.
• Run the Redis subprocess integration test verifying a clean first attempt, a worker-history retry, positive replay counts, and fail-closed behavior for a missing worker history.
• Build the gem with bundle exec rake build.
• Temporarily pin this branch in figma/figma and force the scheduled Sinatra coverage build; cancel and retry a worker, then verify its recovery manifest and the complete collated output.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit cb156e1. Configure here.}