[skycap] Land the W&B record index and pull on main (#2351, #2436); pull writes a directory per phase

[kouroshHakha]

What does this PR do?

TLDR: lands #2351 (the per-step W&B record index) and #2436 (record_index pull) on main. Both were reviewed and merged, but into their stacked base branches after #2350 had already landed, so neither reached main. On top of that, pull writes each phase as a record directory of its own.

Commits

1. The merged content of [skycap] Index each step's skycap records in W&B #2351 and [skycap] Pull a run's records and step index from W&B #2436, unchanged (byte-identical to skycap/wandb-index at its [skycap] Pull a run's records and step index from W&B #2436 merge):

   • SkycapRecordIndex logs one W&B artifact version per step, skycap-records-<phase>-<run id>, aliased <phase>-step-N, holding step.json (the run index, run_index.md) and a reference per mirrored record file. It fails open on a background worker.
   • record_index pull brings a run's records, read from the mirror, and its index into a local directory.
   • trainer.py passes trajectory_ids to on_step_end.

2. New: pull writes <out_dir>/train/ and <out_dir>/eval/. Each holds that phase's records, plus its step indexes as index/step-<N>.json. Each is a plain skycap record directory, so a reader of records (e.g. skycap-viewer) shows a phase with nothing W&B- or index-specific.

3. Review fixes:

   • pull refuses a mirror on this machine's own filesystem unless --allow-local-mirror is passed, because the mirror's location comes from the artifact.
   • A partial download is never moved in.
   • A record goes in whole or not at all, and anything it replaced is put back if it fails.
   • At most 2 abandoned W&B calls may still be running; past that, new calls are refused.

Test plan

• pytest tests/integrations/harbor_skycap tests/train/test_rl_callbacks.py: 72 passed. The pull tests check the per-phase layout, train and eval included.
• Real run (Qwen3-4B, 1 step, mirror on S3, W&B artifact skycap-records-train-4gyajb5w): pull wrote 8 records (16 files) and train/index/step-1.json to <out>/train/. skycap-viewer from main, with no index support, reads <out>/train as a normal run: 8 trajectories, all calls bridged.

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches the RL training loop (on_step_end + new callback) and pull reads mirror URIs from artifacts (mitigated by protocol allowlist and local-mirror opt-in). W&B indexing is isolated on a fail-open background thread so training steps should not block on logging failures.

Overview
Adds per-step W&B indexing of skycap trajectory records for Harbor training: a RecordLog collects every attempt (including retries) per phase; SkycapRecordIndex publishes one artifact version per step (skycap-records-<phase>-<run id>) with step.json (run index per run_index.md) and URI references to mirrored record files—no record bytes uploaded. Logging runs on a background worker that fails open (timeouts, retries, queue drops, shutdown wait).

Wires this through main_harbor_skycap: new skycap.record_mirror / record_mirror_config / skycap.wandb config, generator hooks in harbor_generator, and trainer callback registration. RayPPOTrainer now passes trajectory_ids on on_step_end so the index can mark which trajectories actually trained (via loss_mask).

Adds record_index pull to reconstruct local record trees from W&B: out_dir/train/ and out_dir/eval/ each hold that phase’s *.zst files (fetched from the mirror via fsspec, not W&B blob download) plus index/step-<N>.json. Pull is defensive: mirror protocol allowlist, optional --allow-local-mirror, skip partial records, atomic moves with rollback, plain filename checks.

Docs/README updated; integration tests cover index logging and pull behavior; notes W&B index is not available under the fully-async trainer.

^{Reviewed by Cursor Bugbot for commit cf86829. Bugbot is set up for automated code reviews on this repo. Configure here.}

[gemini-code-assist]

Code Review

This pull request introduces a per-step Weights & Biases (W&B) index for skycap records in the Harbor-Skycap integration, allowing trajectory records to be indexed in W&B artifacts and pulled back locally. It adds the SkycapRecordIndex trainer callback, a CLI tool for pulling runs, comprehensive documentation, and updates the RL trainer to pass trajectory IDs to step-end callbacks. Feedback on the changes suggests adding defensive checks to prevent potential TypeError exceptions: specifically, when converting result.record.files to a list in record_index.py and when casting cfg.skycap.record_mirror_config to a dictionary in main_harbor_skycap.py.

[greptile-apps]

Confidence Score: 4/5

[Medium risk] Adds W&B record indexing and pull for skycap training runs.

Fix failed-download acceptance and the required configuration docstrings before merging.

Findings

1.  Failed downloads become complete records ▶
2.   Artifacts can copy private files ▶
3.  Timed-out calls keep running ▶
4.  Skipped records still change files ▶
5.  Configuration summaries end mid-sentence ▶

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  G["Generator opens and finishes attempts"] --> L["RecordLog"]
  T["Trainer ends a step"] --> C["SkycapRecordIndex"]
  L --> C
  C --> Q["Background queue"]
  Q --> W["W&B artifact: step.json and references"]
  W --> P["pull reads step.json"]
  M["Record mirror"] --> P
  P --> O["train/ or eval/: records and index/"]

Loading

_{Reviews (1) · Last reviewed commit: "[skycap] Pull writes each phase as a rec..." · Reviewed by Greptile}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0f1b861. Configure here.}