[skycap] Index each step's skycap records in W&B

[kouroshHakha]

What does this PR do?

TLDR: each training step's skycap records are indexed in W&B: one artifact version per step listing which trajectories made up the step, and how each one ended up. Mirrored records (#2350) appear as W&B references, so no record bytes are uploaded. It replaces #2333's upload path, and its design (config shape, per-phase log, eval opt-in) comes from that PR. Stacked on #2350.

How it works

• SkycapRecordIndex (examples/train_integrations/harbor_skycap/record_index.py), a TrainingCallback.
  • At on_step_end (and on_eval_end when enabled) it logs one version of skycap-records-<phase>-<run id>, aliased <phase>-step-N and latest, through the trainer's Tracking.
  • The version holds:
    • step.json: the step's run index. One row per trajectory attempt, with instance_id, repetition_id, attempt, status, annotations, superseded, trained, and finish's record {host, path, mirror, files}. The format is in run_index.md.
    • records/<file>: one reference per file of each mirrored record. A local-only record is listed in the index only.
• RecordLog: the generator logs every attempt it opened, per phase, retries included.
  • trained comes from the step's batch: whether the final attempt had trainable tokens.
  • Train entries still in the log at on_step_start came from a batch the trainer dropped unfinished. They're discarded, not indexed with the next step.
• Failure behaviour. Building the index runs on the training thread, so a bug in that code fails the step. Everything that talks to W&B runs on one background worker and fails open:
  • a timeout per version, after which the version is abandoned, not retried;
  • bounded retries for transient errors, only before W&B accepts the version;
  • a bounded queue that drops when full;
  • a deadline at on_train_end;
  • counters in stats().
• Config (skycap.* in the entrypoint):
  • wandb.enabled is on by default when trainer.logger=wandb; wandb.phases is [train] by default;
  • record_mirror and record_mirror_config are passed to every server.
• Core change: trainer.py passes trajectory_ids in CallbackInput at on_step_end (the batch's rows carry their loss masks), so the index knows which trajectories trained.

Test plan

• tests/integrations/harbor_skycap/test_record_index.py (fake W&B, no network):
  • one version per step however many generate() calls the step took;
  • superseded and untrained attempts are marked;
  • mirrored records become references, one per file;
  • eval goes to its own artifact only when enabled;
  • a dropped partial batch isn't indexed with the next step;
  • W&B that fails, is slow, or already accepted a version fails open without duplicates;
  • a full queue drops, and shutdown respects its deadline;
  • a bug in the callback raises into the step.
• pytest tests/integrations/harbor_skycap: 56 passed. pre-commit is clean.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a new SkycapRecordIndex callback to track and index training trajectories in Weights & Biases. It adds support for mirroring records to remote storage, updates the HarborSkycapGenerator to log trajectory metadata, and modifies the RayPPOTrainer to pass trajectory IDs to callbacks. The review feedback identified a potential thread leak in the worker loop and suggested safer dictionary access for record metadata, both of which are actionable improvements.

[greptile-apps]

Confidence Score: 4/5

[Medium risk] Adds W&B logging for skycap training records.

The PR should not merge until discarded dynamic-sampling batches are excluded from later step indexes.

Findings

1.  Discarded trajectories enter later indexes ▶

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  G[Skycap generator] -->|attempts and record locations| L[RecordLog by phase]
  T[Trainer step or eval callback] -->|take phase entries| L
  T -->|batch trajectory IDs and loss mask| I[Build step index]
  L --> I
  I --> Q[Background logging queue]
  Q --> W[W&B step.json and mirror references]

Loading

_{Reviews (1) · Last reviewed commit: "[skycap] Index each step's skycap record..."}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6e73183. Configure here.}