[TomCC7]

Convert a raw recording file into an aligned, validated dataset. Here is the input-to-output artifact structure for a LeRobot export with an image feature:

Before: raw recording input

recordings/session.db                 # native-rate SQLite streams

This PR also accepts a single session.mcap source file.

After: prepared LeRobot dataset

datasets/session/
├── data/chunk-000/file-000.parquet    # aligned observation/action frames
├── videos/observation.images.image/
│   └── chunk-000/file-000.mp4         # camera frames, when configured
├── meta/
│   ├── info.json                     # features, frame rate and totals
│   ├── tasks.parquet                 # task labels
│   ├── stats.json                    # feature statistics
│   └── episodes/chunk-000/file-000.parquet  # episode ranges and statistics
└── dimos_meta.json                   # source, feature/alignment config and quality reports

The source recording stays unchanged. HDF5 export instead produces session.hdf5 plus session.dimos_meta.json.

These export layouts already exist on the base branch; this PR adds explicit feature validation, causal accepted-target alignment, MCAP input and quality/fill provenance. The recording-directory layout (schema.json plus recording.mcap or recording.db) belongs to #4396, not this PR.

What is this feature?

Define feature dtype/shape/names/source semantics and share an alignment plan between inspection and export. Reconstruct sparse accepted joint targets causally, including history before episode start, and support MCAP source timestamps. Write task labels and fill provenance to HDF5.

Why do we need this?

Offline datasets must preserve configured joint order and accepted target history across SQLite and MCAP. Depends on the episode branch. Existing fill semantics are retained; this PR introduces no new fill-age limit.

Documentation

Read the offline alignment guide for the API contract and executable examples. The guide uses APIs available in this PR, without relying on later stack layers.

How to Test

uv run pytest dimos/imitation/dataprep/test_core.py dimos/imitation/dataprep/test_mcap_source.py dimos/imitation/dataprep/formats/test_hdf5.py

67 focused core, MCAP and HDF5 tests passed; commit and lockfile hooks passed. LeRobot dispatch stays on the existing path until the separate exporter PR.

Which issue(s) does this PR close?

None. This is a review slice of #4343, following the Manipulation SDK preview. It does not close or replace the integration PR.

Checklist

• I have read and approved the CLA.

AI assistance: Codex assisted with the split, validation, and description.

Review corrections and validation

SQLite inspection no longer requires the optional MCAP package. Unlabeled explicit ranges retain a configurable default task for HDF5. Standalone LeRobot export at this layer preserves boolean fill flags; the next layer uses the isolated native writer. Alignment retains numeric history and timestamps while decoding video as frames are selected, instead of retaining all decoded images. The HDF5 writer still buffers an episode, as documented in the offline guide.

102 focused tests passed at this layer, including four regressions that failed before the fixes. Four production modules passed mypy; normal hooks and the executable guide passed. The final stack passed 363 focused tests and seven production modules' typing. These are focused local results, not full CI. Exact-head CI remains under verification; benchmark investigation is paused and no gate has been waived.

[github-actions]

go2 replay realtime

Details

Benchmark suite	Current: bb7ad9f	Previous: 7529659	Ratio
first frame wall	10.729 s	11.519 s	0.93
first frame cpu	21.835 s	23.908 s	0.91
run cpu	31.056 s	33.323 s	0.93
run cpu (user)	23.412 s	25.718 s	0.91
run cpu (system)	7.644 s	7.604 s	1.01
peak memory	2719.195 MB	2705.062 MB	1.01
peak threads	409 threads	406 threads	1.01
disk read	27.812 MB	35.621 MB	0.78
disk write	11.223 MB	11.242 MB	1.00
network (transport)	2714.779 MB	2728.796 MB	0.99
network (external rx)	0.093 MB	0.059 MB	1.58
network (external tx)	0.238 MB	0.147 MB	1.62

This comment was automatically generated by workflow using github-action-benchmark.

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'go2 replay realtime'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.08.

Benchmark suite	Current: bb7ad9f	Previous: 7529659	Ratio
network (external rx)	0.093 MB	0.059 MB	1.58
network (external tx)	0.238 MB	0.147 MB	1.62

This comment was automatically generated by workflow using github-action-benchmark.

[codecov]

Codecov Report

❌ Patch coverage is 96.03175% with 35 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/imitation/dataprep/core.py	91.76%	11 Missing and 9 partials ⚠️
dimos/imitation/dataprep/build.py	87.34%	7 Missing and 3 partials ⚠️
dimos/imitation/dataprep/formats/hdf5/writer.py	71.42%	2 Missing and 2 partials ⚠️
dimos/imitation/dataprep/cli.py	80.00%	1 Missing ⚠️

@@            Coverage Diff             @@
##             main    #4394      +/-   ##
==========================================
+ Coverage   80.07%   80.30%   +0.22%     
==========================================
  Files        1615     1619       +4     
  Lines      151866   152546     +680     
  Branches    12866    12930      +64     
==========================================
+ Hits       121603   122498     +895     
+ Misses      26709    26471     -238     
- Partials     3554     3577      +23     

Components	Coverage Δ
Tests	95.23% <100.00%> (+0.14%)	⬆️

Flag	Coverage Δ
OS-macos-latest	74.87% <94.21%> (+0.22%)	⬆️
OS-ubuntu-24.04-arm	74.52% <94.21%> (+0.22%)	⬆️
OS-ubuntu-latest	74.93% <94.21%> (+0.22%)	⬆️
Py-3.10	74.93% <94.21%> (+0.22%)	⬆️
Py-3.11	74.92% <94.21%> (+0.22%)	⬆️
Py-3.12	74.93% <94.21%> (+0.25%)	⬆️
SelfHosted-Large	32.01% <26.87%> (+0.13%)	⬆️
SelfHosted-Linux	39.58% <47.27%> (-0.01%)	⬇️
SelfHosted-macOS	39.92% <47.27%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/imitation/dataprep/formats/lerobot/writer.py	91.12% <100.00%> (+0.57%)	⬆️
dimos/imitation/dataprep/formats/test_hdf5.py	100.00% <100.00%> (ø)
dimos/imitation/dataprep/formats/test_lerobot.py	96.42% <100.00%> (+0.23%)	⬆️
dimos/imitation/dataprep/schema.py	100.00% <100.00%> (ø)
dimos/imitation/dataprep/test_core.py	99.72% <100.00%> (+0.28%)	⬆️
dimos/imitation/dataprep/test_legacy_cli.py	100.00% <100.00%> (ø)
dimos/imitation/dataprep/test_mcap_source.py	100.00% <100.00%> (ø)
dimos/imitation/dataprep/test_schema.py	100.00% <100.00%> (ø)
dimos/imitation/test_datacollection_e2e.py	96.52% <100.00%> (+0.02%)	⬆️
dimos/imitation/dataprep/cli.py	33.33% <80.00%> (+14.28%)	⬆️
... and 3 more

... and 15 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[greptile-apps]

Confidence Score: 1/5

[Medium risk] Refactors dataset preparation configuration and alignment logic.

Not safe to merge until the SQLite, LeRobot, camera-memory, and range-based HDF5 failures are addressed; the repeated command reads are a non-blocking performance concern.

Findings

1.  SQLite commands need optional MCAP ▶
2.  LeRobot loses fill flags ▶
3.  Long camera episodes fill memory ▶
4.  Range-based HDF5 export fails ▶
5.  Old commands are read again ▶

Summary

This PR adds explicit feature schemas and quality checks to offline dataset preparation. Before merging, address the SQLite import failure, lost LeRobot fill flags, whole-episode camera memory use, and failed HDF5 export for range-based episodes. Repeated command-history reads also make long multi-episode recordings slower to prepare.

_{Reviews (1) · Last reviewed commit: "refactor(dataprep): separate schemas and..."}

[github-actions]

go2 replay realtime (arm64)

Details

Benchmark suite	Current: d6ae38f	Previous: aea28cf	Ratio
peak memory	1541.227 MB	1529.824 MB	1.01
peak threads	356 threads	346 threads	1.03
network (transport)	2723.398 MB	2724.733 MB	1.00
disk write	1.559 MB	1.277 MB	1.22
instructions	112.409 G	112.227 G	1.00

This comment was automatically generated by workflow using github-action-benchmark.