Skip to content

feat: support H5 evaluation dataset export - #411

Closed
xtk8532704 wants to merge 12 commits into
new-architecture/mainfrom
support-eval-clean
Closed

xtk8532704 wants to merge 12 commits into
new-architecture/mainfrom
support-eval-clean

Conversation

@xtk8532704

@xtk8532704 xtk8532704 commented Sep 10, 2026 •

Copy link
Copy Markdown

Summary

Adds the rosbag-to-H5 export path required to build closed-loop Diffusion Planner evaluation datasets.

Before frame extraction, each source bag is normalized to the current Autoware message schemas with autoware_msg_bag_converter and cached. The exporter then writes schema-v4 H5 shards and a Parquet frame index containing the pose and route metadata needed by closed-loop replay.

This PR replaces #395 and is based on new-architecture/main.

Changes

Rosbag schema normalization

  • Runs every source bag through autoware_msg_bag_converter before ml_planner_data reads it.
  • Caches normalized bags under the configurable converted_bag_root.
  • Writes conversions through an .incomplete directory and publishes them atomically after validating that metadata.yaml exists.
  • Adds and pins the converter, autoware_auto_msgs, and ros2_numpy workspace dependencies.
  • Exposes the converter's vendored autoware_internal_planning_msgs_v1_13 and autoware_perception_msgs_v1_7 packages to the ROS workspace through top-level symlinks, allowing old CDR layouts to be deserialized during conversion.
  • Keeps defensive per-message deserialization handling in ml_planner_data: remaining incompatible messages are reported as warnings, and dropout validation is disabled only when every message for an optional topic is unavailable.

Closed-loop dataset export

  • Adds the override source split and its map-directory layout.
  • Follows directory symlinks while discovering staged rosbag selections.
  • Adds configurable future_steps (default: 80) and uses it consistently for label generation, valid-frame bounds, and topic-gap exclusion ranges.
  • Extracts ego_x, ego_y, and ego_yaw from odometry and stores them in both H5 metadata and the Parquet index.
  • Adds optional split_routes generation:
    • consecutive route messages with the same start pose form one route group;
    • minimum travel distance is evaluated per route group;
    • each retained group is written to its own H5 shard;
    • route_group_id is included in shard metadata and the Parquet index.
  • Adds the Aaq3qv2Q / ePalette vehicle dimensions used by the evaluation data.

H5 output and documentation

  • Introduces a shared schema-v4 H5 writer for full-bag and single-frame/open-loop exports.
  • Validates the shared frame axis, finite numeric values, and strictly increasing timestamps.
  • Writes H5 files atomically and chunks model tensors one frame at a time.
  • Extends the ml_planner_data Python API with per-frame metadata access for shared open-loop output.
  • Tightens resume validation so incompatible or incomplete shards are rejected rather than silently reused.
  • Documents the H5/Parquet schema and the adjacent frames.tags.json sidecar format for mutable segment tags.
  • Documents ROS dependency import/build steps and the required frame interval for each workflow.

Defaults and compatibility

  • H5 format version is now 4. Existing version-3 shards must be regenerated with overwrite=true; resume=true rejects older shards.
  • future_steps=80 preserves the existing prediction horizon by default.
  • split_routes=false preserves one-shard-per-bag behavior for ordinary train/validation generation.
  • Training and open-loop generation default to frame_interval=0.5.
  • Closed-loop releases must set frame_interval=0.1 to match the 10 Hz rollout clock.
  • Converted bags consume additional disk space under converted_bag_root and are reused on subsequent runs.

Validation

  • Built ml_planner_data in the ROS 2 workspace with the pinned dependencies.
  • Verified that colcon discovers the converter and both vendored legacy message packages.
  • Verified Python syntax for create_h5_dataset.py.
  • Checked modified C++ sources with clang-format --dry-run --Werror.

@xtk8532704 xtk8532704 changed the title Support eval clean feat: support H5 evaluation dataset export Sep 10, 2026
@xtk8532704
xtk8532704 requested a review from yhisaki September 10, 2026 03:28
@xtk8532704
xtk8532704 marked this pull request as ready for review September 10, 2026 03:28
@xtk8532704 xtk8532704 closed this Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant