Skip to content

feat(web): point-cloud view on the operator page - #4791

Closed
mustafab0 wants to merge 1 commit into
mustafa/feat/operator-ui-estopfrom
mustafa/feat/operator-ui-cloud
Closed

mustafab0 wants to merge 1 commit into
mustafa/feat/operator-ui-estopfrom
mustafa/feat/operator-ui-cloud

Conversation

@mustafab0

@mustafab0 mustafab0 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Stacks on #4790

Why do we need this?

The operator sees the scene in 3D in a fixed world frame; a later PR clicks a place point in it.

What is this feature?

A 3D point-cloud view on the operator page, under the camera.

hybrid-teleop-sim turns on the sim's ray-cast cloud (world frame, 5 Hz, 1 cm voxels) as scene_pointcloud, sent as voxels.zlib.v1. The page draws it on a plain canvas with a ground grid and world axes; turn, zoom, pan, Reset view. No external scripts.

How to manually run this

dimos run hybrid-teleop-sim --serve-dir web/hybrid-teleop, then open http://localhost:7780.

The cloud appears under the camera. Drag to turn, wheel to zoom, right or shift drag to pan, Reset view to go back. Moving the arm does not move the cloud.

Tested: live headless sim with a scripted browser: screenshots, frame size, draw time (about 1 ms; 50k synthetic voxels about 9 ms), and a check that moving the arm does not move the cloud.

Which issue(s) does this PR close?

None.

Checklist

  • I have read and approved the CLA.

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go2 replay realtime (arm64)

Details
Benchmark suite Current: b4fec03 Previous: f9c5a5c Ratio
peak memory 1796.73 MB 1795.23 MB 1.00
peak threads 354 threads 366 threads 0.97
network (transport) 2736.313 MB 2730.945 MB 1.00
disk write 1.523 MB 1.723 MB 0.88
instructions 137.331 G 137.801 G 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

@@                        Coverage Diff                         @@
##           mustafa/feat/operator-ui-estop    #4791      +/-   ##
==================================================================
- Coverage                           81.69%   81.68%   -0.01%     
==================================================================
  Files                                1702     1702              
  Lines                              160592   160593       +1     
  Branches                            13448    13448              
==================================================================
- Hits                               131189   131177      -12     
- Misses                              25768    25776       +8     
- Partials                             3635     3640       +5     
Components Coverage Δ
Tests 95.74% <ø> (-0.02%) ⬇️
Flag Coverage Δ
OS-macos-latest 76.34% <100.00%> (+0.01%) ⬆️
OS-ubuntu-24.04-arm ?
OS-ubuntu-latest 76.52% <100.00%> (+<0.01%) ⬆️
Py-3.10 76.53% <100.00%> (+0.72%) ⬆️
Py-3.11 76.51% <100.00%> (+<0.01%) ⬆️
Py-3.12 76.51% <100.00%> (-0.02%) ⬇️
SelfHosted-Linux 40.59% <100.00%> (-0.01%) ⬇️
SelfHosted-macOS 40.85% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
.../manipulators/xarm/blueprints/hybrid_teleop_sim.py 91.42% <100.00%> (+0.12%) ⬆️

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mustafab0
mustafab0 force-pushed the mustafa/feat/operator-ui-estop branch from 7a21ea9 to b2eb928 Compare October 23, 2026 01:00
@mustafab0
mustafab0 force-pushed the mustafa/feat/operator-ui-cloud branch from d55b042 to 154613b Compare October 23, 2026 01:00
@mustafab0
mustafab0 added this pull request to stack #4809 October 23, 2026 19:07
@mustafab0
mustafab0 force-pushed the mustafa/feat/operator-ui-estop branch from b2eb928 to 2d6cbd7 Compare October 23, 2026 19:30
@mustafab0
mustafab0 force-pushed the mustafa/feat/operator-ui-cloud branch 2 times, most recently from 0cd0fd6 to b992afc Compare October 23, 2026 21:56
@mustafab0
mustafab0 force-pushed the mustafa/feat/operator-ui-estop branch from 2d6cbd7 to c2d6a89 Compare October 23, 2026 21:56
hybrid-teleop-sim turns on the sim's ray-cast point cloud: 80 x 60 rays
from the wrist camera at 5 Hz, points in the world frame, 1 cm voxels.
The sim's pointcloud output is renamed to scene_pointcloud so it does not
share a topic with ObjectSceneRegistrationModule's object clouds, and it
goes to the browser as voxels.zlib.v1 (res 0.01, frame_id "world" for the
page), about 0.5-0.7 KB per frame.

web/hybrid-teleop gets a 3D view under the camera: voxel centres drawn
as squares on a 2D canvas with a depth buffer, coloured by height, over a
ground grid and the world axes. Drag turns the view, the wheel zooms,
right or shift drag pans, and Reset view goes back to the start. project()
and unproject() map between world points and canvas pixels. The status
line shows voxel count, resolution, frame, fps and draw time.

This branch was previously deployed

1 inactive deployment
cachix — b4fec039 Deployed Oct 23, 2026 by mustafab0 via cachix-build #14176
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant