[mustafab0]

Stacks on #4788

Why do we need this?

The operator needs to see the robot and hold it from a browser.

What is this feature?

A web page for hybrid-teleop-sim: the arm camera, the operator hold status (ON/OFF, route, reason, time since) and two buttons, "Request operator" and "Acknowledge".

The blueprint declares three web channels (camera as JPEG, hold status, hold command). The hold status is paced, so a change right after another is not dropped. The page is plain HTML in web/hybrid-teleop, served by the local relay the blueprint starts. The hold command channel carries OperatorHoldCommand through the operator_hold_command.json.v1 decoder; a malformed command is rejected (decode_failed) and the page's log line shows it.

How to manually run this

dimos run hybrid-teleop-sim --serve-dir web/hybrid-teleop, then open http://localhost:7780.

Click "Request operator": status goes ON. Click "Acknowledge": status goes OFF.

Which issue(s) does this PR close?

None.

Checklist

• I have read and approved the CLA.

[github-actions]

go2 replay realtime (arm64)

Details

Benchmark suite	Current: ce47c47	Previous: e4b62ac	Ratio
peak memory	1843.207 MB	1840.531 MB	1.00
peak threads	358 threads	365 threads	0.98
network (transport)	2736.197 MB	2730.976 MB	1.00
disk write	1.688 MB	1.695 MB	1.00
instructions	137.595 G	137.761 G	1.00

This comment was automatically generated by workflow using github-action-benchmark.

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
7361	1	7360	64

View the top 1 failed test(s) by shortest run time

dimos.e2e_tests.test_dimsim_spatial_memory::test_go_to_the_bed

Stack Traces | 273s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7608172b3bc0>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7608159bbec0>
wait_for_system_ready = <function wait_for_system_ready.<locals>.wait at 0x7608159bbf60>
human_input = <function human_input.<locals>.send_human_input at 0x760815a500e0>
dim_sim = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x760988857f20>
explore_house = <function explore_house.<locals>.explore at 0x760815a50900>

    @pytest.mark.self_hosted_large
    def test_go_to_the_bed(
        lcm_spy, start_blueprint, wait_for_system_ready, human_input, dim_sim, explore_house
    ) -> None:
        start_blueprint(
            "run",
            "unitree-go2-agentic",
            simulator="dimsim",
        )
        wait_for_system_ready(timeout=300.0)
    
        explore_house()
    
        human_input("go to the bed")
    
>       lcm_spy.wait_until_odom_position(-3.567, -1.332, threshold=2, timeout=180)

dim_sim    = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x760988857f20>
explore_house = <function explore_house.<locals>.explore at 0x760815a50900>
human_input = <function human_input.<locals>.send_human_input at 0x760815a500e0>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7608172b3bc0>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7608159bbec0>
wait_for_system_ready = <function wait_for_system_ready.<locals>.wait at 0x7608159bbf60>

dimos/e2e_tests/test_dimsim_spatial_memory.py:33: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/lcm_spy.py:143: in wait_until_odom_position
    self.wait_for_message_result(
        predicate  = <function LcmSpy.wait_until_odom_position.<locals>.predicate at 0x7608159bba60>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7608172b3bc0>
        threshold  = 2
        timeout    = 180
        x          = -3.567
        y          = -1.332
dimos/e2e_tests/lcm_spy.py:129: in wait_for_message_result
    wait_until(
        event      = <threading.Event at 0x76081727ddf0: unset>
        fail_message = 'Failed to get to position x=-3.567, y=-1.332'
        listener   = <function LcmSpy.wait_for_message_result.<locals>.listener at 0x7608159b9f80>
        predicate  = <function LcmSpy.wait_until_odom_position.<locals>.predicate at 0x7608159bba60>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7608172b3bc0>
        timeout    = 180
        topic      = '/odom#geometry_msgs.PoseStamped'
        type       = <class 'dimos.msgs.geometry_msgs.PoseStamped.PoseStamped'>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

predicate = <bound method Event.is_set of <threading.Event at 0x76081727ddf0: unset>>

    def wait_until(
        predicate: Callable[[], bool],
        *,
        timeout: float,
        interval: float = 0.1,
        message: str | None = None,
    ) -> None:
        """Poll ``predicate`` until it returns truthy or ``timeout`` elapses."""
        deadline = time.monotonic() + timeout
        while time.monotonic() < deadline:
            if predicate():
                return
            time.sleep(interval)
>       raise TimeoutError(message or f"Timed out after {timeout}s waiting for condition")
E       TimeoutError: Failed to get to position x=-3.567, y=-1.332

deadline   = 9265871.283507653
interval   = 0.1
message    = 'Failed to get to position x=-3.567, y=-1.332'
predicate  = <bound method Event.is_set of <threading.Event at 0x76081727ddf0: unset>>
timeout    = 180

.../utils/testing/waiting.py:35: TimeoutError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The change reaches the simulated xArm operator workflow.

Fix the camera fallback before merging so operators do not lose the live image when the camera transport changes.

Findings

1.  Camera stays hidden after fallback ▶

Summary

Adds a browser page for hybrid-teleop-sim that shows the arm camera and operator hold details, sends hold requests and acknowledgements through declared web channels, and keeps hold status paced and available for replay.

The camera fallback needs to be fixed before merging: after WebRTC switches to JPEG, incoming frames are drawn into a hidden canvas while the stopped video remains visible. Browser execution confirmed this loss of the live camera. PRs #4790, #4791, and #4793 retain the same behavior and do not resolve it.

The custom page does not hide /sdk.js; the relay preserves that route ahead of custom static files.

• Operators can view the simulated arm and manage its hold in a browser.

_{Reviews (1) · Last reviewed commit: "feat(web): operator page with camera, ho..." · Reviewed by Greptile}

[greptile-apps]

Camera stays hidden after fallback

When the camera switches from WebRTC to JPEG after a relay reconnect, the page draws incoming JPEG frames into a hidden canvas. The WebRTC branch hides canvas and shows video, but the JPEG branch never reverses those changes. The operator therefore keeps seeing the stopped video instead of the live camera, even though new frames are arriving and decoding successfully.

Restore canvas.hidden = false, set video.hidden = true, and clear the obsolete video.srcObject when JPEG frames arrive. This loss of the operator's live camera should be fixed before merging.

Suggested change

	} else if (value instanceof Uint8Array) {
	pending = value;
	if (!drawing) drawFrames();
	} else if (value instanceof Uint8Array) {
	canvas.hidden = false;
	video.hidden = true;
	video.srcObject = null;
	pending = value;
	if (!drawing) drawFrames();

Knowledge Base Used: Web cockpit and relay transport

Artifacts

Executed Chromium camera transition test

• This executed harness serves unchanged git HTML with an explicit SDK fixture and asserts real track rendering, JPEG drawing, and visibility state, reproducing the hidden-frame defect.

Camera transition results and exact source quotes

• The successful test run records command, working directory, exit code, exact requested source lines, higher-stack comparisons, and browser measurements, confirming JPEG pixels update on a hidden canvas.

▶ Base commit has no operator page

• Chromium opens the harness endpoint backed by the base git-tree check and records its 404 because the operator page does not yet exist, establishing the pre-change scope.

Base page absence screenshot

• The browser captures the missing-page response for the base commit, showing that no comparable pre-change camera UI exists.

▶ Head page hides JPEG frames after RTC video

• Chromium exercises the JPEG control, native video track, and subsequent JPEG frames on the actual head page, demonstrating that the camera remains on the obsolete video instead of showing the newly drawn JPEGs.

Head page after JPEG drawing remains hidden

• The browser captures the final camera state with measured fixture diagnostics after two post-RTC JPEG draws, documenting that the updated canvas remains hidden.

Executed higher-stack inspection commands

• This executed shell script checks camera visibility assignments and SDK/bridge changes across all requested higher-stack commits, checking for a later complete fix.

Higher-stack inspection results

• The inspection output records command, working directory, exit code, visibility assignments, and unchanged tracked files, showing that the higher stack does not repair the transition.

_{Ran code and verified through TREX}