robot_simulation: let a control program be written in Python - #947
Draft
Akshay-2007-1 wants to merge 12 commits into
Draft
robot_simulation: let a control program be written in Python#947Akshay-2007-1 wants to merge 12 commits into
Akshay-2007-1 wants to merge 12 commits into
Conversation
Adds createPythonCSE(code), a Python-flavoured sibling of the
existing Source-only createCSE(...): the string passed to it is
interpreted by an embedded py-slang CSE machine (Control/Stash/
generateCSEMachineStateStream, imported as a plain library dependency
- see companion PR on source-academy/py-slang), stepped in lockstep
with the physics tick exactly like the existing js-slang-driven path.
- pythonRuntime.ts (new): builds a py-slang Context seeded with SICPy
builtins plus the ev3_* robot API (1:1 wrapping of ev3_functions.ts,
so a Python program and a Source one drive the simulation
identically), output routed to the Robot Console.
- evaluate.ts: runPythonECEvaluator, the async-generator mirror of the
existing runECEvaluator.
- Program.ts: language/pyContext constructor params; an async "pump"
drives the Python generator once per tick, since fixedUpdate is
called synchronously but py-slang's stepper is async.
- helper_functions.ts: createPythonCSE export, plus
unwrapCallbackResult() - a defensive fix for an unrelated
pre-existing js-slang bug (closureToJS leaves a tail-call wrapper
{isTail, value} on values returned from native callbacks) that
broke robot_simulation's init_simulation for every program, Python
or Source, before any of this change. Worth reporting upstream to
js-slang separately.
- lib/buildtools/.../commons.ts: acorn ecmaVersion 6 -> 2020, needed
because py-slang's BigInt literals otherwise crash buildtools' post-
esbuild AST pass (py-slang isn't in the external:['js-slang*']
esbuild list, so it gets fully inlined).
- Program.python.test.ts (new): real (non-mocked) test proving
multi-tick Python stepping mutates real py-slang state correctly.
Scope note: this is intentionally NOT a Conductor migration.
robot_simulation still imports js-slang/context and loads exactly as
before; the *setup* program (init_simulation, createWorld, ...) stays
Source. Only the robot's *control* program can now be Python. Making
the whole module - setup included - Python-only requires migrating
robot_simulation to the BaseModulePlugin/attachModuleMethod pattern
(as csg/rune/curve/plotly already did), which is separate, larger,
follow-up work.
Local proof: 123/123 tests pass, including the new Python-path tests
alongside the unchanged existing js-slang-path tests. Manually
confirmed end-to-end in a local frontend + language-directory +
locally-built modules/py-slang stack: a Python control program
(print + two ev3_runToRelativePosition calls) rendered the 3D scene
and visibly drove the robot forward, with print() output reaching the
Robot Console.
Depends on source-academy/py-slang#457 (draft, not yet published) -
robot_simulation/package.json currently points @sourceacademy/py-slang
at a portal: path local to this machine as a result; that'll need to
become a real registry reference once py-slang#457 publishes.
Draft: checkpoint of local proof-of-concept work. Not requesting
review yet.
Replaces the js-slang/context workaround (Step 1) with the same
BaseModulePlugin/attachModuleMethod pattern csg/rune/curve use, so the
whole program - setup and control program alike - can be written in
any Conductor language, not just Source-for-setup.
Architecture: the module runs inside Conductor's runner Worker, which
has no DOM/WebGL, so it can no longer own THREE.WebGLRenderer/
OrbitControls/the canvas the way it used to. Split by concern
(pix_n_flix's precedent, not a "who renders" split): the module keeps
physics (rapier3d-compat, worker-safe) and the robot control program's
CSE stepping, and streams entity transforms to a new RobotSimulation
tab plugin once per physics tick over a dedicated state channel
(entity descriptors once per entity, then transferable Float32Array
snapshots every tick) plus an RPC control channel for console/state/
sensor pushes - see protocol.ts and SceneRegistry's doc comment.
Renderer.ts is gone from the bundle entirely; the tab now owns the
live 3D view, built from engine helpers (getCamera/loadGLTF/
MeshFactory) that were already DOM-safe. ColorSensor's sensing moved
from a GPU render-and-readback (no WebGL in a Worker) to a physics
raycast against a small collider-color registry - documented as a
narrowing versus the original (a colored `create_paper` overlay, which
has no collider, is now invisible to the sensor).
createCSE (a Source-flavoured control program) is deliberately not
wired up: buildtools' `external: ['js-slang*']` esbuild rule means any
js-slang import anywhere in a bundle - even unreachable - compiles to
a top-level `require('js-slang/...')` with nothing to resolve it
inside a Worker. All remaining js-slang-backed code (ProgramError,
GeneralRuntimeError et al.) is replaced with conductor/common's
evaluator error types or plain Error, and js-slang is dropped from
package.json. createPythonCSE is unaffected - py-slang is bundled
normally, same as before.
Also bumps the @sourceacademy/conductor catalog pin to ^0.8.3 to match
what the portal-linked local py-slang checkout requires (was ^0.8.2),
fixing a duplicate-package TS error from two physical copies of
conductor's types; and adds ProgramError to the throw-runtime-error
lint rule's ignoredNames, since it never crosses the evaluator
boundary and no longer extends js-slang's RuntimeSourceError.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dji2eG7jb4tww8LowuSs7n
World.setState() dispatched worldStateChange before assigning this.state, so every listener (including the tab's $worldStateChanged RPC) always observed the previous state - the tab's "World: ..." readout got stuck on "ready" forever, even once the world was actually "running". Assign the new state before dispatching the event. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dji2eG7jb4tww8LowuSs7n
…nd a saved view
The default camera sat 2.8m away from an EV3 that's only ~0.25m across,
rendering the robot as a barely-visible speck. Default to a close-in,
mostly-overhead ("bird's eye") framing instead - ~0.33m out, elevated to
a steep angle - so the robot fills a comfortable fraction of the frame
by default while still reading as a 3D object rather than a flat
silhouette.
Also, in the RobotSimulation tab:
- "F to focus" (Unity/Blender-style): press F while the canvas has focus
to recenter OrbitControls on the EV3's current (live) position and
reframe based on its actual bounding box, preserving whatever orbit
angle the viewer had set up. Scoped to the canvas only (not the whole
page) so it doesn't steal "f" keystrokes from the code editor.
- The viewer's camera position/orbit target now survives a reload or a
re-run of the program (persisted to localStorage, keyed per-browser)
instead of resetting to the default every time.
- OrbitControls damping/zoom-speed tuned, and a short on-canvas hint
line ("drag to orbit, scroll to zoom... press F to focus").
Verified with a headless Playwright run against a local rebuild: default
framing screenshot, before/after F-focus after orbiting away, and a
camera-position screenshot surviving a program re-run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dji2eG7jb4tww8LowuSs7n
Investigated whether the full Joel-faithful "instructor prepend / student
control-code" split can be reached at the module level under Conductor.
It can't yet: frontend's evalEditorSaga Conductor branch concatenates
`${prepend}\n${studentCode}` into one plain string before it ever reaches
an evaluator, and py-slang's PyCseEvaluator.evaluateChunk(chunk: string)
only ever receives that single opaque string - no delimiter, chunk
boundary, or line-count is forwarded into the evaluator or exposed to a
running module (the frontend's own preludeLineOffset is saga-local
bookkeeping for error line numbers, never sent over the wire). Reaching
the real split needs new plumbing above this module (e.g. evaluateChunk
taking structured {prepend, studentCode}) - out of scope here.
As the documented middle ground, add init_default_simulation(control_code):
one call that builds default physics/world/floor/ev3 and wires up a Python
control program, collapsing the createPhysics/createWorld/createFloor/
createEv3/createPythonCSE/addControllerToWorld/saveToContext/init_simulation
boilerplate a student previously had to write by hand. init_simulation
still exists unchanged for anyone who needs a customised World.
init_default_simulation() now only builds the default scene (no more control_code string arg). Two new pieces support the split: - add_wall/add_paper: friendly wrappers so a student can customise the live scene from the main pane without touching physics/world handles. - run_robot_code(code): pass this to repl's set_evaluator so the robot's own code runs from a separate, rerunnable REPL tab instead of a string baked into setup. Reuses one py-slang Context across runs (REPL-style variable persistence) and stops the previous run's Program before starting a new one so they don't stomp the shared context mid-flight. World gains addLiveController for adding a controller to a world that's already running (addController's start() hookup only fires on the worldStart event, which already happened by the time a student calls add_wall/run_robot_code).
…camera - run_robot_code (robot_simulation) now asks its own tab to come to the front once a run is successfully handed to the physics loop, so driving the robot from the Repl tab brings the 3D view into focus automatically. - set_evaluator (repl) now explicitly asks the Repl tab to come to the front on success, so finishing setup in the main pane lands the student on the Repl tab next instead of leaving them on the editor. Both are no-ops on failure (thrown before reaching the focus call), so an error stays wherever it's already shown rather than yanking focus. - RobotSimulation tab: the camera defaults to the same framing "F to focus" produces (centered/scaled to the EV3's live bounding box) the first time the EV3 actually appears, instead of leaving new viewers staring at whatever getCamera()'s hardcoded default happens to show. Skipped once the viewer has a deliberate view (a saved one, or a manual F-focus/drag already this session).
set_evaluator's 'focus' send raced loadTab()'s async, cross-thread tab bootstrap - sent before the Repl tab had subscribed, it was silently dropped. Now deferred to the tab's own 'request' handshake (sent right after it subscribes) when the tab hasn't connected yet, so the message can no longer go out before anyone's listening. Also fixes the RobotSimulation tab's default-camera-focus flag being sticky for the tab's whole lifetime instead of per Run: re-running the program left the camera wherever a previous run's auto-focus (or the viewer's own drag) had settled, even though the new run's World/EV3 may not be there any more. Reset on every fresh World ($worldStateChanged 'loading'), re-checking localStorage instead of unconditionally clearing so a saved view still survives a re-run as before.
The frontend only ever shows one side-content tab at a time, and won't switch tabs away from wherever a student navigated (SideContentManager's "don't yank the student's focus" guard) - so a separate repl tab and the RobotSimulation tab can never actually sit on screen together, no matter what a module does. Rather than fight that guard, this puts both halves in one tab instead: a small Ace-based editor + Run button now lives right next to the 3D canvas in RobotSimulation's own tab body, wired to a new tab -> module RPC ($runReplCode) that has the exact same effect as run_robot_code/the repl module's Run button. - protocol.ts: RobotSimulationModuleRpc, the tab -> module half of the existing makeRpc pairing (previously module -> tab only). - index.ts: run_robot_code and the new $runReplCode handler now share one __runReplCode implementation; run_robot_code still throws synchronously on a missing World (repl displays that as an error), while the RPC handler catches and logs instead, since there's no caller boundary to surface a throw to there. - RobotSimulation tab: two-column layout (3D view left, embedded editor right); editor code persists to localStorage like the camera view does. run_robot_code/set_evaluator's own $focusTab/'focus' sends are left as-is (still correct, still occasionally useful pre-first-navigation) but their doc comments now say plainly that they rarely do anything once a student has looked at any tab - this embedded editor is the actual fix for wanting the view and the code on screen together.
Adds robot_simulation to conductor-modules.json (the allowlist filterDocsVisibleBundles reads) - without it the module was silently excluded from the generated documentation site, same as any not-yet-migrated bundle. Also fills in JSDoc (@param/@category) on every exported method that was missing it - mainly the raw createX/addControllerToWorld/saveToContext setup API and all ev3_* functions, which previously had no doc comment at all. @category groups the sidebar into Scene Setup / Control Program / EV3 instead of one flat list, matching repl's existing convention. No behavior change - doc comments and one allowlist entry only.
…d_paper
add_paper's overlay has no physics collider (documented, known follow-up
in ColorSensor.ts), so the color sensor's raycast can never detect it -
the color sensor demo the module's own docs point to has never actually
worked with add_paper. add_color_patch adds a thin, flat, fixed cuboid
instead (a real collider, registered with Physics.registerColor like any
other cuboid), so ev3_colorSensor can actually read it.
Verified live: init_default_simulation() + add_color_patch('red', 0, 0,
1, 1) immediately shows Color: rgb(255, 0, 0) in the RobotSimulation
tab's sensor readout.
add_paper's doc comment now points to this for anything that needs to be
sensed, not just seen.
…erun Every run_robot_code/$runReplCode call (each REPL/embedded-editor Run) adds a fresh Program controller rather than replacing one in place, and never removes superseded ones from world.controllers.controllers - only marks them isStopped. ev3_pause() looked up "the" Program controller by name via .find(), which always returns the FIRST one ever added - so on any run after the first, it silently paused a dead, already-stopped Program instead of the one actually executing, making ev3_pause() a no-op. This is what looked like consecutive motor commands (e.g. a forward call immediately followed by a reverse one from a separate Run) "canceling out" instantly regardless of how large a pause was inserted - the pause never touched the program actually running. Fixed by searching from the end of the controllers list for the last non-stopped Program - the one currently driving the run - and exposing Program.isStopped (was private) so ev3_functions.ts can tell them apart. Verified live: a long-running forward command stayed at its original motor velocity throughout a second run's ev3_pause(2500) before finally reversing once the pause elapsed, instead of reversing instantly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
createPythonCSE(code), a Python-flavoured sibling of the existing Source-onlycreateCSE(...): the string passed to it is interpreted by an embedded py-slang CSE machine, stepped in lockstep with the physics tick exactly like the existing js-slang-driven path.Why
First step toward Python EV3 support in the simulator.
robot_simulation's cooperative scheduling (ev3_pause()etc.) relies on stepping an interpreter N times per physics tick, which it already does for Source via js-slang's exposedControl/Stash/generateCSEMachineStateStream. This does the same for Python via source-academy/py-slang#457.Scope - intentionally NOT a Conductor migration
robot_simulationstill importsjs-slang/contextand loads exactly as before. The setup program (init_simulation,createWorld, ...) stays Source; only the robot's control program can now be Python. Making the whole module - setup included - Python-only requires migrating to theBaseModulePlugin/attachModuleMethodpattern (as csg/rune/curve/plotly already did) - separate, larger follow-up work, tracked next.What's in here
pythonRuntime.ts(new) - builds a py-slangContextseeded with SICPy builtins + theev3_*robot API (1:1 wrapping ofev3_functions.ts), output routed to the Robot Consoleevaluate.ts-runPythonECEvaluator, async-generator mirror of the existingrunECEvaluatorProgram.ts-language/pyContextconstructor params; an async pump drives the Python generator once per tickhelper_functions.ts-createPythonCSEexport, plusunwrapCallbackResult(): a defensive fix for an unrelated pre-existing js-slang bug (closureToJSleaves a tail-call wrapper on native-callback return values) that brokeinit_simulationfor every program, Python or Source, before this change. Worth reporting upstream to js-slang separately.acornecmaVersion 6 -> 2020 (py-slang's BigInt literals otherwise crash the post-esbuild AST pass)Program.python.test.ts(new) - real, non-mocked test proving multi-tick Python stepping mutates real py-slang stateTesting
123/123 tests pass (existing js-slang-path tests unchanged). Manually confirmed end-to-end in a local frontend + language-directory + locally-built stack: a Python control program (print + two
ev3_runToRelativePositioncalls) rendered the 3D scene and visibly drove the robot forward, withprint()reaching the Robot Console.Depends on
source-academy/py-slang#457 (draft, unpublished).
package.jsoncurrently points@sourceacademy/py-slangat aportal:path local to this machine as a result - will need to become a real registry reference once that PR publishes.Status
Draft - checkpoint of local proof-of-concept work, not requesting review yet.