Repository navigation
fix: let MCP input use capture_runtime image coordinates - #2496
Conversation
Signed-off-by: Бат-Орших <34618353+orshih6@users.noreply.github.com>
|
@orshih6 is attempting to deploy a commit to the playcanvas Team on Vercel. A member of the Team first needs to authorize it. |
|
The Label check is failing only because the PR doesn't have a release label yet, and I can't add one from a fork. Could a maintainer add one? This is a backward-compatible bug fix, so "release: next patch" seems right to me, but I'm fine with "release: next minor" if you'd rather not backport the new |
Signed-off-by: Бат-Орших <34618353+orshih6@users.noreply.github.com>
kpal81xd
left a comment
There was a problem hiding this comment.
Verified on dev with the PR build: on a 1440x900 canvas (captured at 800x500), runtime:input with space: 'capture' at (400, 250) lands at (720, 450), the canvas centre. Without space it still lands at (400, 250). capture_runtime meta reports scaleX/scaleY of 1.8. Thanks!
runtime:capturecopies the canvas drawing buffer (device pixels) and downscales it to at most 800 px wide, butruntime:inputdispatches mouse/touch events at CSS pixels from the canvas top-left. When the device pixel ratio isn't 1, or the canvas is wider than 800 px, a point read off the screenshot lands somewhere else. For example, the centre of a 1600 px canvas is at x=400 in the image, but a click at x=400 lands a quarter of the way across.Changes:
src/launch/mcp/capture.tswith two small pure helpers:captureSize(the existing downscale-to-800 rule, moved out of the capture handler) andcaptureToCss(image point to CSS point).runtime:capturenow also returnscssWidth,cssHeight,scaleXandscaleY(CSS px per image px) inmeta, so callers can convert coordinates themselves.runtime:inputaccepts an optionalspace: 'capture'on the payload. When it's set, mouse/touchx/yare treated as capture-image coordinates and converted to CSS pixels before dispatch. Without it, behaviour is unchanged, so this is backward compatible.The
capture_runtime/inject_inputtool schemas and descriptions (including "match capture_runtime's framing") are defined in the MCP server, not in this repo. Exposingspaceto agents and correcting that description needs a matching change there. Until then, agents can use the newmetascale fields.Testing: added mocha tests in
test/launch/mcp/capture.test.ts. They cover the identity case, the downscale above 800 px, DPR 2 without downscale, and DPR 2 with downscale. eslint and prettier are clean on the touched files.tscreports the same pre-existing errors in unrelated files with and without this change, and none in the touched files. I couldn't test this end to end against a live launch page.Fixes #2493
This change was prepared with an AI coding agent (Claude) and reviewed by me before submitting. I'm happy to make any changes you'd like.