Skip to content

fix: let MCP input use capture_runtime image coordinates - #2496

Merged
kpal81xd merged 3 commits into
playcanvas:mainfrom
orshih6:contribot/issue-2493
Nov 18, 2026
Merged

kpal81xd merged 3 commits into
playcanvas:mainfrom
orshih6:contribot/issue-2493

Conversation

@orshih6

@orshih6 orshih6 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

runtime:capture copies the canvas drawing buffer (device pixels) and downscales it to at most 800 px wide, but runtime:input dispatches mouse/touch events at CSS pixels from the canvas top-left. When the device pixel ratio isn't 1, or the canvas is wider than 800 px, a point read off the screenshot lands somewhere else. For example, the centre of a 1600 px canvas is at x=400 in the image, but a click at x=400 lands a quarter of the way across.

Changes:

  • New src/launch/mcp/capture.ts with two small pure helpers: captureSize (the existing downscale-to-800 rule, moved out of the capture handler) and captureToCss (image point to CSS point).
  • runtime:capture now also returns cssWidth, cssHeight, scaleX and scaleY (CSS px per image px) in meta, so callers can convert coordinates themselves.
  • runtime:input accepts an optional space: 'capture' on the payload. When it's set, mouse/touch x/y are treated as capture-image coordinates and converted to CSS pixels before dispatch. Without it, behaviour is unchanged, so this is backward compatible.

The capture_runtime / inject_input tool schemas and descriptions (including "match capture_runtime's framing") are defined in the MCP server, not in this repo. Exposing space to agents and correcting that description needs a matching change there. Until then, agents can use the new meta scale fields.

Testing: added mocha tests in test/launch/mcp/capture.test.ts. They cover the identity case, the downscale above 800 px, DPR 2 without downscale, and DPR 2 with downscale. eslint and prettier are clean on the touched files. tsc reports the same pre-existing errors in unrelated files with and without this change, and none in the touched files. I couldn't test this end to end against a live launch page.

Fixes #2493


This change was prepared with an AI coding agent (Claude) and reviewed by me before submitting. I'm happy to make any changes you'd like.

Signed-off-by: Бат-Орших <34618353+orshih6@users.noreply.github.com>
@vercel

vercel Bot commented Nov 10, 2026

Copy link
Copy Markdown

@orshih6 is attempting to deploy a commit to the playcanvas Team on Vercel.

A member of the Team first needs to authorize it.

@orshih6

orshih6 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

The Label check is failing only because the PR doesn't have a release label yet, and I can't add one from a fork. Could a maintainer add one? This is a backward-compatible bug fix, so "release: next patch" seems right to me, but I'm fine with "release: next minor" if you'd rather not backport the new space option.

Signed-off-by: Бат-Орших <34618353+orshih6@users.noreply.github.com>
@kpal81xd kpal81xd added the release: next patch Ticket marked for next patch release label Nov 18, 2026

@kpal81xd kpal81xd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on dev with the PR build: on a 1440x900 canvas (captured at 800x500), runtime:input with space: 'capture' at (400, 250) lands at (720, 450), the canvas centre. Without space it still lands at (400, 250). capture_runtime meta reports scaleX/scaleY of 1.8. Thanks!

@kpal81xd
kpal81xd merged commit c7185b5 into playcanvas:main Nov 18, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release: next patch Ticket marked for next patch release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MCP inject_input coordinates don't match capture_runtime screenshots

2 participants