Repository navigation
feat(screen): stream the display framebuffer to local clients (DisplayFrame mirroring) - #11681
Draft
jamesarich wants to merge 21 commits into
Draft
jamesarich wants to merge 21 commits into
jamesarich wants to merge 21 commits into
Conversation
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Aug 31, 2026
Contributor
Author
|
Cross-platform feature spec (client-agnostic wire contract, reassembly rules, input mapping, UX and device requirements): meshtastic/design#142 |
Contributor
⚡ Try this PR in the Web FlasherNote Building this pull request… the flash button, badges and supported-board |
jamesarich
force-pushed
the
screen-mirror-poc
branch
from
September 8, 2026 21:04
62d6d4c to
b823d64
Compare
jamesarich
force-pushed
the
screen-mirror-poc
branch
from
September 9, 2026 16:18
29f4a5b to
891954e
Compare
Contributor
Author
|
Rebased onto develop. It had gone conflicting overnight ( Also follows device-ui#394's new shape. The mirror moved out of device-ui's |
jamesarich
force-pushed
the
screen-mirror-poc
branch
from
October 2, 2026 17:47
48d18ea to
08c0a10
Compare
Contributor
|
…rame chunks ScreenMirror snapshots the 1bpp OLEDDisplay buffer after each committed frame (updateUiFrame), diffing against the last sent snapshot so only changed frames stream. Armed via the new AdminMessage set_display_mirror (continuous) / get_display_frame_request (one-shot); chunks drain through PhoneAPI's STATE_SEND_PACKETS chain like XModem, so BLE, serial and TCP clients all receive them. The lockdown path mirrors the LOCKED frame, preserving display redaction. Excludable with MESHTASTIC_EXCLUDE_SCREEN_MIRROR. Bumps the protobufs submodule to the DisplayFrame PoC commit and regenerates nanopb sources (FromRadio_size stays 510 <= 512).
Gate frame delivery on getAdminAuthorized under access control (screen pixels are operator content, same rule as mesh packets); move the drain to lowest getFromRadio/available priority behind the replay drain; give each PhoneAPI its own (frameId, offset) cursor via ScreenMirror::copyChunk so coexisting BLE/serial/TCP clients each receive complete frames; disarm and free the snapshot on client close and on setMirror(false), with memaudit accounting and a geometry-change realloc guard; honor the local-connection-only contract for both admin verbs (mp.from == 0); demote the per-frame onNotify log to TRACE; hoist the guard into a derived HAS_SCREEN_MIRROR in configuration.h. Regenerates protos for the reviewed contract (FORMAT_UNSPECIFIED = 0, MONO_VLSB = 1, width/height int_size:16).
…mirroring ScreenMirror captures the color-region table with each snapshot when the frame signature changes (regions byte-swapped from panel order to logical RGB565, defaults = White / theme body background) and streams it as FromRadio.display_palette chunks ahead of frames, with per-client (signature, region) cursors like the frame drain. Frames reference the palette via palette_signature; monochrome-only builds send 0 and no palettes. Also classify USE_TFTDISPLAY/HAS_SPI_TFT panels (T-Deck) as TFT in DeviceMetadata.display.
…frame TFTDisplay/HUB75 consume and then clear the per-frame color-region table inside display(), so the post-update capture always photographed an empty table and mirrors rendered monochrome. The drivers now hand ScreenMirror the palette (signature, defaults, regions — all panel-byte-order, swapped to logical RGB565 in the store) right before clearing, and the mirror stamps frames with the signature of the palette they were painted with.
Guard the diff-path palette capture with GRAPHICS_TFT_COLORING_ENABLED too (env:native defines USE_TFTDISPLAY without coloring, so the site referenced an undeclared identifier and broke the CI/test env); test HAS_TOUCHSCREEN by value, not definedness — it defaults to 0, so every screened device reported touch; stamp frames with the signature captured WITH the snapshot and treat a palette-only change as frame-worthy, so theme recolors reach the client without pixel churn and mid-drain captures stay coherent; compute the frame size in 32 bits and refuse panels past the uint16 chunk cursors; make setMirror idempotent (no disconnect log spam); report PanelClass UNSPECIFIED on portduino where the panel is runtime-selected.
On HAS_TFT builds, tftSetup registers device-ui's new flush observer: every LVGL dirty rect arrives pre-byte-swap (native RGB565) and queues into ScreenMirror's bounded rect pool (96 KB, 32 rects), draining through the existing display_frame chunker with the rect fields and per-rect frame_ids. Arming or a one-shot triggers a full repaint through device-ui so a new client synchronizes the whole screen; queue overflow drops rects and requests one resync repaint when drained. Spike scope: the rect queue is single-consumer (per-client cursors cover mono frames only). Requires the device-ui flush-observer branch; the vendored pin is overridden locally and intentionally left uncommitted until device-ui merges. Regenerates protos for Format.RGB565.
A 320x240 RGB565 sync is ~150 KB; the 96 KB pool overflowed mid-repaint and the resync loop could never deliver a complete first frame — clients composited incremental rects over black. 192 KB (PSRAM-first on ESP32) holds a full repaint plus concurrent updates, and 64 rect headers cover icon-sized bursts.
…UI_MIRROR The vendored device-ui pin predates the flush observer; without the gate this branch cannot build t-deck-tft in CI. The define is set locally alongside the device-ui override until the device-ui change merges and the pin moves.
A lazily attached observer (inputBroker is created after tftSetup) maps remote send_input_event traffic onto device-ui's injection seam: broker directions become LV_KEY_* (note the deliberate LEFT/RIGHT cross-map — the broker codes were modeled on LVGL keys but those two are swapped), SELECT with coordinates synthesizes a long press, USER_PRESS taps, BACK/CANCEL map to ESC, and kb_char passes through. Gated with the MUI mirror define until the device-ui side merges.
…tBroker MUI builds never construct an InputBroker — Modules.cpp skips it (along with SystemCommands and buzzer feedback) whenever displaymode is COLOR — so AdminMessage.send_input_event hit AdminModule's null-broker guard and died: remote control was silently inert on every MUI device. AdminModule now offers the event to graphics::muiInjectInputEvent first, which maps it onto device-ui's virtual LVGL devices and reports whether it consumed it, falling through to the broker on BaseUI builds. Verified on a T-Deck with a headless mirror client: injecting USER_PRESS at (35,105) navigates MUI to Group Channels.
getDeviceMetadata derived display geometry from the BaseUI screen object, which MUI builds never create — so color-UI devices advertised no display at all, and clients could not tell they were touch-capable. Ask LVGL for the logical resolution instead (already rotated) and report RGB565 / TFT / has_touch. Verified on a T-Deck: 320x240, RGB565, TFT, touch true.
Vertical becomes encoder rotation (what actually moves focus in an LVGL group) and horizontal becomes the slider keys, deliberately inverted, so remote control matches what the device's own trackball driver emits.
Four defects from review. The idempotence guard added with the wake work skipped cleanup when a one-shot request disarmed, stranding the 192 KB rect pool and leaving the mirror armed for a departed client — cleanup is now unconditional and only the log is guarded. The rect FIFO has a single shared cursor, so it now records the connection that claims it rather than splitting each frame between coexisting BLE and serial clients. Panel dimensions come from LVGL at registration instead of growing as a running max of rect extents, which contradicted both the wire contract and DeviceMetadata. And on overflow the backlog is dropped and a repaint requested immediately, because dropping only the newest rect wedged the pool until the queue drained and thrashed the resync — the reason a fresh arm often never delivered a full frame. Also: notify only on the empty-to-non-empty transition rather than per LVGL flush, log a failed pool allocation, keep PSRAM out of the internal memaudit budget, and enable device-ui injection explicitly at setup.
The previous commit swept in two working-tree files that exist only to build the MUI mirror against a local device-ui checkout: a symlink:// lib_deps pointing at a path on one machine, and the MESHTASTIC_MUI_MIRROR define on t-deck-tft. Neither resolves anywhere else, so t-deck-tft could not build. Restores the pinned device-ui archive and the stock variant flags.
setMirror cleared muiOwner outside any guard, so every board without a TFT failed to compile — the member only exists on the MUI path. The assignment was redundant anyway: freeSnapshotLocked() releases the pool and the ownership claim together. The guards were also drawn on HAS_TFT while the code they protect needs a device-ui carrying the flush observer, so a color build that never opts in still carried the rect queue in .bss. Replaces the ad-hoc HAS_SCREEN_MIRROR && defined(MESHTASTIC_MUI_MIRROR) pairs with a single derived HAS_MUI_MIRROR next to HAS_SCREEN_MIRROR, and gates the queue, the pool, the input seam and their declarations on it. Verified on all three build shapes: t-deck-tft with the opt-in, picomputer-s3 (TFT, opted out) and heltec-v3 (no TFT).
configuration.h defaults USE_TFTDISPLAY to 0 with #ifndef, so it is always defined and `defined(USE_TFTDISPLAY)` is always true — every board with a screen reported panel_class TFT, plain OLEDs included. Value-test it, as HAS_TOUCHSCREEN already is a few lines below. That also brings the conditional under the repo's five-defined() lint cap, which is what surfaced it.
device-ui moved the mirror out of its InputDriver and DisplayDriver base classes into a DisplayMirror class, so the call sites move with it. Mostly renames. The one behavioural change is where the mirror is armed: DisplayMirror::start() runs after DeviceScreen::init() instead of InputDriver::enableInjection() running before it, because device-ui no longer needs an ordering flag to decide whether to build its virtual input devices. All of this sits inside HAS_MUI_MIRROR, which no committed configuration defines, so no build changes.
device-ui now renders straight into RGB565_SWAPPED and hands direct-mode rects inside the whole frame, so the observer reports a stride and DisplayMirror::pixelsByteSwapped(). The copy honours both, keeping the wire little-endian RGB565 and tightly packed.
The mirror verbs are handled in PhoneAPI, beside lockdown_auth, so each connection holds its own subscription or one-frame request. Frames go only to the clients that asked, ScreenMirror captures while any are waiting and frees its buffers when none are, and AdminModule refuses the verbs from a remote node with NOT_AUTHORIZED instead of a silent success. A request now kicks the screen thread and captures the last committed frame while the panel is off (not while lockdown redacts it). TFTDisplay hands the palette its defaults in panel byte order like HUB75, and the theme defaults are part of the palette signature so a recolor alone produces a new frame. has_display follows HAS_SCREEN_MIRROR, the mirror's memory has its own memaudit tag, and the MUI pool's PSRAM accounting is symmetric. Also caps the beacon message by its array size: the protobufs pin carries the shorter broadcast_message.
A touch long-press now asks DisplayMirror to hold past the virtual pointer's own threshold instead of a fixed 600 ms, which fell short of the 700 ms the touch driver uses. SELECT_LONG, which had no MUI mapping and was dropped, becomes a long-pressed LV_KEY_ENTER.
jamesarich
force-pushed
the
screen-mirror-poc
branch
from
October 7, 2026 18:23
d79851f to
002b6b4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streams the device's screen to the local clients that ask for it, and completes the remote-control story started by
send_input_event(meshtastic/protobufs#702). Prior art: meshtastic/web#224. Draft until meshtastic/protobufs#1054 merges; the submodule then moves to protobufs master. The pin is #1054's head on develop's protobufs base, so the regenerated headers differ from develop only by the display messages. Android client: meshtastic/Meshtastic-Android#6987.How it works
graphics::ScreenMirrorsnapshots the 1bppOLEDDisplayframebuffer after each committed frame (updateUiFrame's exits, so a lockdown build mirrors the redacted LOCKED frame). Amemcmpgates capture, so a static screen streams nothing.PhoneAPI, besidelockdown_auth, because the subscription belongs to the connection.set_display_mirrorsubscribes that connection;get_display_frame_requestasks for one frame.ScreenMirrorcaptures while any connection is subscribed or waiting and frees its buffers when none is. A remote request reachesAdminModuleand getsNOT_AUTHORIZED.getFromRadioat lowest priority with a per-connection cursor, and only to connections that asked.TFTDisplay/HUB75Displayhand the mirror their region table at paint time, beforeclearTFTColorRegions(). Palettes stream asdisplay_palettechunks keyed by a signature over the regions and theme defaults; frames carry the signature they were painted with.DeviceMetadata.display(DisplayInfo) reports geometry,PanelClassand touch during the handshake, on builds with mirroring.mirror)MESHTASTIC_PHONEAPI_ACCESS_CONTROLMESHTASTIC_EXCLUDE_SCREEN_MIRROR/ no screenCost
Nothing allocated until a connection asks: snapshot (1 KB on 128×64, 9.6 KB on 320×240) plus a 576 B palette table, freed when the last one leaves.
sizeof(meshtastic_FromRadio)is unchanged (768 B on host); encodedFromRadio_sizestays 510 ≤ 512.Not in this PR
HAS_MUI_MIRROR0) until feat(graphics): DisplayMirror for screen mirroring and remote input device-ui#394 lands and the pin bumps. Known MUI-path follow-ups: pure-MUI boards (HAS_SCREEN=0) cannot enable it yet, the frame notify runs on the LVGL task, and rect-pool overflow with no consumer repaints in a loop.test_screen_mirrornative suite.🤝 Attestations
t-deck(BaseUI) env, before the per-connection reworkrak4631build +pio checkon the per-connection rework; ESP32 envs via CIFeature stack
Five repos, and the release order runs top to bottom. Nothing below can ship until the piece above it lands.
DisplayMirror: frame capture + remote inputDisplayFrame/DisplayPalette/DisplayInfo+ the two admin verbscapture_displaytoolingdevice-ui#394 is the current blocker: without
DisplayMirrorthe MUI/colour path compiles out of every committed firmware configuration, leaving only the 1bpp mono path.