Skip to content

feat: add HarmonyOS support - #1683

Merged
thymikee merged 27 commits into
callstack:mainfrom
vwww-droid:feat/harmonyos-support
Aug 9, 2026
Merged

feat: add HarmonyOS support#1683
thymikee merged 27 commits into
callstack:mainfrom
vwww-droid:feat/harmonyos-support

Conversation

@vwww-droid

Copy link
Copy Markdown
Contributor

Summary

Adds HDC-backed HarmonyOS support to agent-device for both physical devices and DevEco emulators.

  • Adds HarmonyOS discovery, doctor readiness reporting, capabilities, app lifecycle, installation/reinstallation, snapshots, screenshots, selector-driven interaction, text input, keyboard handling, logs, performance sampling, and app-state clearing.
  • Adds native device MP4 recording for physical HarmonyOS hardware, with an explicit unsupported response on emulators rather than an Android or desktop fallback.
  • Keeps unsupported HarmonyOS operations explicit and documents the local-hardware validation policy because GitHub CI has no HarmonyOS/DevEco environment.

This supersedes #679. It is rebased on current main, uses the current platform-plugin and typed-node architecture, and intentionally does not claim the unsupported alert, notification push, or orientation-control behaviors from that earlier implementation.

Validation

  • pnpm check:affected --run
  • Manual UAT on a physical Huawei Mate 60: discovery, lifecycle, snapshots, selector/ref interaction, text input, screenshot, logs, performance diagnostics, install/reinstall, app-state clearing, and native MP4 recording.
  • Manual UAT on the HarmonyOS emulator: the same supported workflow plus the explicit simulator-recording unsupported result.
  • Recorded macOS demonstrations were captured for both physical-device and emulator flows. Compressed H.264 MP4 attachments will be added in a follow-up comment.

Limitations

  • Native HarmonyOS recording requires physical hardware; the emulator returns an explicit unsupported result.
  • CI validates contracts and mocked HDC request/response behavior only. Real HarmonyOS behavior is validated locally against connected hardware and the DevEco emulator.

Scope: 151 files changed relative to current main.

@vwww-droid

Copy link
Copy Markdown
Contributor Author

Real-device validation recordings:

  • Physical Huawei Mate 60: device discovery, lifecycle, snapshot-driven interactions, text input, screenshot, logs, diagnostics, install/state management, and native HarmonyOS MP4 recording.
agent-device-harmonyos-uat-720p.mp4
  • HarmonyOS emulator: the supported workflow plus the explicit unsupported response for native device recording.
agent-device-harmonyos-emulator-uat-720p.mp4

Both recordings use the fixture application and the PR branch shown above.

@thymikee

thymikee commented Aug 8, 2026

Copy link
Copy Markdown
Member

Reviewed exact head 8a98bec. Two P2s and one branch blocker remain: (1) listHarmonyApps accepts the resolved user-installed | all filter but ignores it and always runs bm dump -a; the default apps contract/help promises user-installed-only, so HarmonyOS can return system/OEM apps while the CLI labels them as user apps. Implement the filter or explicitly narrow the capability/contract, and add route-level coverage for both modes. (2) website/docs/docs/commands.md says recording is not advertised on HarmonyOS, while capabilities and the backend advertise/implement physical-device record; document physical-only whole-screen recording and emulator rejection. (3) the branch is now three commits behind current main across overlapping core/contracts/test paths; rebase, preserve these fixes, and rerun validation before readiness. The attached physical Mate 60 and emulator videos are useful and broadly cover the claimed UAT. #679 is a stale conflicting duplicate superseded by this PR and is suitable for maintainer closure once this replacement is accepted.

@vwww-droid
vwww-droid force-pushed the feat/harmonyos-support branch from 8a98bec to 282241e Compare August 8, 2026 08:41
@vwww-droid

vwww-droid commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

@thymikee Thanks — addressing your review at #1683 (comment). All three items are resolved in final head 739773c3c.

  1. HarmonyOS apps now honors both filters. The default user-installed path enumerates bundles, then reads each bundle’s structured applicationInfo.isSystemApp metadata and includes only non-system bundles. --all retains the complete bm dump -a inventory. If Bundle Manager does not expose a classification, the default path fails explicitly with a --all recovery hint rather than mislabeling an unknown bundle. Platform tests and daemon inventory-route coverage exercise both modes.
  2. Updated the command documentation to describe physical-device-only, whole-screen HarmonyOS recording (--scope device or --scope system) and the explicit emulator rejection, including unsupported recording flags.
  3. Rebased the branch onto the current main base 9c25bc66f before applying these fixes.

Validation: pnpm check:affected --run completed successfully after the final rebase and fixes. The previously attached physical-device and emulator recordings remain representative of the supported behavior.

@vwww-droid
vwww-droid force-pushed the feat/harmonyos-support branch from 282241e to 739773c Compare August 8, 2026 08:44
@thymikee

thymikee commented Aug 8, 2026

Copy link
Copy Markdown
Member

Re-reviewed 739773c: both code findings are fixed. Default apps now classifies bundles through Bundle Manager metadata, fails closed when classification is unavailable, and route-level tests distinguish user-installed from all; docs now accurately describe physical-only recording and emulator rejection. The branch is rebased on current main and mergeable. One readiness blocker remains: attached Harmony device/emulator evidence predates this filter commit and all new coverage is mocked. Please attach exact-head physical-device or DevEco output showing default apps excludes a known system bundle while apps --all includes it, including elapsed time (the default path performs one metadata read per bundle). Then this can be marked ready.

@vwww-droid

Copy link
Copy Markdown
Contributor Author

@thymikee Replying to #1683 (comment) with exact-head DevEco emulator evidence.

  • Head: 739773c3c
  • Target: HarmonyOS DevEco emulator 127.0.0.1:5555
$ /usr/bin/time -p ./agent-device apps --platform harmonyos --serial 127.0.0.1:5555 --json
{
  "apps": [
    "application (com.example.application)",
    "resources (com.ohos.backgroundtaskmgr.resources)",
    "formrenderservice (com.ohos.formrenderservice)"
  ]
}
real 16.17

$ /usr/bin/time -p ./agent-device apps --platform harmonyos --serial 127.0.0.1:5555 --all --json
...
"settings (com.huawei.hmos.settings)"
...
real 0.20

com.huawei.hmos.settings is present in the complete --all inventory and absent from the default user-installed result. The default elapsed time includes one Bundle Manager metadata read for each discovered bundle, as expected.

@thymikee

thymikee commented Aug 8, 2026

Copy link
Copy Markdown
Member

P1: the exact-head evidence closes filter correctness but demonstrates the implementation is not bounded enough to ship. Default HarmonyOS apps took 16.17s versus 0.20s for --all because listHarmonyUserInstalledPackages launches one sequential hdc bm dump -n per discovered bundle, each independently allowed 15s, with no aggregate budget/cancellation. On a larger or partially stalled inventory this scales to N×15s and can hit the public apps 90s envelope, whose timeout policy resets the daemon and active sessions. Replace the serial N+1 path with bulk metadata if available, or a bounded/cancelable aggregate strategy; add a regression proving the aggregate bound/no pile-up, then repeat exact-head device timing. The evidence is valuable, but it reveals rather than closes this blocker.

@vwww-droid

vwww-droid commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

@thymikee Addressed the P1 from #1683 (comment) in exact head e800c15617ad8ffa4da84d111c71bb7ca0ec9baa.

The default HarmonyOS inventory now has one 60 s aggregate deadline covering bm dump -a and classification, leaving 30 s below the public 90 s apps envelope. Metadata reads use a four-worker pool, rather than serial N+1 execution. The daemon request cancellation signal is passed through to every HDC child; a worker failure aborts in-flight siblings, waits for them to settle, and prevents any queued bundle from launching.

Regression coverage uses a six-bundle controlled stall: it proves only four metadata requests launch, then verifies the first classification failure aborts all three in-flight siblings and starts none of the remaining two queued bundles. pnpm check:affected --run passed.

Exact-head emulator evidence (127.0.0.1:5555, DevEco HarmonyOS emulator):

$ git rev-parse --short HEAD
e800c1561

$ /usr/bin/time -p ./agent-device apps --platform harmonyos --serial 127.0.0.1:5555 --json
... "application (com.example.application)" ...
real 4.31
user 0.08
sys 0.02

$ /usr/bin/time -p ./agent-device apps --platform harmonyos --serial 127.0.0.1:5555 --all --json
... "settings (com.huawei.hmos.settings)" ...
real 0.22
user 0.08
sys 0.02

The default result does not include com.huawei.hmos.settings; --all does. The previous serial run was 16.17 s on this same emulator.

Follow-up research after comparing the Android pm list packages -3 model:

  • bm dump -g is a fast single-command list, but it means debug bundles, not user-installed bundles. On this emulator it returns only the signed debug fixture com.example.application; it would omit a release-signed third-party app, so using it would weaken the documented default contract.
  • bm dump -a -l is also bulk, but its JSON contains only bundleName and label, not applicationInfo.isSystemApp.
  • bm dump -a -u 0 changes the user profile queried; it returned a different system-user inventory, not a user-installed/system classification.
  • The device rejects the older source-tree -i / --bundle-info candidates. Its current bm dump --help exposes only all, debug-bundle, bundle-name, shortcut, user/device, and label modes.
  • I also inspected the current OpenHarmony Bundle Manager tool and service source. The shell command exposes only list, single-bundle-info, debug-list, label-list, shortcut, and distributed-info dump flags. The service has an internal bulk GetBundleInfos API, but the public shell tool does not expose it as a stable HDC command or as a user-installed filter.

Therefore there is no public HarmonyOS equivalent to Android -3 on the tested current toolchain. The bounded, cancelable metadata strategy remains necessary to preserve default-list semantics without relying on bundle-name heuristics or silently dropping release apps.

@thymikee

thymikee commented Aug 8, 2026

Copy link
Copy Markdown
Member

Re-reviewed e800c156. Two P2s remain:

  1. The pool can mask the initiating classification failure as cancellation. Promise.allSettled(workers).find(...) scans worker index, not completion order. If worker 4 discovers missing metadata while workers 1–3 are still in HDC, its catch aborts them; runCmd normalizes those sibling aborts to COMMAND_FAILED: request canceled, and the lower-index cancellation is selected instead of the real missing-classification error and its apps --all hint. Preserve the first initiating non-abort error before aborting siblings, drain, then rethrow it. Add the inverse-order regression: a later worker fails while earlier workers stall, and assert the original error and hint survive.

  2. The six-bundle regression genuinely proves the four-worker cap and failure-driven abort/drain, but it never drives the 60 s aggregate deadline. Removing/raising that timer, or disconnecting it from the HDC children, would still pass. Add an all-stalled fake-timer case that advances the aggregate budget, asserts the four in-flight reads are aborted, no queued read starts, drain completes, and the timeout-specific error is returned.

The exact-head emulator evidence credibly retains filtering and improves default inventory time to 4.31 s, but it does not cover these error/boundary paths.

Add HDC-backed discovery, snapshots, application lifecycle, and core mobile interactions.

Route HarmonyOS through the platform registry and client contracts.

Cover parsing and capability parity with focused tests.
Install and reinstall signed HAP archives through HDC.

Resolve bundle identities from module metadata and relaunch after package replacement.

Extend deploy routing and capability coverage for HarmonyOS.
Execute pan, fling, and swipe plans through HDC uiInput primitives.

Derive scroll coordinates from the live ArkUI viewport.

Keep unsupported multi-touch gestures explicitly rejected.
Separate session, device, capability, and app inventory response paths.

Preserve the public inventory response contract while reducing handler complexity.
Route HarmonyOS enter, return, and dismiss through HDC key events.

Expose supported keyboard actions through the system command metadata.

Keep keyboard visibility inspection explicitly unsupported.
Keep drag unavailable until HDC can preserve source and destination hold semantics.
1. Stream HarmonyOS app logs through PID-scoped hilog sessions.\n2. Record HarmonyOS app identity during bundle-id opens for app-scoped commands.\n3. Cover backend routing and bundle identity resolution.
1. Read the foreground HarmonyOS mission through aa dump.\n2. Expose HarmonyOS appstate with package and ability metadata.\n3. Add parser coverage for foreground and missing-state cases.
1. Classify appstate in the command descriptor capability matrix.\n2. Surface supported appstate commands in capability inventory.\n3. Cover the advertised Android capability contract.
1. Sample HarmonyOS process CPU and resident memory through HDC.\n2. Expose the verified metrics through the shared perf command.\n3. Keep frame and memory snapshot collection explicitly unavailable.
1. Add HarmonyOS settings clear-app-state through bundle cleanup.\n2. Force stop the app before clearing data and cache.\n3. Reject all unverified HarmonyOS settings explicitly.
1. Describe HarmonyOS HDC prerequisites and HAP installation.\n2. Add HarmonyOS to platform discovery and product documentation.\n3. Document verified performance limits for the public HDC surface.
1. Bind a resolved HarmonyOS bundle after install or reinstall.\n2. Keep app-scoped logs and observability available after deployment.\n3. Cover session identity preservation for HarmonyOS reinstall.
1. Add an independent HarmonyOS capability-matrix oracle and exact advertised-command regression test.
2. Document current HDC-backed support and evidence-based unsupported command boundaries.
1. Split device selection and settings dispatch into focused helpers without changing behavior.
2. Keep HarmonyOS serial selection and lock-policy classification covered by regression tests.
3. Remove Fallow complexity findings from the HarmonyOS diff against upstream main.
1. Apply a 15 second timeout to ordinary HDC operations.
2. Preserve operation-specific timeout budgets for installation and capture paths.
3. Add regression coverage for default and overridden HDC timeouts.
Implement physical-device whole-screen recording through the system recorder and HDC media transfer.

Reject unsupported HarmonyOS recording scopes and export flags.

Cover capability routing, media retrieval, cleanup, and simulator rejection.
Add an HDC version check to the HarmonyOS doctor flow.

Document HarmonyOS as a supported doctor platform and cover the result.
Reduce recording validation and test complexity without changing behavior.
Synchronize public platform expectations across CLI, MCP, replay, and inventory tests.

Mock HarmonyOS inventory probes to preserve concurrent test behavior.
Require a physical HarmonyOS device in the independent capability parity oracle.
Exercise HDC input, lifecycle, installation, and relaunch command sequences.
Exercise discovery, screenshot validation, and process performance sampling.
Keep HDC hardware validation local and require mocked CI contract tests.
1. 限制应用元数据分类并发并为默认清单设置整体时限.
2. 将请求取消信号传递给 HarmonyOS 应用清单读取.
3. 补充失败时中止在飞读取且不继续排队的回归测试.
1. 保留触发应用元数据分类失败的原始错误, 避免被取消同级任务覆盖.
2. 补充总时限中止在飞读取且不启动排队任务的回归测试.
3. 验证后序任务失败时保留默认筛选的恢复提示.
@vwww-droid
vwww-droid force-pushed the feat/harmonyos-support branch from e800c15 to 46a2353 Compare August 8, 2026 15:14
@vwww-droid

Copy link
Copy Markdown
Contributor Author

@thymikee Addressed both P2s from #1683 (comment) in rebased head 46a2353a2c9d4d769d0397a566713939aba9a69d (base 588a41952).

  1. Metadata workers now retain the first initiating failure before aborting siblings, drain all workers, then rethrow that saved failure. A regression makes worker 4 discover missing metadata while workers 1–3 stall and are canceled; it asserts the original missing-classification error and its apps --all hint survive.

  2. Added an all-stalled fake-timer regression. It advances the aggregate 60 s budget, verifies exactly four in-flight metadata reads are aborted, verifies neither queued read starts, drains the workers, and asserts the timeout-specific error and recovery hint.

Validation: focused HarmonyOS lifecycle and inventory-route tests pass, the previously failing request-save-script-transports suite passes locally, and pnpm check:affected --run passes after the rebase. The prior CI Coverage failure was an unrelated temporary-directory cleanup race in that suite; the iOS Smoke failure was the unrelated existing RunnerTests.testHungCustomActionReadIsContainedAndRecovers. The force-push has started a fresh CI run.

@thymikee

thymikee commented Aug 8, 2026

Copy link
Copy Markdown
Member

Re-reviewed exact head 46a2353: both P2s are fixed. The worker pool now records the initiating non-abort classification failure before aborting siblings, drains all workers, then returns that preserved error and its recovery hint; the inverse-order regression confirms a later worker failure is not masked by lower-index sibling cancellation. The new all-stalled fake-timer regression drives the 60 s aggregate deadline, verifies all four in-flight metadata reads are aborted, no queued bundle starts, drain completes, and the timeout-specific recovery response is returned.

The branch is rebased on current main and is mergeable. The prior exact-head DevEco evidence remains probative because the successful filtering and bounded-success route are unchanged. Code review is clean and ready for human review.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Aug 8, 2026

@thymikee thymikee left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's give it a shot! Sorry for lagging for so long. @vwww-droid can I count on you for fixing bugs and implement new features for that platform?

@thymikee
thymikee merged commit 4b432fb into callstack:main Aug 9, 2026
27 checks passed
@vwww-droid

Copy link
Copy Markdown
Contributor Author

@thymikee Absolutely — I would be very happy to continue maintaining and improving the HarmonyOS support: fixing bugs, implementing new platform features, and keeping physical-device and emulator verification healthy as the ecosystem evolves.

Please feel free to reach me here on GitHub or by email at vwvw2025@outlook.com. Thanks for the trust!

@thymikee

thymikee commented Aug 9, 2026

Copy link
Copy Markdown
Member

Thanks! I work on platform-daemon decoupling now at #1696 so maintaining platform details should get easier soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants