Skip to content

perf: collapse app-driving startup turns - #1693

Merged
thymikee merged 2 commits into
mainfrom
perf/bench-turn-efficiency
Aug 9, 2026
Merged

perf: collapse app-driving startup turns#1693
thymikee merged 2 commits into
mainfrom
perf/bench-turn-efficiency

Conversation

@thymikee

@thymikee thymikee commented Aug 8, 2026

Copy link
Copy Markdown
Member

Summary

  • make open <app> --foreground preserve normal app/device selection, work for both fresh and existing sessions, and return the initial interactive snapshot in one call; bare iOS auto-discovery remains fail-closed
  • make the compact skill and canonical CLI guidance start ordinary app-driving tasks directly, without mandatory help/version/session probes, with a conformance test pinning the default recipe
  • extend same-scope iOS tap corroboration from 5s to a bounded 15s so ordinary model turn latency can still prove a tap landed after XCTest reports failure

The wider corroboration window deliberately accepts some additional organic-label-drift exposure. It remains limited to XCTEST_RECORDED_FAILURE, requires the same backend and presentation, and returns an explicit re-observation warning; a false corroboration could still record a tap that did not land, so this is an accepted bounded tradeoff rather than a general success fallback.

Validation

  • pnpm check:affected --run — all runnable checks passed, including 2,559 affected Vitest tests, 100% changed-line coverage, Node integration, package, architecture/progress, and replay compatibility
  • red-before: reverting the review fixes makes the existing-session explicit-app foreground test fail with INVALID_ARGS and the skill conformance test find both foreground and plain-open recipes; both pass on this head
  • focused suite: 11 files / 245 tests passed; pnpm build passed
  • live iOS at exact head: open xyz.blueskyweb.app --foreground --platform ios --udid C4578E05-BB63-4A3D-AB53-B011EFA2D981 --session review-1693 --json returned a healthy interactive snapshot on a fresh session, then the identical command succeeded again on the existing session with a new refs generation; the session was closed afterward
  • controlled GPT-5.4 runs with isolated config/binary: Element-14 improved from 75.3s / 14 calls to 59.3s / 7 (Argent: 53.6s / 10); Element-03 improved from 174.8s / 30 to 85.4s / 12 (Argent: 122.8s / 18)
  • Bluesky reproduced the XCTest recorded-failure path and the changed post-action capture correctly corroborated the landed tap; timing was excluded because local golden verification proved that benchmark fixture was logged out

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 2.00 MB 2.00 MB -207 B
JS gzip 646.5 kB 646.4 kB -92 B
npm tarball 785.0 kB 785.2 kB +147 B
npm unpacked 2.74 MB 2.74 MB +396 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.2 ms 27.6 ms +0.4 ms
CLI --help 65.8 ms 66.3 ms +0.5 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/registry.js -117 B -72 B
dist/src/session.js -91 B -16 B
dist/src/cli-help.js 0 B -5 B
dist/src/interaction.js +1 B +1 B

@thymikee

thymikee commented Aug 8, 2026

Copy link
Copy Markdown
Member Author

[P1] Keep the compact default loop self-consistent. skills/agent-device/SKILL.md tells ordinary tasks to use open <app> --foreground only when the prompt explicitly says the app is already foregrounded; its Otherwise recipe is plain open <app>. The following guidance then tells the agent to continue from current refs, but production composes the initial snapshot only when foreground === true, so plain open <app> supplies no refs. This recreates an open -> snapshot/targetless-action turn on the common path and contradicts the canonical top-level help, which now makes open <app> --foreground the default. Make the known-app default recipe use --foreground regardless of whether the prompt states foreground state (normal open resolution still owns app/device selection), and add a skill/default conformance case that would fail if the recipe regresses to plain open without a snapshot.

@thymikee

thymikee commented Aug 9, 2026

Copy link
Copy Markdown
Member Author

Addressed in d28221adb.

  • The compact known-app recipe is now unconditionally agent-device open <app> --foreground; the contradictory plain-open branch is gone. A structural conformance test reads the shipped skill and fails if that default loses --foreground or stops promising the initial snapshot.
  • Explicit-app foreground resolution now precedes the existing-session rejection. Bare auto-discovery still fails closed with an existing session, while open <app> --foreground works uniformly as open/switch + snapshot.
  • Red-before proof: reverting those two fixes makes the existing-session test fail with INVALID_ARGS and the skill test find both foreground and plain-open recipes. The focused suite passes 245/245 on the fix.
  • Exact-head iOS proof: the fully scoped Bluesky command returned a healthy interactive snapshot, and repeating the identical command on the existing session succeeded with a new refs generation. The session was closed afterward.
  • The PR body now explicitly records the accepted 15-second corroboration drift tradeoff. The stale Haiku tuning comment and remaining RFC branding were also removed/replaced with current evidence.

The full affected gate passed before push, including package validation, 2,559 affected Vitest tests, changed-line coverage, Node integration, architecture/progress, and replay compatibility.

@thymikee

thymikee commented Aug 9, 2026

Copy link
Copy Markdown
Member Author

Re-reviewed exact head d28221adb: the compact-skill P1 and the existing-session foreground concern are fixed. The shipped skill has one known-app recipe (open <app> --foreground) with a structural conformance guard; explicit-app foreground resolution now works on an existing session while bare auto-discovery remains fail-closed. Red-before proof and exact-head live iOS fresh/repeated-open snapshot evidence are adequate, and the bounded 15-second corroboration tradeoff is documented. No actionable code finding remains; ready for human review.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Aug 9, 2026
@thymikee
thymikee merged commit 18291ba into main Aug 9, 2026
30 checks passed
@thymikee
thymikee deleted the perf/bench-turn-efficiency branch August 9, 2026 08:10
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-09 08:10 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant