diagnose(1874): instrument the synthesized commit wait and add a dispatchable stall loop - #1941
diagnose(1874): instrument the synthesized commit wait and add a dispatchable stall loop#1941thymikee wants to merge 7 commits into
Conversation
Size Report
npm unpacked components
Startup median (7 runs, lower is better):
Top changed chunks: no changes in the largest emitted chunks. Top changed packed files
|
|
P1 — the diagnostic instrumentation writes raw observed field contents to NSLog on the shipped synthesized-text path: observedText.prefix(40). This is not confined to the workflow or unit-test build, so ordinary type commands can persist credentials/PII in runner.log; the diagnostic workflow then uploads those logs on a stall. Please keep the cadence evidence value-free (for example observed length, expected-prefix match length, progress state, and timestamps), or gate any raw fixture-only trace behind a compile-time diagnostic build that cannot ship. Add a regression/source guard that makes reintroducing raw observed text into runner logging fail. |
A generic simulator destination leaves the active arch undefined; Xcode 26.6 defaults it to x86_64, running the whole runner under Rosetta on arm64 hosts. Pin ARCHS=arm64 across every lane that builds the iOS runner and bump the derived-data cache suffixes. Measured ~30% faster commits on identical CI hardware; delivery-throttle episodes still occur but start from a lower base.
The per-poll trace logged the observed field's contents (prefix(40)) on the shipped type path; that value is user content and runner.log persists. Log lengths and the expected-prefix walk instead, allowlist every string-interpolating NSLog format in the module behind a source-scan guard, and pin commonPrefixLength in the host-lane policy tests.
|
Addressed in 634367f. The per-poll trace now logs only |
|
P1 — The shipped poll path is value-free now, but the new guard does not enforce that invariant. It only reviews literal NSLog formats containing %@, so Swift interpolation of observedText is skipped entirely, and passing observedText to an already-allowlisted %@ format also passes. Please put cadence logging behind a typed value-free boundary that accepts only lengths, timestamps, and enums, then test its emitted output with a sentinel secret instead of partially parsing Swift format strings. Update the PR body too: item 1 still says the trace includes the observed value, which is no longer true. Finally, #1874 and #1844 are both closed while the PR says this temporary instrumentation remains until #1874 is root-caused; reopen the cleanup issue or name an active owner and removal trigger before shipping it. |
…ndary logCommitCadence accepts Int lengths and a timestamp only, so observed field contents are unrepresentable at the poll call site; its emitted line is pinned by a sentinel-secret test in the host-lane policy tests. The source guard becomes structural — boundary present, poll path logs through it, no raw NSLog in the observe closure — instead of parsing Swift format strings. #1874 is reopened as the removal-tracking thread for this temporary instrumentation.
|
Addressed in e7524f9.
|
1 participant
Refs #1874, #1844. Follow-up to #1924 (merged): production now reports
TEXT_INPUT_COMMIT_NOT_OBSERVEDinstead ofok: trueover a partial commit. This PR ships the diagnosis tooling for the remaining question — why synthesized-event delivery occasionally stalls on CI.What we know (evidence from 2026-08-18..21 failures)
32176591415typed"hardware-keyboard", held"ha"at failure time, and the field read"hardware-key"~40s later with nobody typing. The burst drains late.macos-26-arm64image; Xcode 26.6 builds x86_64 from-destination generic/platform=iOS Simulator).32485228371) passed 50/50 but caught one sub-critical episode:type-all2487ms across 5 polls — the same mechanism at recoverable severity. Episodes that outlast the 3s commit window are the rare failures (~4 in days of full-suite runs).Conclusion: transient throttling of the synthesized input pipeline on CI VMs; most episodes recover inside the commit window, the tail does not.
What this PR adds
[DEBUG-1874]instrumentation inawaitSynthesizedFirstResponderCommit: per-poll timestamp, observed length, and expected-prefix length through the real injectedobserveclosure, plus post time of the synthesize call and the final outcome — value-free by construction (typed Int-only boundary, sentinel-tested). Any future red lane now carries a cadence trace instead of a bare value mismatch, and no typed field content reaches runner.log..github/workflows/1874-diagnose.yml—workflow_dispatchmatrix (isolated vs pair-after-predecessor) looping the bare-type test N times on the CI image, with per-iteration verdicts and stall evidence artifacts.Validation
pnpm gate swift-runner-ios—** TEST BUILD SUCCEEDED **; local run prints[DEBUG-1874] poll ... outcome=settled elapsedMs=34.pnpm check:affected --run— all runnable checks passed.32479566150,32485228371); the second completed 50/50 green with evidence artifacts.Intent
Keep the workflow dispatchable until #1874's underlying stall is root-caused; when a lane goes red again, its logs will show whether the prefix walks slowly (delivery throttling) or freezes (queue wedge). Remove both once resolved.
Mitigation shipped in this PR
The 2×2 matrix (runs
32485228371,32497125475) measured native arm64 at ~30% faster commits than Rosetta x86_64 on identical CI hardware (mean type-all ~860ms vs ~1150ms), with slow-delivery episodes occurring on both architectures. Rosetta amplifies episode severity; it is not the fundamental cause. This PR pinsAGENT_DEVICE_XCUITEST_ARCHS=arm64across every lane that builds the iOS runner (ios.yml,xctest-nightly,perf-nightly,conformance-differential,replays-manual) via the new build-script override, and bumps derived-data cache suffixes.TEXT_INPUT_COMMIT_NOT_OBSERVED(#1924) remains the backstop for episodes that outlast the commit window.Cleanup ownership
This instrumentation and the dispatchable workflow are temporary. Removal trigger: #1874 (reopened — the delivery throttle is not root-caused) identifies the mechanism, at which point the
[DEBUG-1874]boundary, its guard tests, and1874-diagnose.ymlare deleted in one commit. Owner: the iOS runner maintainers; the reopened issue is the tracking thread.