Repository navigation
Description
CAO-dispatched review-mode workers failed at least 25 times between 11:59 and 14:59 UTC on 2026-11-15, and CAO evaluation workers cannot read the evidence they need to report. Sampled failures passed the agent and safe-output jobs and failed only at the failure-reporter conclusion step, which matches report_incomplete. The 24-hour window could only be partly enumerated, so counts are minimums.
Action: A maintainer who owns CAO policy and agentic-workflow access must decide whether CAO evaluation workers receive approved-integrity job-log, usage, and health read access or must use a documented fallback; acceptance is that the next CAO reliability run either reads job-level evidence for the full 24-hour window or names the approved fallback in its report, and the conclusion-step signature stays tracked in #20797.
Blocking clusters (visible):
- Blocked evidence for CAO evaluation workers: 4 incomplete or missing-data reports in the window (efficiency twice, catalog advisor, reliability). Impact: no reliability, efficiency, or catalog recommendation was produced.
- Review-mode conclusion-step failures: at least 25 failed runs in the 3-hour slice across CRA, Dependabot, CAO integrity, efficiency, and catalog advisor, UK AI, and AW compiler security. Impact: no safe output, only failure notices. This overlaps
#20797, so it is not filed separately.
Window and coverage
- Window: 2026-11-14 15:02 UTC to 2026-11-15 15:02 UTC (workflow start).
- Enumerated: the 100 most recent failed runs, 11:49 to 14:59 UTC. Earlier failures in the window were not enumerated.
- Unavailable: agentic-workflows logs, audit, and usage were denied for this role; the health query timed out and rejected a 1-day horizon; job logs are filtered by integrity policy. Failure-cause analysis therefore rests on job and step metadata.
Run-level evidence
- Sampled review-mode failures 43724056349 (CRA security requirements), 43724066176 (CRA supply chain), and 43723815563 (Dependabot planner): activation, agent, detection, and safe-output jobs succeeded; the conclusion job failed at "Handle agent failure". Root cause is not established because logs are filtered.
- Agent-declared, unverified reasons from CAO worker reports: "job-level evidence is blocked" (reliability), "health queries timed out" (catalog advisor), and "integrity policy filters job logs; usage access denied (insufficient role)" (efficiency).
Not filed
- Live SelfCare: 50 failed runs in the same slice show the same conclusion-step signature. They belong to another campaign, so they are excluded except as evidence that the signature is shared.
- Integrity worker (2026-11-14 23:08 UTC): one occurrence, a dependency-install certificate failure at a different stage.
- Main-branch CI failures (campaign lifecycle, Lighthouse, Playwright) are non-agentic and out of scope.
References: https://github.com/githubnext/gh-cao/actions/runs/43724056349, https://github.com/githubnext/gh-cao/actions/runs/43723815563, https://github.com/githubnext/gh-cao/actions/runs/43724066176
cao: githubnext/gh-cao, correlation: 43727014254-1274
Generated by CAO Evolution / Reliability · copilot · routed: haiku55 xhigh · ~51 AIC · ⌖ ~2 AIC · ⊞ 8.4K · ◷
- expires on Nov 29, 2026, 3:17 PM UTC
Activity
mnkiefer commentedon Nov 15, 2026
mnkiefer commentedon Nov 15, 2026
mnkiefer commentedon Nov 16, 2026
mnkiefer commentedon Nov 16, 2026
mnkiefer commentedon Nov 16, 2026
mnkiefer commentedon Nov 16, 2026
mnkiefer commentedon Nov 16, 2026
mnkiefer commentedon Nov 16, 2026
mnkiefer commentedon Nov 16, 2026
mnkiefer commentedon Nov 16, 2026
mnkiefer commentedon Nov 16, 2026
mnkiefer commentedon Nov 16, 2026
mnkiefer commentedon Nov 17, 2026
mnkiefer commentedon Nov 17, 2026
mnkiefer commentedon Nov 17, 2026
mnkiefer commentedon Nov 17, 2026
mnkiefer commentedon Nov 17, 2026
mnkiefer commentedon Nov 17, 2026
mnkiefer commentedon Nov 18, 2026
mnkiefer commentedon Nov 18, 2026
mnkiefer commentedon Nov 18, 2026
mnkiefer commentedon Nov 18, 2026
mnkiefer commentedon Nov 18, 2026
mnkiefer commentedon Nov 18, 2026
mnkiefer commentedon Nov 18, 2026
mnkiefer commentedon Nov 18, 2026
mnkiefer commentedon Nov 18, 2026
mnkiefer commentedon Nov 19, 2026
Control-plane reliability for 2026-11-18 06:04 to 2026-11-19 06:04 UTC has one materially changed blocking cluster. Scheduled CAO orchestrators failed again at
Run CAO control precomputeinpre_activation: three confirmed failures today at 05:33–05:48 UTC, after the 23:15 UTC update recorded no recurrence. Job logs remain integrity-filtered, so the cause is unverified.Action: A maintainer with unfiltered job-log access on the CAO control runtime reads the
Run CAO control precomputestep log for run 44467674183 and records the cause in this issue. Acceptance: the cause is recorded here, and no scheduled CAO orchestrator fails atpre_activationover the next 24 hours.Blocking clusters (visible):
Handle agent failure(unchanged, tracked in#20797): not re-filed.References: https://github.com/githubnext/gh-cao/actions/runs/44467674183, https://github.com/githubnext/gh-cao/actions/runs/44467202850, https://github.com/githubnext/gh-cao/actions/runs/44466003788
Window and coverage
control_roleworker, no candidate repositories, no activity-cache section.Stage evidence (run 44467674183, 05:48 UTC)
Evaluate Central Agentic Ops admissionsucceededRun CAO control precomputefailed about one second after starting. TheCAO precompute blockedAPI-limit branches were skipped, so this is not a recorded API-limit block. The artifact validation step was skipped.activation,agent,safe_outputs, andconclusionwere skippedWatch item, not filed
Process Safe Outputsfailed in two EU CRA workers within seven minutes at 05:24–05:31 UTC (runs 44464908661 and 44464895428). Logs are unavailable, and these are another campaign's workers, not a verified CAO routing defect. Not filed; file it if it recurs.