Skip to content

[cao-evolution:reliability] Control-plane reliability requires attention #20987

Open

Description

@mnkiefer

CAO-dispatched review-mode workers failed at least 25 times between 11:59 and 14:59 UTC on 2026-11-15, and CAO evaluation workers cannot read the evidence they need to report. Sampled failures passed the agent and safe-output jobs and failed only at the failure-reporter conclusion step, which matches report_incomplete. The 24-hour window could only be partly enumerated, so counts are minimums.

Action: A maintainer who owns CAO policy and agentic-workflow access must decide whether CAO evaluation workers receive approved-integrity job-log, usage, and health read access or must use a documented fallback; acceptance is that the next CAO reliability run either reads job-level evidence for the full 24-hour window or names the approved fallback in its report, and the conclusion-step signature stays tracked in #20797.

Blocking clusters (visible):

  • Blocked evidence for CAO evaluation workers: 4 incomplete or missing-data reports in the window (efficiency twice, catalog advisor, reliability). Impact: no reliability, efficiency, or catalog recommendation was produced.
  • Review-mode conclusion-step failures: at least 25 failed runs in the 3-hour slice across CRA, Dependabot, CAO integrity, efficiency, and catalog advisor, UK AI, and AW compiler security. Impact: no safe output, only failure notices. This overlaps #20797, so it is not filed separately.
Window and coverage
  • Window: 2026-11-14 15:02 UTC to 2026-11-15 15:02 UTC (workflow start).
  • Enumerated: the 100 most recent failed runs, 11:49 to 14:59 UTC. Earlier failures in the window were not enumerated.
  • Unavailable: agentic-workflows logs, audit, and usage were denied for this role; the health query timed out and rejected a 1-day horizon; job logs are filtered by integrity policy. Failure-cause analysis therefore rests on job and step metadata.
Run-level evidence
  • Sampled review-mode failures 43724056349 (CRA security requirements), 43724066176 (CRA supply chain), and 43723815563 (Dependabot planner): activation, agent, detection, and safe-output jobs succeeded; the conclusion job failed at "Handle agent failure". Root cause is not established because logs are filtered.
  • Agent-declared, unverified reasons from CAO worker reports: "job-level evidence is blocked" (reliability), "health queries timed out" (catalog advisor), and "integrity policy filters job logs; usage access denied (insufficient role)" (efficiency).
Not filed
  • Live SelfCare: 50 failed runs in the same slice show the same conclusion-step signature. They belong to another campaign, so they are excluded except as evidence that the signature is shared.
  • Integrity worker (2026-11-14 23:08 UTC): one occurrence, a dependency-install certificate failure at a different stage.
  • Main-branch CI failures (campaign lifecycle, Lighthouse, Playwright) are non-agentic and out of scope.

References: https://github.com/githubnext/gh-cao/actions/runs/43724056349, https://github.com/githubnext/gh-cao/actions/runs/43723815563, https://github.com/githubnext/gh-cao/actions/runs/43724066176

cao: githubnext/gh-cao, correlation: 43727014254-1274

Generated by CAO Evolution / Reliability · copilot · routed: haiku55 xhigh · ~51 AIC · ⌖ ~2 AIC · ⊞ 8.4K · ◷

  • expires on Nov 29, 2026, 3:17 PM UTC

Activity

  1. mnkiefer commented on Nov 15, 2026

    @mnkiefer
    Author
  2. mnkiefer commented on Nov 15, 2026

    @mnkiefer
    Author
  3. mnkiefer commented on Nov 16, 2026

    @mnkiefer
    Author
  4. mnkiefer commented on Nov 16, 2026

    @mnkiefer
    Author
  5. mnkiefer commented on Nov 16, 2026

    @mnkiefer
    Author
  6. mnkiefer commented on Nov 16, 2026

    @mnkiefer
    Author
  7. mnkiefer commented on Nov 16, 2026

    @mnkiefer
    Author
  8. mnkiefer commented on Nov 16, 2026

    @mnkiefer
    Author
  9. mnkiefer commented on Nov 16, 2026

    @mnkiefer
    Author
  10. mnkiefer commented on Nov 16, 2026

    @mnkiefer
    Author
  11. mnkiefer commented on Nov 16, 2026

    @mnkiefer
    Author
  12. mnkiefer commented on Nov 16, 2026

    @mnkiefer
    Author
  13. mnkiefer commented on Nov 17, 2026

    @mnkiefer
    Author
  14. mnkiefer commented on Nov 17, 2026

    @mnkiefer
    Author
  15. mnkiefer commented on Nov 17, 2026

    @mnkiefer
    Author
  16. mnkiefer commented on Nov 17, 2026

    @mnkiefer
    Author
  17. mnkiefer commented on Nov 17, 2026

    @mnkiefer
    Author
  18. mnkiefer commented on Nov 17, 2026

    @mnkiefer
    Author
  19. mnkiefer commented on Nov 18, 2026

    @mnkiefer
    Author
  20. mnkiefer commented on Nov 18, 2026

    @mnkiefer
    Author
  21. mnkiefer commented on Nov 18, 2026

    @mnkiefer
    Author
  22. mnkiefer commented on Nov 18, 2026

    @mnkiefer
    Author
  23. mnkiefer commented on Nov 18, 2026

    @mnkiefer
    Author
  24. mnkiefer commented on Nov 18, 2026

    @mnkiefer
    Author
  25. mnkiefer commented on Nov 18, 2026

    @mnkiefer
    Author
  26. mnkiefer commented on Nov 18, 2026

    @mnkiefer
    Author
  27. mnkiefer commented on Nov 18, 2026

    @mnkiefer
    Author
  28. mnkiefer commented on Nov 19, 2026

    @mnkiefer
    ContributorAuthor

    Control-plane reliability for 2026-11-18 06:04 to 2026-11-19 06:04 UTC has one materially changed blocking cluster. Scheduled CAO orchestrators failed again at Run CAO control precompute in pre_activation: three confirmed failures today at 05:33–05:48 UTC, after the 23:15 UTC update recorded no recurrence. Job logs remain integrity-filtered, so the cause is unverified.

    Action: A maintainer with unfiltered job-log access on the CAO control runtime reads the Run CAO control precompute step log for run 44467674183 and records the cause in this issue. Acceptance: the cause is recorded here, and no scheduled CAO orchestrator fails at pre_activation over the next 24 hours.

    Blocking clusters (visible):

    • Scheduled orchestrator precompute failure (recurred): 3 confirmed failures in the window (05:33, 05:44, 05:48 UTC) at the same step. The 21:15 UTC update also reported 9 failures at 18:44–19:44 UTC on 2026-11-18. Impact: those scheduled runs skipped activation, agent, and safe-output jobs, so they produced no dispatch or safe output. Counts are minimums.
    • Review-mode Handle agent failure (unchanged, tracked in #20797): not re-filed.

    References: https://github.com/githubnext/gh-cao/actions/runs/44467674183, https://github.com/githubnext/gh-cao/actions/runs/44467202850, https://github.com/githubnext/gh-cao/actions/runs/44466003788

    Window and coverage
    • Window: 2026-11-18 06:04:31 to 2026-11-19 06:04:31 UTC.
    • Health: partial. The 45-second collection budget ran out, with 18 failed workflows observed and 66 not collected. A 2026-11-18 18:30–20:00 UTC health sample also hit its budget and found 6 precompute failures, consistent with the 21:15 UTC update.
    • Unavailable: job logs (integrity-filtered), audit (role-denied), and agentic-workflows logs. Causes rest on job and step metadata.
    • Precompute: control_role worker, no candidate repositories, no activity-cache section.
    Stage evidence (run 44467674183, 05:48 UTC)
    Stage Observation Status
    Admission Evaluate Central Agentic Ops admission succeeded Passed
    Precompute Run CAO control precompute failed about one second after starting. The CAO precompute blocked API-limit branches were skipped, so this is not a recorded API-limit block. The artifact validation step was skipped. Failed
    Downstream activation, agent, safe_outputs, and conclusion were skipped Blocked
    Watch item, not filed
    • Review-mode Process Safe Outputs failed in two EU CRA workers within seven minutes at 05:24–05:31 UTC (runs 44464908661 and 44464895428). Logs are unavailable, and these are another campaign's workers, not a verified CAO routing defect. Not filed; file it if it recurs.

    cao: githubnext/gh-cao, correlation: 44468562664-1361

    Generated by CAO Evolution / Reliability · copilot · routed: haiku55 xhigh · ~39 AIC · ⌖ ~2 AIC · ⊞ 8.4K · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions