Frontier CI: build retry deletes build/venv, reinstalls from a compute node, and records a false cluster-wide PyPI outage

[sbryngelson]

Frontier compute nodes have no internet, and the build runs inside a SLURM job. The venv is fetched on the login node first (Fetch Dependencies, --deps-only) so the compute-node build never has to reach PyPI. The retry path breaks that contract.

.github/scripts/retry-build.sh reacts to a failed first attempt with rm -rf build, which deletes build/venv. Attempt 2 then runs ./mfc.sh ... on the compute node, sees no venv, and tries to uv install from PyPI, which cannot succeed there:

+ echo '  Build failed — nuking build directory before retry...'
+ rm -rf build
+ ./mfc.sh test -v --dry-run -a -j 8 --gpu acc
mfc: OK > Created a Python virtual environment (venv).
mfc: (venv) Using uv for fast installation...
  ├─▶ Failed to fetch: `https://pypi.org/simple/hatchling/`
  ├─▶ error sending request for url (https://pypi.org/simple/hatchling/)
mfc: ERROR > (venv) Installation failed.
+ bash .github/scripts/classify-build-failure.sh build-build-gpu-acc.log frontier

classify-build-failure.sh matches that Failed to fetch ... pypi line and records a cluster-wide outage (ci-outage.sh mark frontier "PyPI/uv dependency install failed during build"). For the next 20 minutes every Frontier job on every PR exits 78 without submitting anything, and each of them re-reads the marker rather than the code.

Consequences seen today (2026-09-03):

• A compile error on one branch (job 100781459427, a private function referenced from a probe kernel) became a "PyPI outage": six Frontier lanes on State-dependent equations of state: Mie-Gruneisen (linear and cubic Hugoniot), JWL and Vinet, with temperature, per-phase evaluation on the 5- and 6-equation models #1811 (run 33787766585), the AMD lanes on Add a Mie-Gruneisen equation-of-state backend #1805 (run 33783140779) and the rerun of toolchain: precheck passes under the pre-commit hook from a worktree #1807 (run 33704696860, attempt 2) all exited 78 with zero tests run.
• The real failure is hidden: the job that had the compile error is reported as a dependency outage, and the compile diagnostics are only in the attempt-1 part of its log.
• The marker is refreshed by every later job that hits the same path, so one bad build on a busy day keeps Frontier CI dark well past 20 minutes.

Two fixes, either is enough:

1. retry-build.sh: keep build/venv when nuking (remove build/staging, build/install, and the CMake cache instead), so attempt 2 never needs PyPI. On Frontier the venv was fetched on the login node and is still valid.
2. classify-build-failure.sh: only mark an outage when the PyPI failure happened on attempt 1 (or on a login node). A fetch failure on a compute node is expected and says nothing about the cluster.

Not related to the code under test; MFC itself is unchanged by the retry.

[sbryngelson]

Scan of today's Frontier jobs: the breaker was tripped twice by this path, at 18:01 UTC (branch ci/gpu-fault-diagnostics, attempt 1 failed with "Failed to install the syscheck target ... operation timed out") and at 19:14 UTC (probe/cce-acc, attempt 1 failed on a compile error). Every other Frontier failure today is a job exiting 78 on the marker with zero tests run.

[sbryngelson]

The fix now lives in #1800 rather than #1814, which is closed — its two commits were cherry-picked because the two changes interact.

Both halves of this are addressed there:

• The cause: retry_build no longer removes build/venv, so attempt 2 does not try to reinstall from a compute node that has no route to PyPI.
• The false outage: the cluster-wide breaker is removed entirely. Frontier fetches dependencies on the login node, before any allocation is committed, so a failed download costs no allocation and there is nothing to protect the matrix from. Every recorded instance of the breaker firing was a false positive — a syscheck install timeout, a compile error, and this bug — and it reddened Add a Mie-Gruneisen equation-of-state backend #1805, toolchain: precheck passes under the pre-commit hook from a worktree #1807 and State-dependent equations of state: Mie-Gruneisen (linear and cubic Hugoniot), JWL and Vinet, with temperature, per-phase evaluation on the 5- and 6-equation models #1811 for twenty minutes at a time, including jobs whose tests had already passed.

Leaving this open until #1800 lands.