slurm: make run-parts.sh exclusive detection work with custom prefix and recent Slurm

[100milliongold]

Problem

The exclusive-job detection in roles/slurm/templates/etc/slurm/shared/bin/run-parts.sh fails in two independent ways:

1. PATH. It calls scontrol/squeue through PATH. slurmd's environment does not include a custom slurm_install_prefix, so both commands return nothing, numcpus_sys and numcpus_job are both empty, "" == "" is true, and every job runs the *-exclusive-* scripts. On a shared node that resets power limits and application clocks on all GPUs and drops page caches for each job (prolog took 6–8 s; srun: Prolog hung on node).

2. Parsing. grep -Eio "TRES=cpu=[0-9]+" matches both ReqTRES= and AllocTRES= lines on recent Slurm, so numcpus_job becomes a multi-line value ('1\n56' in a bash -x trace) that never compares equal. With scontrol on PATH, exclusive jobs are therefore never detected.

Reproduced on DGX OS 7.5.0, Slurm 26.05.1, slurm_install_prefix: /raid/slurm/usr/local.

Fix

• Call {{ slurm_install_prefix }}/bin/squeue by absolute path (the script is deployed with the template module, so the variable is available).
• Read allocated CPUs and node count with squeue -o %C / -o %D instead of parsing scontrol show job.
• Guard against an empty result so a lookup failure means "not exclusive".

Verification

On the system above, before the fix every srun --gres=gpu:1 job logged Running .../50-exclusive-gpu; after symlinking the binaries onto PATH (which exercises failure 2) a bash -x run showed numcpus_job='1\n56'. The patched logic yields numcpus_job=56, numcpus_sys=256, exclusive=0 for that job.

[dholt]

run-parts.sh has set -e, so the direct numcpus_job=$(squeue ...) and numnodes_job=$(squeue ...) assignments terminate the script when squeue returns nonzero; redirecting stderr does not suppress that exit status. Please perform both lookups inside a conditional or otherwise explicitly handle failure so exclusive remains 0 and normal non-exclusive scripts still run. Add the required decision-matrix proof for failed squeue as well as empty output.

---

Automated triage review (agent-generated on the maintainer's behalf; a human maintainer decides merges).

[100milliongold]

Thanks — the set -e point is correct and I reproduced it before changing anything:

$ bash -c 'set -e; v=$(false); echo "reached"' ; echo "rc=$?"
rc=1

The echo never runs, so a transient squeue failure took down the whole prolog/epilog run rather than just the exclusive detection.

What changed (commit ebb66b8):

• Both allocation fields now come from a single -o "%C %D" call placed inside an if condition, so the failure is visible and handled. A failed, empty or non-numeric lookup leaves exclusive=0 and logs a warning.
• The same reasoning is applied to the last-user-job count, which you did not ask about but has the same shape with a worse failure direction: piping squeue into wc -l hides its exit status, and a failed lookup would be read as "no other jobs" and run the *-lastuserjob-* cleanup scripts while another job of the same user is still on the node. A failed lookup now leaves last_user_job=0.

Decision matrix, verified on a DGX B300 (Ubuntu 24.04, bash 5.2, 256 CPUs) with a squeue stub that honours the -o format and the -j flag:

case	before	after
exclusive	rc=0, all three parts ran	rc=0, all three parts ran
shared	rc=0, exclusive part skipped	rc=0, exclusive part skipped
other jobs present	rc=0, lastuserjob part skipped	rc=0, lastuserjob part skipped
empty output	rc=0, silently non-exclusive	rc=0, non-exclusive + 1 warning
non-numeric output	rc=0, silently non-exclusive	rc=0, non-exclusive + 1 warning
nonzero exit	rc=1, no part ran at all	rc=0, normal part ran + 2 warnings

The three normal cases are unchanged, so there is no regression; the failure cases stop taking the whole run down and stop deciding silently.

One note on the test itself: my first stub ignored -o and returned both fields to every query, which does not exercise the old code fairly (it asks for %C and %D separately). The table above is from the corrected stub.