Spare localusers from the epilog's killall

[100milliongold]

What happens

epilog.d/41-lastuserjob-ssh checks /etc/slurm/localusers.backup before it touches a user:

if grep -q -w "$SLURM_JOB_USER" /etc/slurm/localusers.backup ; then
    exit 0  # don't revoke access for these users
fi

epilog.d/40-lastuserjob-processes does not, even though it is the broader of the two:

if [ "$SLURM_JOB_USER" != root ]; then
    if killall -9 -u "$SLURM_JOB_USER" ; then

killall -9 -u reaps every process the user owns on that node. That includes an interactive login shell and the sshd session carrying it.

So an operator listed in localusers.backup — listed there precisely so their access is not cut off — is disconnected the moment their job's epilog runs.

How we hit it

Two-node cluster where the controller is also a compute node, so the account submitting jobs also has a login shell on a node that runs the epilog.

11:09:55  slurm-epilog[36623]: START user=ubuntu job=2
11:09:55  Running /etc/slurm/epilog.d/40-lastuserjob-processes ...
11:09:55  slurm-epilog[36633]: Killed residual user processes
11:09:55  sshd[3167]: pam_unix(sshd:session): session closed for user ubuntu
11:09:55  Running /etc/slurm/epilog.d/41-lastuserjob-ssh ...
11:10:03  epilog for JobId=2 ran for 8 seconds

and on the operator's terminal, in that same second:

Connection to <node> closed by remote host.

That sshd pid is a long-lived login session, not one of the short-lived connections the job itself opened.

There is a second symptom nearby. Node Health Check runs check_ps_service -u root -d sshd sshd and fails on these nodes:

error: health_check failed: rc:1 output:ERROR: nhc: Health check failed:
       check_ps_service: Service sshd (process sshd) owned by root not running

It fails every interval, not only after a job, so it is a separate problem and is not addressed here. We have not established why yet: on these nodes both ssh.socket and ssh.service report active, so the simple "socket activation leaves no daemon" explanation does not hold. Mentioned only so it is not mistaken for a consequence of this change.

The change

Apply the guard 41 already uses, unchanged in form.

Cleanup behaviour is identical for every user not in localusers.backup. And with ProctrackType=proctrack/cgroup the job's own processes are already reaped by Slurm, so this script is a sweep for strays that escaped the cgroup rather than the primary teardown — skipping it for a handful of operator accounts does not leave the node dirty.

Verified

Applied to both nodes of the cluster above:

guard fires for ubuntu (in localusers.backup)  : YES — killall skipped
guard fires for an ordinary user               : no  — killall still runs

Not changed

42-lastuserjob-cleanup has the same asymmetry — it removes the user's files under /tmp and /dev/shm with no exemption check. Its blast radius is much smaller than killall -9, so it is left alone here rather than widened into this PR. Happy to follow up if you would rather the three scripts agree.

[dholt]

Protecting local operator sessions is a useful improvement. The new guard needs two corrections:

• The backup is created under slurm_config_dir, but this check hardcodes /etc/slurm/localusers.backup. Use the configured path.
• A missing/unreadable file or other grep error currently falls through to killall, just like a definite nonmatch. Treat lookup errors as unknown and do not authorize destructive cleanup. Use exact literal account-line matching rather than word/regex matching.

Please cover the default/custom path, match, definite nonmatch and lookup-error cases. The broader job-ownership cleanup issue is tracked separately in #1407; this is not a request to expand this PR into that redesign.

[100milliongold]

Both corrections are in 9e69e1e, with the decision matrix below.

1. Configured path

The backup is installed at {{ slurm_config_dir }}/localusers.backup (roles/slurm/tasks/compute.yml:156), and the epilog parts are rendered with template: (roles/slurm/tasks/prolog_epilog.yml:92), so the variable is available here. The guard now reads:

localusers_backup='{{ slurm_config_dir }}/localusers.backup'

2. Lookup error is no longer a licence to kill

grep exits 1 for a definite nonmatch and >1 when the lookup itself failed. Both used to land in the same else branch:

lookup_rc=0
grep -qxF -- "$SLURM_JOB_USER" "$localusers_backup" || lookup_rc=$?

case "$lookup_rc" in
    0) exit 0 ;;                  # listed
    1) : ;;                       # definitely not listed -> cleanup
    *) logger -s -t slurm-epilog \
           "localusers lookup failed (grep rc=${lookup_rc}, file=${localusers_backup}); skipping process cleanup"
       exit 0 ;;
esac

|| lookup_rc=$? keeps the non-zero status from tripping the script's set -e.

3. Literal whole-line matching

grep -w treats the pattern as a regex, and real deployments use accounts with dots. This is not hypothetical -- the list on our DGX B300 contains s.shin, j.baek, je.kim.

Decision matrix

Produced by rendering the actual template (sed substituting slurm_config_dir, 0 Jinja expressions left) and running it with killall and logger replaced by recording stubs. SLURM_JOB_USER is the job's user; the list holds root, je.kim, s.shin unless noted.

case	slurm_config_dir	user	exit	killall called
listed, default path	/etc/slurm	je.kim	0	no
listed, custom path	/opt/slurm	je.kim	0	no
definite nonmatch	/etc/slurm	alice	0	yes (intended)
lookup failed: file absent	/etc/slurm	alice	0	no, warning logged
lookup failed: unreadable (mode 000)	/etc/slurm	alice	0	no, warning logged

Same five cases against the pre-fix script, for contrast:

case	user	exit	killall called
listed	je.kim	0	no
definite nonmatch	alice	0	yes
lookup failed: file absent	alice	0	yes
lookup failed: unreadable	alice	0	yes

Regex wildcard

List contains root and jeXkim; the job user is je.kim, who is not listed and should be cleaned up.

	exit	killall called
before, grep -w	0	no -- wrongly exempted
after, grep -xF	0	yes -- correct

The custom-path row also exercises the first correction: under the old hardcoded path that case read a nonexistent /etc/slurm/localusers.backup, which the old control flow turned into "not listed" and a kill.

[dholt]

The revision addresses the configured backup path, exact account matching, and the distinction between a definite nonmatch and a lookup error. The reported default/custom-path and error cases cover the requested behavior, and current CI is green.

Approved for this scoped fix. The broader cleanup-ownership follow-up remains in #1407. The original commits will be retained with a merge commit.