fix(nhc): carry the sshd process-title colon to all Ubuntu releases

[100milliongold]

Problem

On Ubuntu 24.04, NHC reports sshd as down while sshd is running normally. HealthCheckInterval is 300 s by default, so it is one failure every five minutes:

[2026-09-09T11:07:30] error: health_check failed: rc:1 output:ERROR:  nhc:  Health check failed:  check_ps_service:  Service sshd (process sshd) owned by root not running
[2026-09-09T11:12:30] error: health_check failed: rc:1 output:ERROR: ...
[2026-09-09T11:17:30] error: health_check failed: rc:1 output:ERROR: ...

Cause

OpenSSH rewrites its process title, so every sshd carries argv[0] of sshd: -- with a trailing colon -- whether it is the listener, the privsep parent or a session:

  PID USER     COMMAND
 2967 root     sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
39859 root     sshd: ubuntu [priv]
39914 ubuntu   sshd: ubuntu@notty

check_ps_service compares the daemon name against that first token (lbnl_ps.nhc: MATCH="*$DAEMON", then ARGS=( ${PS_ARGS[$THIS_PID]} )), so the name it is given has to carry the colon too.

The role already knows this:

roles/nhc/vars/ubuntu-20.04.yml:  nhc_ssh_daemon: "sshd:"
roles/nhc/vars/ubuntu-22.04.yml:  nhc_ssh_daemon: "sshd:"
roles/nhc/vars/main.yml:          nhc_ssh_daemon: "sshd"

There is no ubuntu-24.04.yml, so with_first_found in roles/nhc/tasks/main.yml falls through to ubuntu.yml -- which sets nhc_build_deps but not nhc_ssh_daemon. The role default "sshd" applies, nothing matches, and the check fails.

slurmd on the next line is the control: it does not rewrite its title, its argv[0] is the plain path /usr/local/sbin/slurmd, and that check passes on the same node in the same run.

Fix

Set nhc_ssh_daemon: "sshd:" in ubuntu.yml rather than adding an ubuntu-24.04.yml, so the next Ubuntu release does not regress the same way. The 20.04 and 22.04 files keep their own copies and are unaffected -- with_first_found stops at the more specific file.

Scope is Ubuntu only. I have no RHEL node to measure on, so redhat.yml and the main.yml default are left alone.

Verification

Two Ubuntu 24.04 nodes, OpenSSH_9.6p1 Ubuntu-3ubuntu13.19, Slurm HealthCheckProgram=/usr/sbin/nhc, HealthCheckInterval=300.

Before -- consecutive periodic failures, 5 minutes apart:

dgx-vm1   22 failures, 09:41:23 through 11:27:30
dgx-vm2   21 failures, 09:43:52 through 11:27:30

With the rendered line carrying the colon ( <host> || check_ps_service -u root -d sshd: sshd):

dgx-vm1   nhc exit 0, no output
dgx-vm2   nhc exit 0, no output

Control, same nodes, colonless name restored:

nhc exit 1
ERROR:  nhc:  Health check failed:  check_ps_service:  Service sshd (process sshd) owned by root not running

No periodic failure after the change, and neither node was left drained.

🤖 Generated with Claude Code