fix(nhc): match sshd despite its rewritten process title

[100milliongold]

Problem

On every node this role configures, NHC reports sshd as down while sshd is running normally. With the default HealthCheckInterval, that is a failure on every interval.

ERROR:  nhc:  Health check failed:  check_ps_service:  Service sshd is not running.

Root cause

check_ps_service in lbnl_ps.nhc defaults its match pattern to *$DAEMON when -m is not given:

# lbnl_ps.nhc:405
MATCH="*$DAEMON"

and compares it against the first whitespace-separated token of the process arguments:

# lbnl_ps.nhc:192
ARGS=( ${PS_ARGS[$THIS_PID]} )

OpenSSH rewrites its process title. Every sshd -- listener, privsep parent and session alike -- therefore carries argv[0] of sshd:, with a trailing colon:

  PID USER     COMMAND
 2967 root     sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
39859 root     sshd: ubuntu [priv]
39914 ubuntu   sshd: ubuntu@notty

The glob *sshd requires the token to end with sshd. sshd: does not, so no process ever matches and the check reports the service down.

slurmd on the line immediately below is unaffected, and that is the control: it does not rewrite its title, so its argv[0] is the plain path /usr/local/sbin/slurmd, which *slurmd matches.

This is not about socket activation or a renamed binary. On the affected nodes a resident root listener exists (PPID 1), pgrep -u root -x sshd finds it, and both ssh.socket and ssh.service are active. The process title alone accounts for the failure.

Fix

Pass -m '*sshd*' so the pattern tolerates the trailing colon.

Verification

Two Ubuntu 24.04 nodes, OpenSSH_9.6p1 Ubuntu-3ubuntu13.19.

Before -- one failing check on each node:

ERROR:  nhc:  Health check failed:  check_ps_service:  Service sshd is not running.

After:

dgx-vm1   nhc exit code: 0   ERROR lines: 0
dgx-vm2   nhc exit code: 0   ERROR lines: 0

No node was left drained, and no other check changed behaviour.

🤖 Generated with Claude Code

[100milliongold]

Closing this in favour of a smaller fix that follows the pattern already in this role.

After opening this I found that the role already handles the trailing colon, per OS:

roles/nhc/vars/ubuntu-20.04.yml:  nhc_ssh_daemon: "sshd:"
roles/nhc/vars/ubuntu-22.04.yml:  nhc_ssh_daemon: "sshd:"
roles/nhc/vars/main.yml:          nhc_ssh_daemon: "sshd"

There is no ubuntu-24.04.yml, so with_first_found in roles/nhc/tasks/main.yml falls through to ubuntu.yml, which sets nhc_build_deps but not nhc_ssh_daemon. The role default "sshd" then applies and the check fails.

So the template did not need changing at all -- the variable simply never reaches Ubuntu 24.04. Widening the match pattern here would also have produced a redundant -m '*sshd:*' -d sshd: on 20.04 and 22.04.

Replaced by #1406.