pyxis: allow unprivileged user namespaces so enroot works on Ubuntu 23.10+

[100milliongold]

Problem

Ubuntu 23.10 and later set kernel.apparmor_restrict_unprivileged_userns=1 by default. Under that default, enroot-nsenter fails as soon as pyxis starts a container:

enroot-nsenter: failed to create user namespace: Permission denied

The nvidia.enroot galaxy role does not handle this, and DeepOps lists Ubuntu 24.04 LTS as a supported OS, so container jobs do not work out of the box on a supported platform.

Observed on DGX OS 7.5.0 (Ubuntu-based), Slurm 26.05.1, pyxis 0.11.1.

Fix

Add a sysctl task in roles/pyxis for compute nodes. The task is gated on the presence of /proc/sys/kernel/apparmor_restrict_unprivileged_userns rather than on ansible_distribution_version, so it is a no-op on kernels built without AppArmor userns restrictions and does not need updating for future releases.

Note on hardening

This relaxes a system-wide hardening default: unprivileged user namespaces are what AppArmor is restricting here. A narrower alternative is an AppArmor profile scoped to enroot-nsenter (as packaged by some distributions for other userns consumers). If maintainers prefer that approach, I am happy to rework the patch.

The sysctl is written to /etc/sysctl.d/60-enroot-userns.conf so it is visible and revertible, rather than being applied only at runtime.

[dholt]

Please replace the unconditional host-wide sysctl change with a narrowly scoped AppArmor policy for enroot-nsenter and provide evidence from a real Pyxis/Enroot container launch on an affected supported OS. If a global sysctl fallback is still needed for unusual installations, make it an explicit, documented, default-off administrator choice rather than changing the security default whenever the kernel knob exists.

---

Automated triage review (agent-generated on the maintainer's behalf; a human maintainer decides merges).

[100milliongold]

Agreed on all three points — clearing kernel.apparmor_restrict_unprivileged_userns host-wide removes exactly the attack surface the Ubuntu default was added to reduce, and the original patch did it unconditionally.

What changed (commit 86ecc83):

• The role now installs an AppArmor profile that grants the userns capability to /usr/bin/enroot-nsenter alone and leaves the host default in place.
• The host-wide sysctl remains available but is now an explicit, documented, default-off choice: pyxis_userns_allow_globally, false by default. If the profile fails to load, the play stops with the parser error and points at that variable rather than silently falling back to the weaker setting.
• Both paths stay a no-op where /proc/sys/kernel/apparmor_restrict_unprivileged_userns does not exist, and the profile path additionally requires apparmor_parser.

Why the profile only covers enroot-nsenter:

$ grep -l CLONE_NEWUSER /usr/bin/enroot-*
/usr/bin/enroot-nsenter

Checked against enroot 3.2.0-1, so enroot-mount, enroot-switchroot, enroot-mksquashovlfs and enroot-aufs2ovlfs do not need it.

Evidence so far, on a DGX B300 running DGX OS 7.5 (Ubuntu 24.04.4, AppArmor 4.0 ABI available):

$ apparmor_parser -Q -T -v enroot-nsenter
Addition succeeded for "enroot-nsenter".

What is still missing, and I would rather say so than leave it implied: I do not have a container launch on a host where the restriction is active. This node was already worked around out-of-band before I picked up the issue — /etc/sysctl.d/91-enroot.conf, not owned by any package, sets the knob to 0 — so it is not in the affected state. Restoring the restriction to reproduce the failure would break the Pyxis jobs currently running on it (an MLPerf Inference run occupying all 8 GPUs).

I will add the before/after launch output once the node is free. Please hold the merge until then if that evidence is required — I would rather this sit than be merged on a parser check alone.

[100milliongold]

The node is free now, so here is the container launch evidence I said I would add.

To produce it I had to put this host into the affected state first: it carries an out-of-band /etc/sysctl.d/91-enroot.conf that sets the knob to 0, so Pyxis works here for the wrong reason. I moved that file aside, set the knob back to 1, ran the launch, installed the profile, ran the same launch again, and restored the original state afterwards.

Environment

OS:              Ubuntu 24.04.4 LTS
kernel:          6.8.0-106-generic
enroot:          3.2.0-1
apparmor_parser: AppArmor parser version 4.0.1
hardware:        DGX B300, 8x SXM

Before — restriction active, no profile

$ sysctl kernel.apparmor_restrict_unprivileged_userns
kernel.apparmor_restrict_unprivileged_userns = 1

$ srun --partition=test --gres=gpu:1 --cpus-per-gpu=8 --time=5 \
      --container-image=ubuntu:24.04 -- sh -c 'echo container-started-ok'
pyxis: importing docker image ...
error: pyxis: container start failed with error code: 1
error: pyxis: printing contents of log file ...
error: pyxis:     enroot-nsenter: failed to create user namespace: Permission denied
error: pyxis: couldn't start container
error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
error: Failed to invoke spank plugin stack
srun: error: dgx01: task 0: Exited with exit code 1

exit status: 1

Install the profile

$ apparmor_parser -r /etc/apparmor.d/enroot-nsenter
$ echo $?
0

$ aa-status | grep enroot
   enroot-nsenter

After — same launch, profile loaded, restriction still active

$ sysctl kernel.apparmor_restrict_unprivileged_userns
kernel.apparmor_restrict_unprivileged_userns = 1

$ srun --partition=test --gres=gpu:1 --cpus-per-gpu=8 --time=5 \
      --container-image=ubuntu:24.04 -- sh -c 'echo container-started-ok'
srun: job 136 queued and waiting for resources
srun: job 136 has been allocated resources
pyxis: importing docker image ...
container-started-ok

exit status: 0

The knob reads 1 during both launches. That is the part I wanted on the record: the container starts with the OS security default left in place, which is what your review asked for and what the original patch did not do.

Afterwards the profile was unloaded and removed, 91-enroot.conf was put back, and the knob returned to its original value of 0, so the node is as it was.

Two notes on the run itself. The image is plain ubuntu:24.04 rather than one of our local squashfs images, so the reproduction does not depend on anything specific to this site. And pyxis_userns_allow_globally stayed at its default of false throughout — the global sysctl path was never taken.