You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Minor CUDA driver compatibility issue with updated GPU sanity check #1491
On MN5 we are currently running with CUDA driver version 12.2. After the recent update to the GPU sanity check in the software-layer scripts (see PR EESSI/software-layer-scripts#202), modules built for CUDA 12.4.0 are now failing to load due to a stricter driver version check.
Your driver CUDA version is 12.2 but the module you want to load requires CUDA 12.4.0.
This looks like a minor version compatibility scenario (12.2 vs 12.4), where execution may still be valid but is now blocked by the updated check. This affects module usability on systems where the CUDA driver cannot be upgraded in sync with module updates.
I've transferred this issue, the bot repository isn't really the right place.
A better option would've been to open a support ticket for this, see https://eessi.io/docs/support/, but this issue will do for now, no need to splinter the discussion further (since this is also being discussed in the EESSI Slack).
Limited feature set: if a feature introduced in a new CUDA toolkit spans both the toolkit and the driver, and if this is actually used in a code, this could cause a cudaErrorCallRequiresNewerDriver error. We don't know this from the EESSI side, as we don't know what codes may or may not use. My gut feeling is that these cases will be rather exceptional (and the error is fairly descriptive)
Applications that compile device code to PTX will not work on older drivers. I'm pretty sure PyTorch does this when you run torch.compile.
Requires target architecture argument to nvcc (-arch=sm_xx). I think this should be fine, we'd at least like this to happen in all builds we do (whether the build systems -actually- do this is a bit hard to guarantee of course).
So yeah, it might be an issue in some cases - but there also might be many cases where it would just work fine.
Should we consider raising an LmodWarning instead when where in what Nvidia calls the 'Minor Version Compatibilty' regime? (and maybe just link to the nvidia docs to explain what that means...)
@ocaisa agreed with that approach, and remarked that making EESSI compatibility library aware (which means we could be even less restrictive) is possible but tricky. I agree. For now, I'd say:
Minor Version Compatiblity regime = LmodWarning with link to nvidia docs
Major Version Compatibility regime = LmodError (the current one, essentially)
As a temporary workaround, one can set EESSI_OVERRIDE_GPU_CHECK to make sure this check isn't done - but I wouldn't recommend that as a general solution, as you'd also miss this check in case of major version incompatiblity.
With EESSI/software-layer-scripts#226, this has now been switched to a warning (so the module will complain once per session but will load):
[aocais00@lrdn0041 ~]$ source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash
Modules purged before initialising EESSI
Module for EESSI/2023.06 loaded successfully
EESSI has selected x86_64/intel/icelake as the compatible CPU target for EESSI/2023.06
EESSI has selected accel/nvidia/cc80 as the compatible accelerator target for EESSI/2023.06
(for debug information when loading the EESSI module, set the environment variable EESSI_MODULE_DEBUG_INIT)
{EESSI/2023.06} [aocais00@lrdn0041 ~]$ module load OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0
Lmod Warning:
Your driver CUDA version is 12.2 but the module you want to load requires CUDA 12.4.0. You will therefore be in minor version compatibility mode as described in
https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html .
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
UCX-CUDA/1.15.0-GCCcore-13.2.0-CUDA-12.4.0 /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/icelake/accel/nvidia/cc80/modules/all/UCX-CUDA/1.15.0-GCCcore-13.2.0-CUDA-12.4.0.lua
NCCL/2.20.5-GCCcore-13.2.0-CUDA-12.4.0 /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/icelake/accel/nvidia/cc80/modules/all/NCCL/2.20.5-GCCcore-13.2.0-CUDA-12.4.0.lua
OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0 /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/icelake/accel/nvidia/cc80/modules/all/OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0.lua
{EESSI/2023.06} [aocais00@lrdn0041 ~]$ module load OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0
{EESSI/2023.06} [aocais00@lrdn0041 ~]$
The solution will only land at BSC after the next sync with EESSI (I don't know what the sync frequency is there, but I guess you can ask).
Activity
boegel commentedon Apr 29, 2026
I've transferred this issue, the bot repository isn't really the right place.
A better option would've been to open a support ticket for this, see https://eessi.io/docs/support/, but this issue will do for now, no need to splinter the discussion further (since this is also being discussed in the EESSI Slack).
@ocaisa Is this something you can follow up on?
casparvl commentedon Apr 29, 2026
Let me just paste here what I mentioned on chat:
@ocaisa agreed with that approach, and remarked that making EESSI compatibility library aware (which means we could be even less restrictive) is possible but tricky. I agree. For now, I'd say:
As a temporary workaround, one can set
EESSI_OVERRIDE_GPU_CHECKto make sure this check isn't done - but I wouldn't recommend that as a general solution, as you'd also miss this check in case of major version incompatiblity.ocaisa commentedon May 13, 2026
With EESSI/software-layer-scripts#226, this has now been switched to a warning (so the module will complain once per session but will load):
The solution will only land at BSC after the next sync with EESSI (I don't know what the sync frequency is there, but I guess you can ask).