Skip to content

Minor CUDA driver compatibility issue with updated GPU sanity check #1491

Closed

Description

@JonNavarroGaritano

On MN5 we are currently running with CUDA driver version 12.2. After the recent update to the GPU sanity check in the software-layer scripts (see PR EESSI/software-layer-scripts#202), modules built for CUDA 12.4.0 are now failing to load due to a stricter driver version check.

Loading modules such as:

module load LAMMPS/29Aug2024_update2-foss-2023b-kokkos-CUDA-12.4.0

results in the following error:

Your driver CUDA version is 12.2 but the module you want to load requires CUDA 12.4.0.

This looks like a minor version compatibility scenario (12.2 vs 12.4), where execution may still be valid but is now blocked by the updated check. This affects module usability on systems where the CUDA driver cannot be upgraded in sync with module updates.

Thanks for the update and for looking into this.

Activity

  1. boegel commented on Apr 29, 2026

    @boegel
    Contributor

    I've transferred this issue, the bot repository isn't really the right place.

    A better option would've been to open a support ticket for this, see https://eessi.io/docs/support/, but this issue will do for now, no need to splinter the discussion further (since this is also being discussed in the EESSI Slack).

    @ocaisa Is this something you can follow up on?

  2. casparvl commented on Apr 29, 2026

    @casparvl
    Collaborator

    Let me just paste here what I mentioned on chat:

    @Alan O'Cais (CECAM) I'm wondering if we are too strict in raising an error on this. What I understand from the table at the bottom of https://docs.nvidia.com/deploy/cuda-compatibility/why-cuda-compatibility.html is that Jon is effectively in scenario 2: Driver is older than the CUDA runtime (provided by the CUDA toolkit), but it has the same major version. Then, you have what NVIDIA calls 'Minor version compatibility', which is compatibility with some caveats (see https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html for details). The limitations are listed here https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html#application-considerations-for-minor-version-compatibility .

    1. Limited feature set: if a feature introduced in a new CUDA toolkit spans both the toolkit and the driver, and if this is actually used in a code, this could cause a cudaErrorCallRequiresNewerDriver error. We don't know this from the EESSI side, as we don't know what codes may or may not use. My gut feeling is that these cases will be rather exceptional (and the error is fairly descriptive)
    2. Applications that compile device code to PTX will not work on older drivers. I'm pretty sure PyTorch does this when you run torch.compile.
    3. Requires target architecture argument to nvcc (-arch=sm_xx). I think this should be fine, we'd at least like this to happen in all builds we do (whether the build systems -actually- do this is a bit hard to guarantee of course).

    So yeah, it might be an issue in some cases - but there also might be many cases where it would just work fine.

    Should we consider raising an LmodWarning instead when where in what Nvidia calls the 'Minor Version Compatibilty' regime? (and maybe just link to the nvidia docs to explain what that means...)

    @ocaisa agreed with that approach, and remarked that making EESSI compatibility library aware (which means we could be even less restrictive) is possible but tricky. I agree. For now, I'd say:

    • Minor Version Compatiblity regime = LmodWarning with link to nvidia docs
    • Major Version Compatibility regime = LmodError (the current one, essentially)

    As a temporary workaround, one can set EESSI_OVERRIDE_GPU_CHECK to make sure this check isn't done - but I wouldn't recommend that as a general solution, as you'd also miss this check in case of major version incompatiblity.

  3. ocaisa commented on May 13, 2026

    @ocaisa
    Member

    With EESSI/software-layer-scripts#226, this has now been switched to a warning (so the module will complain once per session but will load):

    [aocais00@lrdn0041 ~]$ source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash
    Modules purged before initialising EESSI
    Module for EESSI/2023.06 loaded successfully
    EESSI has selected x86_64/intel/icelake as the compatible CPU target for EESSI/2023.06
    EESSI has selected accel/nvidia/cc80 as the compatible accelerator target for EESSI/2023.06
    (for debug information when loading the EESSI module, set the environment variable EESSI_MODULE_DEBUG_INIT)
    {EESSI/2023.06} [aocais00@lrdn0041 ~]$ module load OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0
    Lmod Warning:
    Your driver CUDA version is 12.2  but the module you want to load requires CUDA 12.4.0. You will therefore be in minor version compatibility mode as described in
    https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html .
    
    While processing the following module(s):
        Module fullname                                   Module Filename
        ---------------                                   ---------------
        UCX-CUDA/1.15.0-GCCcore-13.2.0-CUDA-12.4.0        /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/icelake/accel/nvidia/cc80/modules/all/UCX-CUDA/1.15.0-GCCcore-13.2.0-CUDA-12.4.0.lua
        NCCL/2.20.5-GCCcore-13.2.0-CUDA-12.4.0            /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/icelake/accel/nvidia/cc80/modules/all/NCCL/2.20.5-GCCcore-13.2.0-CUDA-12.4.0.lua
        OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0  /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/icelake/accel/nvidia/cc80/modules/all/OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0.lua
    
    {EESSI/2023.06} [aocais00@lrdn0041 ~]$ module load OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0
    {EESSI/2023.06} [aocais00@lrdn0041 ~]$

    The solution will only land at BSC after the next sync with EESSI (I don't know what the sync frequency is there, but I guess you can ask).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions