Skip to content

Central spider cache with optional module paths #783

Description

@Micket

I provide a "test" tree for modules that are not yet ready, perhaps need testing by some users to verify that it's ok before proceeding.

For this purpose i have a /apps/test vs /apps/prod. In each tree, I provide a central spider cache via

/apps/test/modules/cacheDir/...
/apps/prod/modules/cacheDir/...

Users have to opt in to using the test tree via module use /apps/test/modules/all.

I don't see any way of providing spider cache for both of these. or, even either one of them.

$ cat /etc/lmodrc.lua 
scDescriptT =  {
   {
      ["dir"]       = "/apps/test/modules/cacheDir/",
      ["timestamp"] = "/apps/test/modules/cacheTS.txt",
   },
   {
      ["dir"]       = "/apps/prod/modules/cacheDir/",
      ["timestamp"] = "/apps/prod/modules/cacheTS.txt",
   },

won't work as it just concatenates both caches. module spider isn't filtering on MODULEPATH at all.

I currently opted for a /etc/lmodrc.lua that just conditionally checks for each subdirectory

local function appendCacheForModulePath(mp)
   local cacheDir = mp .. "/../cacheDir"
   local timestampFile = mp .. "/../cacheTS.txt"

   table.insert(scDescriptT, {
      dir = cacheDir,
      timestamp = timestampFile,
   })
end

local modulePathList = os.getenv("MODULEPATH")
scDescriptT =  {}
if modulePathList then
   for mp in modulePathList:gmatch("[^:]+") do
      print(string.format("echo mp = %s", mp))
      appendCacheForModulePath(mp)
   end
end

but I'm also not sure if i simply reinventing some feature that already existed and i just missed.

Activity

  1. mrcawood commented on Aug 5, 2025

    @mrcawood
    Member

    Hi there, you can now enable dynamic spider cache selection by adding the following to your site's lmodrc.lua file:

    filter_spider_cache_by_modulepath = true

    When this option is enabled, Lmod will only use the system spider caches that correspond to directories in your active MODULEPATH. This will solve the issue you described by keeping your 'prod' and 'test' environments separate for module spider, depending on which module paths you are currently using.

    Please use the IS783-cache_select branch to test this out, and report back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions