This repository installs the software stacks of the Gysela projects with Spack, on the supercomputers we use and on personal machines.
It keeps two things apart:
- projects say what to install: the list of packages and their versions
(
projects/<project>/spack.yaml); - platforms say how to install it on a given computer and architecture:
where Spack lives, which system packages (compilers, MPI, CUDA/ROCm) to reuse,
which variants to use (
platforms/<site>/…).
install.sh combines one project with one platform and architecture into a
Spack environment, and activate.sh activates it in your shell.
install.sh build an environment (Spack, concretize, mirror, install)
activate.sh activate an installed environment (to be sourced)
vendor-crates.sh download Rust crates for offline builds (used by install.sh)
version.txt toolchain version, part of every environment name
spack-version.txt Spack version used on every platform
environment-name.txt naming rule of the environments: {project}-{arch}-{version}
cmake/CMakePresets.json CMake presets to build a project against an environment
projects/<project>/
spack.yaml specs, concretizer settings, package repositories
platforms/<site>/
site.sh site settings: modules, Spack location, group, jobs, Slurm
<arch>/spack/
externals.yaml system packages Spack must reuse
packages.yaml variants, providers, targets, permissions
env_vars.yaml compilers and flags used by the CMake presets
The available platforms and projects are the directories of platforms/
and projects/; ./install.sh --help lists them. The local platform, with
its generic architecture, is meant for a workstation or laptop.
- bash, git, and a Python 3 for Spack itself;
- internet access where
install.shruns (on clusters: the login node); - on platforms in Slurm mode (Jean Zay): a Slurm account allowed on the GPU partition;
- on
local: a GCC with C, C++ and Fortran compilers (gfortran), which is detected automatically.
On a workstation:
git clone https://github.com/gyselax/spack-stack.git
cd spack-stack
./install.sh --platform local --arch generic --project gyselalibxx
. ./activate.sh --platform local --arch generic --project gyselalibxxOn Jean Zay, from a login node (inside tmux or screen: the steps that run on
compute nodes block until their job ends):
export GYSELA_SLURM_ACCOUNT=abc@h100
./install.sh --platform jean-zay --arch h100 --project gyselalibxx
. ./activate.sh --platform jean-zay --arch h100 --project gyselalibxx./install.sh --platform <site> --arch <arch> --project <project> [options]
--step <step> all (default) | spack | env | concretize | mirror | crates | install
--slurm-account <acct> Slurm account (or set GYSELA_SLURM_ACCOUNT)
--slurm / --no-slurm force or disable Slurm mode (default: GYSELA_USE_SLURM from site.sh)
The environment is named after environment-name.txt, for example
gyselalibxx-h100-v2026-11-12-0. The steps run in this order:
| Step | What it does | Where (Slurm mode) |
|---|---|---|
spack |
clone Spack (spack-version.txt) into the platform's prefix, spack bootstrap now |
login node |
env |
(re)create the environment, copy in the project and platform files, spack repo update, detect system packages if the platform asks for it |
login node |
concretize |
concretize, print the specs and the spack.lock checksum |
compute node |
mirror |
download every source into <prefix>/var/spack/mirror and use it as a mirror |
login node |
crates |
download the Rust crates the packages need (see below) | login node |
install |
build and install | compute node |
Without Slurm mode (local, and clusters whose builds can run where
install.sh runs) the mirror and crates steps are skipped and everything
runs in place.
- One step at a time:
--step <step>runs only that step. This is how to resume after a failure, or after a compute-node job reached its time limit:--step installagain continues where Spack stopped. - Versions side by side:
envdeletes and recreates the environment of the current version only. To keep the current environment and build a new one next to it, bumpversion.txt. - Permissions: at the end, every file you own under the Spack prefix is given to the platform's group and made group-readable and writable, so that the install can be shared.
. ./activate.sh --platform <site> --arch <arch> --project <project> [--version <version>]It must be sourced. It applies the platform's site.sh, loads Spack, leaves
any active Spack environment, activates the requested one (the current
version.txt by default, or an older installed --version), and adds Kokkos
Tools to LD_LIBRARY_PATH when the environment contains it. Activating the
environment also sets the compilers and flags used by the CMake presets
(see below); deactivating it removes them.
In a job script:
#!/bin/bash
#SBATCH --account=abc@h100
#SBATCH --constraint=h100
#SBATCH --gres=gpu:1
. /path/to/spack-stack/activate.sh --platform jean-zay --arch h100 --project gyselalibxx
srun ./build/my_simulationcmake/CMakePresets.json provides the presets release, debug and coverage
(configure, build and test presets). They contain no platform: the compilers
and flags come from environment variables, which each architecture declares in
platforms/<site>/<arch>/spack/env_vars.yaml (Spack's
env_vars
section). Spack sets them when the environment is activated, so the same
presets work on every computer.
| Variable | Passed to CMake as |
|---|---|
GYSELA_CXX_COMPILER, GYSELA_C_COMPILER, GYSELA_Fortran_COMPILER |
CXX, CC, FC: the compilers |
GYSELA_CXX_FLAGS |
CXXFLAGS: initial CMAKE_CXX_FLAGS (warnings, compiler workarounds) |
GYSELA_CXX_FLAGS_DEBUG |
appended to CMAKE_CXX_FLAGS_DEBUG by the debug preset |
GYSELA_EXE_LINKER_FLAGS |
LDFLAGS: initial linker flags (e.g. Cray's GPU-aware MPI library) |
An unset variable keeps CMake's default. Like the other platform files,
env_vars.yaml is copied into the environment by the env step of
install.sh: after changing it, run ./install.sh --step env again (or edit the
environment's copy in <environment>/gysela/). The build type and its flags come
from the preset: release (-O3 -DNDEBUG), debug (-g -O1, restart tests
on), coverage (-g -Og --coverage -fprofile-update=atomic, restart tests on).
Each preset builds in build/<preset> with Ninja, and its test preset shows the
output of failing tests.
To use them in a project, include them from its CMakeUserPresets.json (not
committed; this needs CMake 3.30 or later, which the environments provide):
{
"version": 9,
"include": ["$penv{GYSELA_STACK_ROOT}/cmake/CMakePresets.json"]
}then, after activate.sh:
cmake --preset release
cmake --build --preset release
ctest --preset releaseOther options are still given on the command line, for example
cmake --preset debug -DGYSELALIBXX_POISSON_2D_BUILD_TESTING=OFF.
Most of the scripts' logic exists to work around one of these limitations.
1. Compute nodes without internet. On some clusters (Jean Zay), GPU nodes
cannot reach the internet, but builds must run there rather than on the login
nodes, whose resources are limited. In Slurm mode install.sh therefore runs
concretize and install on a compute node through srun. Every download
happens on the login node beforehand: the sources go into a local Spack mirror,
which the build on the compute node then uses.
2. Rust crates are not mirrored by Spack. Packages built with Rust
(py-maturin, py-rpds-py, py-libcst, …) let cargo download their
dependencies (crates) during the build. The Spack mirror does not contain them,
so the build fails without internet. The crates step (vendor-crates.sh)
fixes this:
- on the login node, it downloads the official standalone
cargoof the same version as the environment'srust(checksum verified), stages each Rust package from the mirror and runscargo vendorinto<prefix>/var/spack/cargo-vendor; - on the compute node, before building,
install.shwrites<stage root>/.cargo/config.toml, telling cargo to take the crates from that directory. Cargo reads this file from every parent of its build directory, and Spack builds every package below its stage root. Environment variables cannot do this, because cargo ignores them for source replacement.
An environment without rust skips the step and downloads nothing.
CARGO=/path/to/cargo uses another cargo, and RUST_DIST_SERVER a mirror of
https://static.rust-lang.org. Crates coming from git repositories are not
handled.
3. Git versions need the network to concretize. A spec such as
foo@git.<commit> makes Spack clone the repository during concretization, to
work out which version that commit is, and that fails on a compute node. Always
give the version: foo@git.<commit>=<version>, where <version> exists in the
package recipe (for example py-gysmc@git.f62d…=main).
4. specs: cannot be included. Spack includes can carry configuration
(packages, repos, …) but not the spec list of an environment. Each project
therefore keeps its full spack.yaml, and the platform files are attached to
it: install.sh copies platforms/<site>/<arch>/spack/ into
<environment>/gysela/, which the projects include. externals.yaml is
included by its own name, because a directory include only reads the files
named after a configuration section (packages.yaml, repos.yaml, …).
5. Shared installs without newgrp/sg. Spack's permissions: settings
cover the installed packages, but not its other directories (environments,
repository checkouts, caches). New files also get your primary group rather than
the project's. Since newgrp and sg are not available, install.sh ends by
setting the platform's group (GYSELA_SPACK_GROUP) and the group bits on what
you own under the prefix.
6. Configuration leaking from the user's account. Every site.sh sets
SPACK_DISABLE_LOCAL_CONFIG=true, so that ~/.spack cannot change a build, and
purges the loaded modules. Where /tmp is too small for builds (Jean Zay),
site.sh points TMPDIR, and thus Spack's build stages, to the job's scratch
space. Python's bytecode cache goes to a private per-user directory,
${TMPDIR:-/tmp}/<user>/pycache (PYTHONPYCACHEPREFIX, set by
platforms/python-cache.sh), rather than next to read-only shared installs. It
is private because Python runs a cached .pyc that matches its source's size and
date: whoever can write the cache can run code in your Python processes.
7. Run-time library paths. Spack does not add packages to
LD_LIBRARY_PATH when activating an environment, so activate.sh does it for
Kokkos Tools, whose libraries are loaded at run time.
A platform is a computer (or site); an architecture is a kind of node on it. A platform has one Spack, shared by all its architectures.
1. Create platforms/<site>/site.sh (to add only an architecture, skip to
step 2). It is sourced by install.sh and activate.sh, and must export:
| Variable | Meaning |
|---|---|
GYSELA_SPACK_PREFIX |
where Spack is cloned and everything is installed: <base>/gysela-spack/${GYSELA_SPACK_VERSION}, with <base> typically a project scratch or work space |
GYSELA_SPACK_VERSION |
read it from spack-version.txt, as the existing site.sh files do |
GYSELA_SPACK_GROUP |
Unix group the install is shared with |
GYSELA_SPACK_JOBS |
build parallelism outside Slurm |
SPACK_DISABLE_LOCAL_CONFIG |
true |
and optionally:
| Variable | Meaning |
|---|---|
GYSELA_USE_SLURM=1 |
Slurm mode: concretize and install on compute nodes |
GYSELA_SLURM_QOS, GYSELA_SLURM_TIME |
QoS and time limit of those jobs |
GYSELA_SLURM_CPUS_PER_GPU |
CPUs requested per GPU |
GYSELA_SLURM_GPUS_CONCRETIZE, GYSELA_SLURM_GPUS_INSTALL |
GPUs requested by the two jobs |
GYSELA_DETECT_EXTERNALS |
packages to detect with spack external find instead of describing them in externals.yaml (local uses gcc) |
Start from the closest existing site.sh. Keep the check that refuses to run it
directly, and guard module purge with command -v module: module may not
exist under srun.
2. Create platforms/<site>/<arch>/spack/ with:
externals.yaml: the system packages Spack must reuse instead of building, withbuildable: false, usually the compilers, MPI and CUDA or ROCm.spack external find <package>run on the machine gives a starting point. Only compiler packages (gcc,llvm-amdgpu, …) needextra_attributes: compilers:withc,cxx(andfortran) paths; for other packages it is ignored.packages.yaml: the choices for this hardware: the CPU target, the GPU backend and architecture (+cuda cuda_arch=90,+rocm amdgpu_target=gfx90a, …), the MPI and LAPACK providers, and thepermissionsof the installs.env_vars.yaml(to build projects with the CMake presets): theGYSELA_*compilers and flags underenv_vars: set:, as described in Building a project. Start from the architecture with the closest compiler (GCC,hipccornvcc_wrapper).
An architecture is recognised by its spack/packages.yaml; install.sh lists
the available ones. In Slurm mode, the directory name is also passed to srun
as --constraint, so it must be a valid Slurm constraint of the site (for
example h100).
3. Try it: ./install.sh --platform <site> --arch <arch> --project gyselalibxx --step concretize
shows whether the externals and preferences give a sensible concretization,
before spending hours building.
1. Create projects/<project>/spack.yaml, starting from an existing one.
Keep its first lines, which attach the platform files:
spack:
include:
- path: gysela/externals.yaml
optional: true
- path: gysela
optional: true
specs:
- cxx
- mpi
- kokkos@5.1
# ...
concretizer:
reuse: false
unify: true
repos:
patches:
git: https://github.com/gyselax/spack.git
commit: <commit>
destination: $env/repos/gysela-patches
builtin:
tag: v2026.10.1
destination: $spack/var/spack/repos/builtin-v2026.10.12. Pin what you install. Give versions to the specs, and a commit or tag to
the package repositories (repos:), so that a toolchain version always builds
the same software. A project with no repos: gets Spack's default repository,
and install.sh warns about it.
3. Keep it platform-neutral. Variants that depend on the hardware (GPU
backend, MPI, targets) belong to the platforms' packages.yaml. Project-wide
preferences can go in the project's own packages: section, which takes
precedence over the platform files.
4. Follow the rules above for air-gapped builds: give git versions their
=<version> suffix, and remember that packages built with Rust only work there
through the crates step.
The project is then available to install.sh and activate.sh under its
directory name.
- New package versions or settings: edit the project or platform files, then
bump
version.txt, so that the new environment is built next to the old one and existing users and job scripts are not disturbed. - New Spack version: change
spack-version.txt. Since the prefix contains the version, every platform then gets a new Spack, and--step spack(or a full run) clones it. - New package repositories: update the
commitandtagin the projects'repos:sections.