Skip to content

timer_enable_nvtx has no effect in CUDA builds #7956

Description

@chengleizheng

Describe the bug

Setting timer_enable_nvtx 1 in INPUT produces no NVTX ranges in any CUDA build.The parameter is parsed and documented, but the code that consumes it is removedby the preprocessor, so profiling timelines contain no ABACUS stage labels.

Root cause: __USE_NVTX is defined only on the final executable target(source/CMakeLists.txt:520), whose single translation unit source_main/main.cppnever includes an NVTX header and never calls NVTX. The two targets that actually need the macro — source_base/timer.cpp (the base OBJECT library) and source_main/driver.cpp (the driver OBJECT library) — guard their NVTX calls
with #if defined(__CUDA) && defined(__USE_NVTX), so those blocks are deleted in every CUDA build and timer::enable_nvtx_ stays at its initial value false.

Expected behavior

timer_enable_nvtx 1 should emit NVTX ranges (e.g. :Operator:hPsi, :HSolverPW:solve, :ElecStatePW:psiToRho) visible in Nsight Systems, matching the documentation in docs/parameters.yaml and docs/advanced/input_files/input-main.md.

To Reproduce

  1. Configure with CUDA:
    cmake -B build -S . -DUSE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89
  2. Build: cmake --build build -j
  3. Add timer_enable_nvtx 1 to a PW INPUT (e.g. tests/11_PW_GPU/scf_cg)
  4. Run under Nsight Systems and check for NVTX ranges:
    nsys profile --trace=cuda,nvtx -o prof ./build/abacus_basic_gpu
    nsys stats --report nvtx_pushpop_sum prof.nsys-rep
    → no NVTX ranges are reported.

Compile-level evidence (no GPU needed):

  • nm -C build/source/source_base/CMakeFiles/base.dir/timer.cpp.o | grep -i nvtx
    returns only ModuleBase::timer::enable_nvtx_ — no nvtxRangePushA/nvtxRangePop.
  • driver.cpp.o contains no reference to enable_nvtx_, i.e. the enable call
    at driver.cpp:129 was also removed by the preprocessor.

Environment

  • OS: WSL2 (Ubuntu)
  • Compiler: g++ / nvcc 13.1
  • CUDA: 13.1 (NVTX v2 removed in 12.9; nvtx3/nvToolsExt.h is used via cuda_compat.h)
  • GPU: NVIDIA RTX 4060 Laptop

Additional Context

No response

Task list for Issue attackers (only for developers)

  • Verify the issue is not a duplicate.
  • Describe the bug.
  • Steps to reproduce.
  • Expected behavior.
  • Error message.
  • Environment details.
  • Additional context.
  • Assign a priority level (low, medium, high, urgent).
  • Assign the issue to a team member.
  • Label the issue with relevant tags.
  • Identify possible related issues.
  • Create a unit test or automated test to reproduce the bug (if applicable).
  • Fix the bug.
  • Test the fix.
  • Update documentation (if necessary).
  • Close the issue and inform the reporter (if applicable).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

BugsBugs that only solvable with sufficient knowledge of DFTGPU & DCU & HPCGPU and DCU and HPC related any issues

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions