[Fix] Release GEMM cache-flushing buffer on thread exit

[divedb]

Motivation

CacheFlushing allocates an L2-sized CUDA buffer for each thread that performs GEMM tuning, but the buffer is never freed. When an executor thread exits, its thread-local CacheFlushing object loses ownership of the allocation, resulting in a memory leak.

On an RTX 4080 SUPER, this leaks 64 MiB per thread that initializes the helper.

This PR fixes the lifetime management of the CUDA buffer and adds a regression test to verify that the allocation is released when the thread-local object is destroyed.

Modification

• Add a non-throwing destructor to CacheFlushing that releases the CUDA buffer.
• Initialize the buffer-related members explicitly.
• Disable copy and move operations to preserve unique ownership of the CUDA allocation.
• Keep the existing static flush interface and same-thread buffer reuse behavior unchanged.
• Add a standalone test_cache_flushing target under BUILD_TEST.

The regression test:

• exercises two successive worker threads and main-thread cleanup;
• calls the helper twice per thread to cover same-thread buffer reuse;
• retires non-default CUDA streams before thread-local destruction; and
• links only the cache-flushing helper and CUDA Runtime so unrelated kernel-registration errors do not obscure the leak check.

Validation was performed with CUDA 13.0.88, an RTX 4080 SUPER, and GCC 13.3.

Before the fix:

• 201,326,592 bytes leaked across three allocations;
• Compute Sanitizer exited with code 99.

After the fix:

• zero leaked bytes;
• zero sanitizer errors;
• Compute Sanitizer exited with code 0.

The regression test can be built and checked with:

cmake --build build --target test_cache_flushing
compute-sanitizer --tool memcheck --leak-check full --error-exitcode 99 build/bin/test_cache_flushing

The build must be configured with BUILD_TEST=ON.

Running test_cache_flushing without Compute Sanitizer exercises the relevant lifetime paths, but does not by itself assert leak freedom.

BC-breaking (Optional)

No. This change does not modify the public CacheFlushing::flush interface or its existing same-thread buffer reuse behavior.

Use cases (Optional)

N/A. This PR fixes a memory leak in the GEMM tuning helper and adds regression coverage; it does not introduce a new user-facing feature.

Checklist

• Pre-commit or other linting tools are used to fix potential lint issues.
• The modification is covered by a dedicated regression test.
• No new dependency on downstream projects or newer downstream versions is introduced.
• Relevant code documentation/comments have been updated where applicable.

[Copilot]

🟡 Changes recommended

cache_utils.h still isn’t self-contained (uses cudaStream_t/size_t without including the needed headers), and the new .cc test currently depends on include order, which is a fragile build hazard.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes a CUDA memory leak in TurboMind’s GEMM tuner by ensuring the thread-local cache-flushing helper releases its per-thread L2-sized device buffer when the worker thread exits, and adds a focused regression test intended to be run under Compute Sanitizer leak checking.

Changes:

• Add a noexcept destructor to CacheFlushing and make its CUDA buffer ownership non-copyable/non-movable.
• Explicitly initialize buffer-related members to safe defaults.
• Add a standalone test_cache_flushing executable under BUILD_TEST for leak-regression verification.

File summaries

File	Description
src/turbomind/kernels/gemm/tuner/cache_utils.h	Adds destructor declaration, deletes copy/move, and value-initializes members for safer ownership semantics.
src/turbomind/kernels/gemm/tuner/cache_utils.cu	Implements destructor to cudaFree the thread-local buffer on thread exit.
src/turbomind/kernels/gemm/test/test_cache_flushing.cc	Adds a regression test exercising TLS reuse/destruction paths across worker threads and main thread.
src/turbomind/kernels/gemm/CMakeLists.txt	Adds test_cache_flushing target gated by BUILD_TEST, linking only cudart and threads.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.