Conversation
Denny991
force-pushed
the
fix/glm52-dsa-offload-oom
branch
from
September 15, 2026 08:23
d63f29a to
335ed4c
Compare
Signed-off-by: liutong <liutong@pjlab.org.cn>
Denny991
force-pushed
the
fix/glm52-dsa-offload-oom
branch
from
September 15, 2026 08:29
335ed4c to
9e5d80c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes the per-step GPU memory growth in GLM-5.2 training when DSA Top-K
activation offload is enabled.
Why is this needed?
The saved Top-K tensors are stored in the process-global
OffloadManager.After the backward pass of the final offload window, the runtime entries were
not cleared in the SFT training path. As a result, each training step retained
one GPU tensor and its pinned CPU buffer. With the GLM-5.2 16K packed input,
this caused approximately 128 MiB of additional GPU memory to remain allocated
per step and could eventually lead to OOM during long runs.
Python
gc.collect()does not solve this because the tensors are stillstrongly referenced by
OffloadManager.What has changed?
OffloadManager.clear_step()lifecycle API.complete, while retaining reusable pinned CPU buffers.
isolation, and pinned-buffer retention.
Validation
pytest tests/utils/test_activation_offload.py -q: 1 passedrecompute_ratio=0.25, DSA Top-K offload,DeepEP, and CUTLASS for 60 steps: all 8 ranks completed successfully.
memory_allocated()stayed at about61.628 GiBfrom step 1 through step 60, with zero remaining runtimeoffload entries and zero GPU offload bytes.
Notes
The fix is applied after the complete
TrainEngine.train_step()backward path,before gradient clipping and optimizer update. No checkpoint or optimizer
state behavior is changed.