Skip to content

fix(activation-offload): clear runtime state after train step - #2088

Open
Denny991 wants to merge 1 commit into
InternLM:mainfrom
Denny991:fix/glm52-dsa-offload-oom
Open

Denny991 wants to merge 1 commit into
InternLM:mainfrom
Denny991:fix/glm52-dsa-offload-oom

Conversation

@Denny991

@Denny991 Denny991 commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Fixes the per-step GPU memory growth in GLM-5.2 training when DSA Top-K
activation offload is enabled.

Why is this needed?

The saved Top-K tensors are stored in the process-global OffloadManager.
After the backward pass of the final offload window, the runtime entries were
not cleared in the SFT training path. As a result, each training step retained
one GPU tensor and its pinned CPU buffer. With the GLM-5.2 16K packed input,
this caused approximately 128 MiB of additional GPU memory to remain allocated
per step and could eventually lead to OOM during long runs.

Python gc.collect() does not solve this because the tensors are still
strongly referenced by OffloadManager.

What has changed?

  • Register the H2D/D2H streams used by each saved-tensor offload group.
  • Add a step-scoped OffloadManager.clear_step() lifecycle API.
  • Synchronize the registered offload streams before releasing runtime entries.
  • Clear offload runtime tensors and counters after all microbatch backwards
    complete, while retaining reusable pinned CPU buffers.
  • Add a unit test covering stream synchronization, runtime cleanup, group
    isolation, and pinned-buffer retention.

Validation

  • pytest tests/utils/test_activation_offload.py -q: 1 passed
  • Ruff check and format check: passed
  • GLM-5.2 8-GPU H200 run with recompute_ratio=0.25, DSA Top-K offload,
    DeepEP, and CUTLASS for 60 steps: all 8 ranks completed successfully.
  • In the fixed run, rank0 step-end memory_allocated() stayed at about
    61.628 GiB from step 1 through step 60, with zero remaining runtime
    offload entries and zero GPU offload bytes.

Notes

The fix is applied after the complete TrainEngine.train_step() backward path,
before gradient clipping and optimizer update. No checkpoint or optimizer
state behavior is changed.

@Denny991
Denny991 force-pushed the fix/glm52-dsa-offload-oom branch from d63f29a to 335ed4c Compare September 15, 2026 08:23
Signed-off-by: liutong <liutong@pjlab.org.cn>
@Denny991
Denny991 force-pushed the fix/glm52-dsa-offload-oom branch from 335ed4c to 9e5d80c Compare September 15, 2026 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant