Skip to content

[Feature] Add optional FlashMLA backend for DSA Sparse MLA - #2083

Open
qiyueliuhuo23 wants to merge 1 commit into
InternLM:mainfrom
qiyueliuhuo23:ws-flashmla
Open

qiyueliuhuo23 wants to merge 1 commit into
InternLM:mainfrom
qiyueliuhuo23:ws-flashmla

Conversation

@qiyueliuhuo23

@qiyueliuhuo23 qiyueliuhuo23 commented Sep 11, 2026

Copy link
Copy Markdown

Motivation

Add an optional FlashMLA forward backend for GLM-5.2 DSA Sparse MLA.

The default PyTorch, TileLang, and cuDNN DSA behavior remains unchanged.
The FlashMLA runtime is maintained separately in the public repository:

https://github.com/DeepLink-org/flashmla-optimization/tree/precise-flashmla

The precise-flashmla branch aligns FlashMLA sparse-prefill output and LSE
with the TileLang reference on SM90/H200.

Changes

  • Add flashmla to the SparseMLABackend protocol.
  • Add lazy FlashMLA backend dispatch in xtuner.v1.ops.sparse_mla.
  • Add runtime availability and SM90 checks.
  • Reuse the existing TileLang indexer and backward implementation.
  • Add a compiled unit test for FlashMLA forward plus TileLang backward.
  • Add FlashMLA installation instructions to the English and Chinese
    installation documentation.

Runtime constraints

  • Requires the public precise-flashmla branch.
  • Requires SM90 or newer GPUs.
  • H200/CUDA 12.8 users should set FLASH_MLA_DISABLE_SM100=1.
  • Current FlashMLA sparse-prefill integration supports kv_group=1,
    value_dim=512, and top-k values divisible by 128.
  • Existing backends and default behavior are unchanged.

Validation

  • Python static compilation passed for the modified Python files.
  • The remote H200 validation record contains 3/3 passing CI checks.
  • 16K packing performance:
    • Median step time: 4.846% faster.
    • TGS: 5.086% higher.
    • Peak memory: reduced by 0.154 GiB.
  • 32K packing performance:
    • Median step time: 4.650% faster.
    • TGS: 4.432% higher.
    • Peak memory: reduced by 6.875 GiB.

CI Comparison

Round local_loss (before → after) Δ reduced_llm_loss (before → after) Δ grad_norm (before → after) Δ TGS (before → after) Δ Memory (before → after) Δ
R1 6.613820 → 6.631019 +0.2600% 6.281574 → 6.274125 -0.1186% 10.159761 → 11.102958 +9.2836% 1660.282 → 1660.762 +0.0289% 116.354 → 116.358 +0.0034%
R2 5.883135 → 5.911429 +0.4809% 5.488452 → 5.508216 +0.3601% 7.448886 → 13.050797 +75.2047% 1660.282 → 1679.164 +1.1373% 116.354 → 116.358 +0.0034%
R3 6.093675 → 6.110326 +0.2732% 6.088449 → 6.093432 +0.0818% 10.159761 → 8.957117 -11.8373% 1660.282 → 1667.970 +0.4631% 116.354 → 116.358 +0.0034%

Performance Comparison

Packing Length Median Step-Time (s) P90 Step-Time (s) P99 Step-Time (s) TGS (tokens/s) Memory (GiB)
16K 4.548683 / 4.328250 4.571737 / 4.351493 4.618507 / 4.387413 28,489.920 / 29,939.000 104.919 / 104.765
32K 19.688900 / 18.773400 20.814340 / 20.062140 21.862641 / 21.509888 13,099.526 / 13,680.117 133.956 / 127.081

Performance Difference

Packing Length Median Step-Time P90 Step-Time P99 Step-Time TGS Memory
16K -4.846% -4.817% -5.004% +5.086% -0.146% (-0.154 GiB)
32K -4.650% -3.614% -1.613% +4.432% -5.132% (-6.875 GiB)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant