perf(math-cuda): one LDE buffer — transpose the fused commit's LDE in place - #960
Open
MauroToscano wants to merge 1 commit into
Open
perf(math-cuda): one LDE buffer — transpose the fused commit's LDE in place#960MauroToscano wants to merge 1 commit into
MauroToscano wants to merge 1 commit into
Conversation
…uffer live The fused row-major R1 commit allocated the row-major LDE for all columns and then, while it was still live, a second full-size column-major buffer for the transpose (from both the fused commit and the split-tree path): peak ~ 2*LDE + trace + tree (24.9 GiB for a 2^21 x 316 table at blowup 2 by the memory model; 44.7 GiB at blowup 4, which no 32 GiB card holds). The transpose now happens inside the one allocation, over the same tiled kernel: a block pass transposes each row block through one spare block of scratch, then the resulting column runs are permuted into column-major order cycle by cycle, issued as batched device copies. Scratch is one block plus one run, capped at 256 MiB, instead of a second LDE. Bytes are unchanged: the passes only move runs of already-computed values, the row-major host D2H is queued ahead of them on the same stream, the trace snapshot transpose is untouched, and no kernel changes. New tests pin the handle's column-major bytes against the row-major host copy the same call returns (base, ext3, split trees), plus an ignored production-shape VRAM/time arm for the sampler. LAMBDA_VM_LDE_TRANSPOSE_UNBATCHED=1 issues the run pass as single copies for measurement. Same change as the per-table-gpu lane (PR #956), rebuilt against main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
perf(gpu): transpose the fused commit's LDE in place — one LDE-sized buffer live, not two
Problem (✓ VERIFIED by reading
crypto/math-cuda/src/lde.rs@52e88991).expand_row_major_on_streamallocates the row-major LDE for all columns (alloc_zeros::<u64>(lde_size * total_cols), :467).coset_lde_row_major_innerkeeps it alive until it returns (:695) and, at :667, callslaunch_row_to_col_major, which allocates a SECONDlde_size * colsbuffer (:408) while the first is live;coset_lde_row_major_split_treesdoes the same at :871. Peak ≈ 2·LDE + trace snapshot + tree.Change (only
lde.rs; no kernel, dispatch or caller changes). The LDE is transposed inside its own allocation:rows × colsmatrix isblocksrow blocks ofrows_per_blockrows; each block is transposed with the existingmatrix_transpose_stridedkernel intocolsruns (one column,rows_per_blockconsecutive rows), ping-ponging through ONE spare block of scratch (block 0 → scratch, block b → slot b−1, scratch → last slot).cuMemcpyBatchAsyncbatches (about2·blocks + 3·cyclesdriver calls: ≈136 for 2^22×316).Scratch =
(cols + 1) · rows_per_block · 8bytes, capped at 256 MiB (2^16 rows per block for 316 or 436 columns). The trace-snapshot transpose (n×C → its own small buffer) is unchanged.LAMBDA_VM_LDE_TRANSPOSE_UNBATCHED=1issues the run pass as single D2D copies (measurement knob; same bytes).Why byte-identical. Nothing computes: both passes only move runs of already-computed values, and the mapping is exactly the old kernel's
dst[c·L + r] = src[r·C + c](block b, row r′, column c lands atc·L + b·R + r′). The leaves are hashed before the transpose; the row-major host D2H (retain_host_lde) is enqueued on the same stream ahead of it, so the host copy sees row-major bytes; thereadyevent is recorded after it, as before. Same kernels, same values, same roots, same proof bytes.Tests.
lde.rsunit tests (cargo test -p math-cuda --lib inplace_transpose): geometry invariants over 2^1..2^27 rows × {1..65535} cols, and a host model of the run pass that asserts every batch is independent (distinct destinations, no destination aliasing a source) and that every run lands at its column-major position, for 15 (blocks, cols) shapes incl. 64×316 / 64×436 / 128×612. Newtests/one_lde_buffer.rs: for 12 shapes × {keccak, blake3} the handle's column-major device bytes equal (raw u64) the row-major host copy the same call returns — base, ext3 and split-tree entry points, plus the snapshot; and the#[ignore]dvram_arm.Pre-registered predictions (written before any measurement)
Model = LDE + snapshot + tree + scratch + twiddles/weights, GiB = 2^30 bytes. The sampler adds the CUDA context (~0.3–0.6 GiB) to BOTH old and new, so the delta is the robust number.
Expected sampler readings: new model + 0.3–0.6 GiB context;
vram_arm's in-process 1 kHz peak likewise. WithLAMBDA_VM_VRAM_ARM_PREDEV=1add the resident trace (4.94 / 4.94 / 6.81 GiB) to both columns.Time (separate line, pre-registered). Transpose stage: old ≈ 2 LDE passes (≈15 ms at 2^22×316·8 B, blowup 2); new ≈ 6 passes + ~140 batch calls ≈ 45 ms → +30 ms (b2), +60 ms (b4), +42 ms (436 cols). Unbatched knob: +55 / +115 / +80 ms. Whole
vram_armiteration (host input, ~5 GB H2D + NTT + leaves + tree): today ≈ 650–750 ms ⇒ ≤ +5%; with_PREDEV=1(no H2D, ≈400–450 ms) ⇒ +7–8%. Block level: one LFM_HASH main commit per epoch ⇒ ≈ +0.15 s per block. If measured Δ ≫ +60 ms the run pass is launch-bound and the next lever is a larger run (fewer, longer runs), traded against the 256 MiB scratch cap.Roots.
vram_armprints the root; the same seed onper-table-gpu(test file copied in) must print the same root.This port
Same change as PR #956 on the
per-table-gpuintegration branch, applied tomain(8064a8e): the row-major LDE is allocated once (lde.rs:466) and both callers of the old second-buffer transpose (:634fused commit,:820split trees) now transpose in place; the trace-snapshot transpose is unchanged. The test file carries the split-tree pin; the only difference from #956's is the absent hash argument (main has one hash family).Gate on
main(RTX 5090, box A, b14521a)cargo test -p math-cuda --release --lib inplace_transpose→test result: ok. 2 passed; 0 failedmerkle_root_parityok. 3 passed·comp_poly_treeok. 4 passed·one_lde_bufferok. 3 passed; 1 ignoredmake test-cuda-integration→ok. 7 passed; 0 failed ... 15.29s·make test-cuda-d1→ok. 1 passedmake lint→ exit 0Validation of the identical section on
per-table-gpu(PR #956)VRAM arms at 2^21 rows × 316 cols: 25.39 → 15.67 GiB at blowup 2; the blowup-4 (44.7 GB two-buffer) and 436-column (34.2 GB) shapes that failed allocation on the old code now commit at 25.67 / 21.36 GiB; roots byte-identical old vs new. Commit-level time +13% at blowup 2 (≈ +0.2% per block); the named lever is a one-launch run-permutation gather kernel.