Skip to content

perf(math-cuda): one LDE buffer — transpose the fused commit's LDE in place - #960

Open
MauroToscano wants to merge 1 commit into
mainfrom
pt/one-lde-buffer-main
Open

perf(math-cuda): one LDE buffer — transpose the fused commit's LDE in place#960
MauroToscano wants to merge 1 commit into
mainfrom
pt/one-lde-buffer-main

Conversation

@MauroToscano

Copy link
Copy Markdown
Contributor

perf(gpu): transpose the fused commit's LDE in place — one LDE-sized buffer live, not two

Problem (✓ VERIFIED by reading crypto/math-cuda/src/lde.rs @52e88991). expand_row_major_on_stream allocates the row-major LDE for all columns (alloc_zeros::<u64>(lde_size * total_cols), :467). coset_lde_row_major_inner keeps it alive until it returns (:695) and, at :667, calls launch_row_to_col_major, which allocates a SECOND lde_size * cols buffer (:408) while the first is live; coset_lde_row_major_split_trees does the same at :871. Peak ≈ 2·LDE + trace snapshot + tree.

Change (only lde.rs; no kernel, dispatch or caller changes). The LDE is transposed inside its own allocation:

  1. Block pass — the rows × cols matrix is blocks row blocks of rows_per_block rows; each block is transposed with the existing matrix_transpose_strided kernel into cols runs (one column, rows_per_block consecutive rows), ping-ponging through ONE spare block of scratch (block 0 → scratch, block b → slot b−1, scratch → last slot).
  2. Run pass — the runs are now ordered (slot, column); column-major wants (column, block). That permutation of whole runs is followed cycle by cycle in place, with the scratch runs as parking space, and issued as cuMemcpyBatchAsync batches (about 2·blocks + 3·cycles driver calls: ≈136 for 2^22×316).
    Scratch = (cols + 1) · rows_per_block · 8 bytes, capped at 256 MiB (2^16 rows per block for 316 or 436 columns). The trace-snapshot transpose (n×C → its own small buffer) is unchanged. LAMBDA_VM_LDE_TRANSPOSE_UNBATCHED=1 issues the run pass as single D2D copies (measurement knob; same bytes).

Why byte-identical. Nothing computes: both passes only move runs of already-computed values, and the mapping is exactly the old kernel's dst[c·L + r] = src[r·C + c] (block b, row r′, column c lands at c·L + b·R + r′). The leaves are hashed before the transpose; the row-major host D2H (retain_host_lde) is enqueued on the same stream ahead of it, so the host copy sees row-major bytes; the ready event is recorded after it, as before. Same kernels, same values, same roots, same proof bytes.

Tests. lde.rs unit tests (cargo test -p math-cuda --lib inplace_transpose): geometry invariants over 2^1..2^27 rows × {1..65535} cols, and a host model of the run pass that asserts every batch is independent (distinct destinations, no destination aliasing a source) and that every run lands at its column-major position, for 15 (blocks, cols) shapes incl. 64×316 / 64×436 / 128×612. New tests/one_lde_buffer.rs: for 12 shapes × {keccak, blake3} the handle's column-major device bytes equal (raw u64) the row-major host copy the same call returns — base, ext3 and split-tree entry points, plus the snapshot; and the #[ignore]d vram_arm.

Pre-registered predictions (written before any measurement)

Model = LDE + snapshot + tree + scratch + twiddles/weights, GiB = 2^30 bytes. The sampler adds the CUDA context (~0.3–0.6 GiB) to BOTH old and new, so the delta is the robust number.

shape (LFM_HASH @2^21 rows) today (model) new (model) Δ brief target
316 cols, blowup 2 24.88 15.16 (9.875 + 4.94 + 0.125 + 0.155 + 0.06) −9.72 ≤ 15.5 ✓
316 cols, blowup 4 44.78 25.19 (19.75 + 4.94 + 0.25 + 0.155 + 0.09) −19.60 ≤ 25.5 ✓
436 cols (RPO), blowup 2 34.25 20.84 (13.625 + 6.81 + 0.125 + 0.213 + 0.06) −13.41 ≤ 21 ✓

Expected sampler readings: new model + 0.3–0.6 GiB context; vram_arm's in-process 1 kHz peak likewise. With LAMBDA_VM_VRAM_ARM_PREDEV=1 add the resident trace (4.94 / 4.94 / 6.81 GiB) to both columns.

Time (separate line, pre-registered). Transpose stage: old ≈ 2 LDE passes (≈15 ms at 2^22×316·8 B, blowup 2); new ≈ 6 passes + ~140 batch calls ≈ 45 ms → +30 ms (b2), +60 ms (b4), +42 ms (436 cols). Unbatched knob: +55 / +115 / +80 ms. Whole vram_arm iteration (host input, ~5 GB H2D + NTT + leaves + tree): today ≈ 650–750 ms ⇒ ≤ +5%; with _PREDEV=1 (no H2D, ≈400–450 ms) ⇒ +7–8%. Block level: one LFM_HASH main commit per epoch ⇒ ≈ +0.15 s per block. If measured Δ ≫ +60 ms the run pass is launch-bound and the next lever is a larger run (fewer, longer runs), traded against the 256 MiB scratch cap.

Roots. vram_arm prints the root; the same seed on per-table-gpu (test file copied in) must print the same root.

This port

Same change as PR #956 on the per-table-gpu integration branch, applied to main (8064a8e): the row-major LDE is allocated once (lde.rs:466) and both callers of the old second-buffer transpose (:634 fused commit, :820 split trees) now transpose in place; the trace-snapshot transpose is unchanged. The test file carries the split-tree pin; the only difference from #956's is the absent hash argument (main has one hash family).

Gate on main (RTX 5090, box A, b14521a)

  • cargo test -p math-cuda --release --lib inplace_transposetest result: ok. 2 passed; 0 failed
  • merkle_root_parity ok. 3 passed · comp_poly_tree ok. 4 passed · one_lde_buffer ok. 3 passed; 1 ignored
  • make test-cuda-integrationok. 7 passed; 0 failed ... 15.29s · make test-cuda-d1ok. 1 passed
  • make lint → exit 0

Validation of the identical section on per-table-gpu (PR #956)

VRAM arms at 2^21 rows × 316 cols: 25.39 → 15.67 GiB at blowup 2; the blowup-4 (44.7 GB two-buffer) and 436-column (34.2 GB) shapes that failed allocation on the old code now commit at 25.67 / 21.36 GiB; roots byte-identical old vs new. Commit-level time +13% at blowup 2 (≈ +0.2% per block); the named lever is a one-launch run-permutation gather kernel.

…uffer live

The fused row-major R1 commit allocated the row-major LDE for all columns
and then, while it was still live, a second full-size column-major buffer
for the transpose (from both the fused commit and the split-tree path):
peak ~ 2*LDE + trace + tree (24.9 GiB for a 2^21 x 316 table at blowup 2
by the memory model; 44.7 GiB at blowup 4, which no 32 GiB card holds).

The transpose now happens inside the one allocation, over the same tiled
kernel: a block pass transposes each row block through one spare block of
scratch, then the resulting column runs are permuted into column-major
order cycle by cycle, issued as batched device copies. Scratch is one block
plus one run, capped at 256 MiB, instead of a second LDE.

Bytes are unchanged: the passes only move runs of already-computed values,
the row-major host D2H is queued ahead of them on the same stream, the
trace snapshot transpose is untouched, and no kernel changes. New tests pin
the handle's column-major bytes against the row-major host copy the same
call returns (base, ext3, split trees), plus an ignored production-shape
VRAM/time arm for the sampler. LAMBDA_VM_LDE_TRANSPOSE_UNBATCHED=1 issues
the run pass as single copies for measurement.

Same change as the per-table-gpu lane (PR #956), rebuilt against main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant