Repository navigation
[WIP] linalg.generic lowering - #3536
Draft
AlexanderViand wants to merge 23 commits into
Draft
AlexanderViand wants to merge 23 commits into
AlexanderViand wants to merge 23 commits into
Conversation
Collaborator
|
How hard would it be to extend ArithmeticDag to support this, so that we can do unit/fuzz testing of the generated kernels? |
AlexanderViand
force-pushed
the
alex/linalg-generic-lowering
branch
2 times, most recently
from
October 27, 2026 22:03
87d05d8 to
596abca
Compare
AlexanderViand
changed the base branch from
main
to
alex/dag-elementwise-gather
October 27, 2026 22:03
AlexanderViand
force-pushed
the
alex/linalg-generic-lowering
branch
from
October 27, 2026 22:10
596abca to
0eba726
Compare
AlexanderViand
force-pushed
the
alex/linalg-generic-lowering
branch
from
October 27, 2026 22:12
5ed165c to
35f77ae
Compare
AlexanderViand
force-pushed
the
alex/linalg-generic-lowering
branch
from
October 27, 2026 22:16
35f77ae to
9cc5193
Compare
AlexanderViand
force-pushed
the
alex/linalg-generic-lowering
branch
4 times, most recently
from
October 27, 2026 23:35
37a89f4 to
d80f1a7
Compare
AlexanderViand
force-pushed
the
alex/linalg-generic-lowering
branch
from
October 28, 2026 00:26
d80f1a7 to
7747e19
Compare
AlexanderViand
force-pushed
the
alex/linalg-generic-lowering
branch
from
October 28, 2026 00:42
7747e19 to
a387702
Compare
AlexanderViand
force-pushed
the
alex/linalg-generic-lowering
branch
from
October 28, 2026 00:51
8f74e4b to
ad181c8
Compare
AlexanderViand
force-pushed
the
alex/linalg-generic-lowering
branch
2 times, most recently
from
October 28, 2026 15:37
1db8613 to
1112c8f
Compare
AlexanderViand
force-pushed
the
alex/linalg-generic-lowering
branch
from
October 28, 2026 18:10
1112c8f to
ed7f40b
Compare
This was referenced Oct 28, 2026
A linalg.generic body that reads a tensor defined outside the op with tensor.extract at indices affine in the loop indices (e.g. a flip, a trace or a diagonal) hides that tensor from analyses of the op's operands, such as secretness. linalg-canonicalizations now turns each such read into an extra input with the corresponding indexing map, hoisting constant tables defined in the body.
Layout propagation and convert-to-ciphertext-semantics only handled collapse_shape and expand_shape ops that add or drop unit dimensions. Any reshape keeps every element in its slot: the reshaped tensor's layout is the source layout composed with the row-major index map of each reassociation group, and the op lowers to nothing.
Layout propagation gave the destination of an insert_slice the layout pushed from the slice, which assumes the slice spans every non-unit dimension of the destination. A slice that covers only part of a non-unit dimension (e.g. a 1x4 piece of a 1x8 concatenation) now leaves the destination in its own layout, and the lowering moves the slice into it. The lowering no longer requires the slots of the slice to be among those of the destination, which only held when the destination took the slice's layout; it converts the slice to the destination layout anyway.
implementGenericKernel builds, as an arithmetic DAG, the FHE kernel of a structured op (a linalg.generic-style loop nest with affine indexing maps) from the layouts of its operands. groupInstances walks the loop nest with a table of each layout and groups the (iteration, output slot) instances by output ciphertext and, per secret input, the ciphertext holding the value read and the rotation that moves it into the output slot. Each group is one evaluation of the body on rotated ciphertexts, masked to the slots the group writes, and the groups are combined per output ciphertext: - Cleartext inputs have no layout; each group reads them through a gather of the elements its slots read. A matvec with a cleartext matrix and a replicated vector gives the Halevi-Shoup kernel. - Elements a layout does not place read as zero, as zero padding does. Groups that only read zeros are dropped, and a cleartext factor that is zero outside its group replaces the mask. - Reductions combine with add or mul in a balanced tree, or with an idempotent op (max, min, and, or) as acc + mask * (op(acc, t) - acc). - The init either seeds the accumulation or keeps the slots the op does not write. A fuzz test compares the kernel with a scalar loop nest on random ops with cleartext inputs, padded, reversed, split and replicated layouts, and idempotent reductions.
convert-to-ciphertext-semantics lowers a linalg op that has a layout and no applicable dedicated kernel with the layout-driven kernel. The pattern translates the body (straight-line elementwise ops) into DAG nodes and reads: - captured scalars, cleartext or replicated secret; - linalg.index, as an iota cleartext input. An init in another layout, or a cleartext init without one, is first moved into the result layout by the same kernel applied to a copy. The pattern has benefit 0, below the dedicated kernels, and the catch-all conversion no longer applies to structured ops with a layout, so an op the kernel rejects gets an error naming the reason and fails to legalize. linalg.reduce over several dimensions now reaches the generic kernel instead of failing, so its test checks the lowering.
layout-propagation first generalizes secret linalg ops that have no dedicated kernel in convert-to-ciphertext-semantics to linalg.generic, and reruns the secretness analysis. For a linalg.generic: - only secret inputs and the init need a layout; cleartext inputs are packed per group by the kernel; - a partial write (e.g. a piece of a concatenation) keeps the layout of its init; - otherwise each result element stays in the slots of the secret input element read in the same iteration (for reductions, the first of them), falling back to the init's or the default layout. No conversion implemented the VecmatDiagonal layout rule, so linalg.vecmat is now generalized and lowered by the generic kernel, and the kernel name is removed.
A CKKS end-to-end test of a zero-padded depthwise convolution, a sum pool and a linear layer, which all lower through the generic kernel.
A max pool starts its max reduction from -inf. The kernel updated the accumulator as acc + written * (max(acc, t) - acc), which turns into NaN with an infinite acc. When the init is the identity of the combiner (-inf for max, +inf for min, the extreme integers, all-ones for and, zero for or), or tensor.empty, the kernel now never reads it: each group sets the slots it writes first and combines into the slots written before.
… apply The layout rules of named ops with dedicated kernels reject some operands: the tricyclic batch matmul layout needs coprime dimensions, the conv kernels need secret data, a cleartext filter, and data and result in one ciphertext, and the matvec kernel needs a cleartext matrix. These failed layout-propagation. Two cases the rules accepted crashed later: a ciphertext-ciphertext tricyclic batch matmul whose rotations leave too few slots for its result, and a conv_1d_ncw_fcw whose expanded filter matrix does not fit in one ciphertext. Each dedicated op now has a check that decides, without changing the IR, whether its rule applies. Matmul, batch matmul and conv_2d_nchw_fchw share their kernel choice with their rules. Layout propagation runs the check before it gives the op's operands layouts; an op the check rejects gets a remark naming the reason and is generalized to linalg.generic for the generic kernel. The checks replace the rules' errors and the secretness errors of hasCompatibleArgumentLayouts, and also reject the two cases above. One list of the dedicated ops serves both the dispatch and the pre-pass that generalizes ops without a rule.
A max that also tracks its argmax (torch log_softmax) has an unused second result, which the single-result generic kernel cannot lower. Layout propagation now drops unused results before generalizing, with the upstream erase patterns. Those only drop a result yielded straight from its accumulator, so the body is first rewritten to yield the accumulator of each unused result that no used result reads. This runs after the softmax raising, which matches the two-result form.
…getCtComplementPoints - In convertRelationToBasicMap, substitute each local that FPL's getLocalReprs expresses as a floor division by that division, so isl knows its definition instead of eliminating an unknown div; only locals without a representation become existentials. - In enumeratePoints, build one isl AST that scans the relation with its locals lifted to ordinary coordinates and walk it in C++ with exact arithmetic, deduplicating consecutive witnesses in lexicographic order. - In getCtComplementPoints, lift the wrapped relation once and test isl_basic_set_is_empty on each fixed-ciphertext slice so isl checks integer feasibility directly without eliminating coupled existentials. - Check isl_point_is_void in anyRangePoint.
collapseDimensions and expandDimensions compose a layout with the map between the reshaped and the source indices (floor divisions for a collapse, a row-major sum for an expand) for any reassociation, not only ones that add or drop unit dimensions. Layout propagation of collapse_shape uses collapseDimensions, and layout optimization hoists conversions through collapse_shape and expand_shape ops that merge or split non-unit dimensions.
Layout optimization hoists a conversion through an op with several convert_layout users only when the hoisted layout places every element of the original one, since hoisting rewrites the from_layout of the other users. The precomposing matvec and conv1d hoisters require the single-ciphertext 1-D layouts they assume. LayoutIsFreeAnalysis treats the results of linalg ops and of ops that cannot hoist a layout conversion as fixed, instead of listing linalg.reduce by name, and layout propagation marks the results of generalized ops the same way.
Layout propagation and the generic kernel skip linalg.generic inputs the body does not read, such as the window tensor of a generalized pooling op, so they neither split groups nor emit dead gathers, and an empty init the body does not read is not moved into the result layout. A generic result reuses the kernel info of an operand in the same layout, and convertToLayout carries the kernel info of the value it converts. getAccessMap pulls the operand layout back along the indexing map. implementGenericKernel allocates the gather rows of a group once.
LayoutAttr::composeLayouts composes the layouts with isl_basic_map_apply_range instead of FPL's compose. isl keeps the definitions of the divisions of each layout, so it eliminates the intermediate coordinates through them, while FPL's compose leaves them as locals without division definitions. Layout propagation composes the layouts of a value with it.
forEachPoint scanned a lifted relation with an isl AST over the identity schedule. When a coordinate is tied to others only through a lattice constraint (for example a slot determined modulo a large constant by the lifted locals), loop bounds over the original coordinates cannot express the tie, and the AST scans the bounding box: 2.5e11 iterations for a 163840-point layout conversion in layout-optimization. Schedule the lifted set by isl_basic_set_reduced_basis instead, the basis isl_basic_set_scan walks. The basis is unimodular, so the schedule maps the points one to one, and in its coordinates such ties become loop bounds: on the relations that hung, the AST iterates exactly once per point and builds in milliseconds. forEachPoint sorts the points to keep its lexicographic order, which also drops repeated witnesses of locals that are not floor divisions.
inferInducedOutputLayout takes the lexmin of the induced map for reductions: each output element goes to the smallest position among the input elements it reads. isl_map_lexmin ran for minutes on the read windows of padded convolutions over replicated layouts, and on composed layouts whose divisions carry large modular coefficients, where isl computes the divisions of existential variables by sampling. The smallest position among the reads is the smallest among their first positions, so take the lexmin of the input layout first. Compose it with the input indexing map by isl_map_apply_range, which derives the divisions afresh; a preimage keeps the divisions of the lexmin result, whose large multipliers made the next lexmin slow. Keep the loops in the range, o -> (ct, slot, d), so the lexmin has no existential variables for them, and project them out of the result: the smallest (ct, slot, d) has the smallest (ct, slot). Detecting equalities and removing redundancies before each lexmin turns the remaining existential variables into explicit divisions where possible. On the layout-propagation inputs of 38 models the pass now takes at most 21 seconds (it hung on five of them and took 75 seconds on three), and its output is unchanged.
An assign_layout with a chain of layouts packs its input step by step, materializing the result of every step. When the input is a constant that every step folds (a resource constant by default, or any constant with fold-when-possible), each intermediate is a folded constant too, and an intermediate can be far larger than the result: the filter layout chain of a 32x16x3x3 stride-2 convolution at 16384 slots has an 8192x16x16384 step before the 8192x16384 result, so ResNet-style models ran out of memory in convert-to-ciphertext-semantics. Such a constant is now folded once, through the composition of the chain. Folding enumerates the points of a layout, and the points of the composed layout are those of the packed result. Chains that emit packing loops keep the stepwise path.
Folding an assign_layout collected every point of the layout as a pair of vectors before writing the packed buffer, about 120 bytes per point. The filter of a main-path convolution has hundreds of millions of points at 16384 slots (a 256x256x3x3 filter packs into 16384x16384 slots), so the collection alone ran CResNet18Cifar and CRepVGGA0 out of memory. The points are now written into the buffer as forEachPoint visits them.
convert-to-ciphertext-semantics lowers a secret tensor.extract_slice to a tensor_ext.remap of the source ciphertexts. It recorded the remap's ciphertext-to-ciphertext relation as the layout of the result instead of the slice's own layout. A later reshape of the slice composed with that two-dimensional relation as if it had the slice's rank, which reads past the relation's columns in an optimized build and corrupted the heap on the strided shortcut slices of BResNet32.
forEachPoint buffers every point the scan visits so that it can report them in lexicographic order and once each. Folding an assign_layout needs neither: a point packed twice writes the same value. The new forEachPointUnordered calls back as the scan visits the points, and the fold uses it, so it keeps no memory per point of a packed filter.
The generic kernel needs one rotation per distinct (input ciphertext, shift) pair. When a secret input is read with a stride, or with a row pitch that differs from the result's, the shift depends on the output position: a stride-2 3x3 stem convolution at 16384 slots had 2.86M groups for 10.8M instances, and every strided window op in the timm models failed to compile. getPolyphaseInputLayout builds a layout under which the shift depends only on the loops the input does not share with the result. Each input index s * p + r (p written to output dim m) splits into q = floor((i - o) / s), placed with the result layout's pitch of m, and the residue (i - o) mod s. The residues and the indices read only through reductions number classes, which take the positions of the output dims the input does not read (a convolution's output channels) and, beyond those, further copies of the output's span. It places each element the op reads once, in no more ciphertexts than the input and result layouts use, and is validated in isl (covers the read elements, one-to-one, one basic map). Whether a layout already has the property is decided in isl as well: the shift, restricted to the unpaired loops, must be single-valued. Layout propagation converts a generic op's operand to that layout when its layout is free to choose, so that layout optimization folds the conversion into the producer (a function argument for a stem). It no longer takes an induced result layout that needs more ciphertexts than the dense default, such as the strided positions of a strided input, unless the result is smaller than a ciphertext: its default layout repeats it across the slots, and the kernel would need rotations for every copy.
When the output layout of a sum (no reduction or an additive one) repeats with a slot period tau, the generic kernel now groups one output copy per class of slots congruent mod tau, the copy at or before the first secret input's position, so that input's shift is below tau. One rotate-and-reduce over the multiples of tau per output ciphertext fills in the other copies before the init is added. A reduction of a large tensor into a small replicated result (squeeze-excite) thus needs about tau groups per pair of ciphertexts instead of one per output copy: AMobileNetV3Small's 16-channel squeeze goes from 65,536 groups to 64. The period is found on the output's slot table. groupInstancesByCost also walks one instance per copy, stopping once it needs more rotations than the class walk, and keeps the cheaper grouping by (rotations, groups), so elementwise ops on replicated operands keep their single unmasked group.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lowers structured ops without a dedicated kernel through a layout-driven kernel.
implementGenericKernel(lib/Kernel/GenericKernelImplementation.h,lib/Utils/Layout/GenericKernel.cpp) builds the FHE kernel of a linalg.generic-style loop nest from its operand layouts.groupInstancesgroups (iteration, output slot) instances by output ciphertext and, per secret input, the source ciphertext and rotation; each group evaluates the body on rotated ciphertexts masked to its slots. Cleartext inputs are read through gathers, unplaced elements read as zero, reductions combine with add/mul trees or idempotent ops, and the init seeds the accumulation or keeps unwritten slots. A fuzz test compares the kernel with a scalar loop nest.linalg.indexare supported; an init in another layout is moved first). Rejected ops get an error naming the reason.linalg.vecmat, whose VecmatDiagonal rule had no conversion, now goes through it.dedicated_rule_fallback.mlir. Unused results (e.g. torch log_softmax's argmax) are erased first.pullBackInputLayoutgives whencountInstanceGroups(a symbolic, ISL-based count, checked against the walk by a fuzz test) shows the kernel then needs at most half the groups and rotations. A secret x secret 64x128x64 matmul goes from 8064 groups to 128 without rotations.insert_sliceleaves its destination in its own layout; linalg-canonicalizations turnstensor.extractat affine indices inside a generic body into an extra input.tests/Examples/lattigo/ckks/depthwise_conv_pool) runs a padded depthwise conv, sum pool and linear layer through the generic kernel.