Repository navigation
[LTX-2.5] Keyframe-aware decoding for the diffusion decoder - #15244
Open
DoronGenzelHass wants to merge 2 commits into
Open
DoronGenzelHass wants to merge 2 commits into
DoronGenzelHass wants to merge 2 commits into
Conversation
Add optional `keyframe_latents` / `keyframe_frame_indices` to `LTX2VideoDiffusionDecoderModel.decode`, `LTX2VideoDiffusionDecodePipeline` and the modular `LTX2DiffusionVaeDecoderStep`. Each video position also attends to its two nearest keyframe planes, which carry the learned `decoder.type_emb` tag enabled by `decoder_keyframe_type_embedding`. The converter sets that flag when the original VAE has the tag.
DoronGenzelHass
force-pushed
the
ltx25-keyframe-diffusion-decode
branch
from
November 11, 2026 11:57
4e91230 to
203d102
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds keyframe-aware decoding to the LTX-2.5 diffusion decoder (
LTX2VideoDiffusionDecoderModel), and exposes it inLTX2VideoDiffusionDecodePipelineand the modularLTX2DiffusionVaeDecoderStep.A decode can now take keyframe planes: single-frame latents at known pixel frames, for example the generated slots
returned by
LTX2DFRPipelinewithoutput_type="latent". Every video position also attends to its two nearest planes, and the planes carry the learned tagdecoder.type_emb. Without keyframes, decoding is unchanged (bit-identical).Refs #14981. HDR and colour handling are out of scope here; this is only the decoder capability and its two entry points.
Changes
keyframe_latents/keyframe_frame_indicesarguments ondecode(), jointneighborhood attention over video and planes, the
decoder_keyframe_type_embeddingconfig flag fortype_emb,and keyframe support in tiled decoding.
LTX2VideoDiffusionDecodePipelineandLTX2DiffusionVaeDecoderStepforward the two arguments.They are optional, so modular workflow selection is unchanged.
decoder_keyframe_type_embedding=Truewhen the original VAE hasdecoder.type_emb.modular tests on tiny configs.
The decoder code is based on @christopher5106's port in [LTX-2.5] Keyframe-aware diffusion decoding and SDR-To-HDR seam keyframes #14975 (closed), with these changes:
decoder.type_embraises instead of decoding the planes untagged;Checkpoint
The
diffusion_decoderweights inLightricks/LTX-2.5-Diffusersalready containdecoder.type_emb(bit-identicalto the original VAE file). We will set
decoder_keyframe_type_embedding: truein its config once this is released.Until then, pass
decoder_keyframe_type_embedding=Truetofrom_pretrained, as the doc examples do. Older diffusersversions ignore the new config key with a warning.
Validation (real weights, NVIDIA GB200)
Compared with Lightricks' reference decoder, using the same weights, inputs and noise:
49x480x736, 2.19 s vs 1.05 s at 97x512x768), with similar peak memory.
without keyframes, 59.5 dB with keyframes every 24 frames, and 51.8 dB with only 2 early keyframes (frames 24, 48).
Each temporal tile keeps the planes inside it plus the nearest one on each side.
Overlap with #14694
#14694 refactors the same decoder methods. Whichever lands second will need a rebase; happy to rebase this onto it.
Self-review notes
Findings left for review, and why
F.scaled_dot_product_attentiondirectly ratherthan going through a processor with
dispatch_attention_fn, soset_attn_processor/set_attention_backenddon't apply to keyframe decodes (documented). No existing backend expresses this attention pattern; open to
shaping it as a processor if preferred.
torch.compilegraph breaks (torch.unique, Python loops over slot groups). Plaindecoding is unaffected.
the existing plain path.
Before submitting
self-reviewskill on the diff?Who can review?
@yiyixuxu @dg845