Repository navigation
LFM3: Liquid's LFM3 models (model_type lfm3) - #57
Merged
Merged
Conversation
LFM2's gated short convolutions with two kinds of attention layers: full attention with partial RoPE (the first pairs follow the whole head's frequency progression, the rest unrotated: infinite periods to mx.fast.rope) and sliding-window attention (a RotatingKVCache, its own mask); a sigmoid gate per head on the attention output; QK norms; sandwich norms after the operator and the FFN. Checkpoints nest the text model under language_model. LiquidAI/LFM3-1.7B-MLX-8bit answers through the server with its reasoning split from the content and tool calls parsed. Tests: partial RoPE leaves the tail unrotated; cached steps match one pass past the sliding window; sanitize; the shared model runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhEwWJCoJPXFyyCzp988Bx
Review (codex, muse): the real 1.7B checkpoint matches the reference; configs leaning on defaults didn't. Defaults now as in the HF configuration (rope parameters, the layer pattern, sizes, conv width), tie_embedding read as its alias, and a non-default RoPE type or a missing layer type's entry is refused instead of computed as plain RoPE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhEwWJCoJPXFyyCzp988Bx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LiquidAI/LFM3-1.7B-MLX-8bit failed to load with "Model type lfm3 not supported". This adds
mlx_lm/models/lfm3.py, written from the Hugging Face reference that ships in the checkpoint (modeling_lfm3.py).What LFM3 adds over LFM2:
mx.fast.ropeas infinite periods. Checked against the HF formula.RotatingKVCacheand its own mask.o_proj.language_model..Checked:
generateand the server: a correct answer, reasoning split from the content, a tool call parsed (get_weather {"city": "Paris"}).test_server.pypasses.There's an open upstream PR for the same model (ml-explore#2016); this one was written independently from the reference.
Reviews (codex, muse): the real checkpoint matches the reference numerically. Defaults for sparse configs and the RoPE-type refusal were added in the second commit.
🤖 Generated with Claude Code
https://claude.ai/code/session_01XhEwWJCoJPXFyyCzp988Bx