mlboydaisuke/LFM2.5-350M-ExecuTorch

LFM2.5-350M — ExecuTorch XNNPACK 8da4w

lfm2_5_350m_xnnpack_8da4w.pte (252.8 MB)

• Source: LiquidAI/LFM2.5-350M (hybrid conv/attention, 16 layers)
• License: LFM Open License v1.0
• Quantization: 8da4w (8-bit dynamic activation / 4-bit weight) + 8-bit embedding (embedding_quantize: "8,0")
• Export: executorch 1.4.0 export_llm, dynamic shape, max_seq_length 2048, XNNPACK extended_ops
• Config: llm_params/lfm2_5_350m_xnnpack_8da4w.yaml

Verification (Mac arm64, 2026-08-13)

Generation gate 3/3 via examples/models/llama/runner/native.py (greedy):

prompt	output
capital of France?	"The capital of France is Paris."
日本の首都は?(日本語)	「日本の首都は東京です。」
haiku about autumn leaves	3-line poem, coherent

Known cosmetic warning: params vocab_size (65536) != tokenizer vocab size.

iPhone 17 Pro (ETBench, XNNPACK CPU, default threads, 2026-08-13)

metric	value
load	0.5 s
ttft (short prompt)	0.03 s
decode	171-173 tok/s

Outputs correct (Paris; coherent 128-token story).

Usage note — chat template is required. This is an instruct model: raw untemplated text makes it emit <|im_end|> immediately (looks like broken generation but is not). Always wrap prompts as <|startoftext|><|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n, eos ids [7].

Core ML build (Neural Engine, iOS 18+)

lfm2_5_350m_coreml.pte (715 MB, no quantization)

The same model lowered to ExecuTorch's Core ML delegate instead of XNNPACK. This is the first Core ML LLM build on this shelf, because until now the path did not run: a model holding a mutable buffer was handed to the delegate as a buffer, Core ML compiled it into a state, and the runtime — told by take_over_mutable_buffer=False that it has no state — failed at execute with The input feature for layers_7_conv_conv_state must be an MLState. The cause is in tag_constant_data, which decided a buffer was mutated by looking at its direct users; LFM2.5's short-convolution state is written a step after it is read, so it was taken for a constant. Fix in pytorch/executorch#21979.

The .pte needs no patched runtime. The fix is export-side; the file below was verified on stock executorch 1.4.0 from pip.

Verification (Mac arm64, macOS 15, 2026-08-21)

Greedy next-token argmax against LiquidAI/LFM2.5-350M in fp32 eager, five prompts, every position after the second counted — 27 of 27 agree:

prompt	Core ML	eager
The capital of France is	Paris	Paris
Water boils at a temperature of
The largest planet in our solar system is	Jupiter	Jupiter
Shakespeare wrote a play called Romeo and	Juliet	Juliet
Two plus two equals	four	four

Decode 88-93 tok/s on the Mac, median of 16 steps after a warm-up of four.

Reset the cache between sequences. The KV cache persists across execute calls, so a second prompt started at position 0 reads the first one's keys. Load a fresh method per sequence. Measuring without this is what made an earlier version of these numbers look like the model was wrong.

Why no quantization

coreml_quantize: c4w builds and runs but loses accuracy: same gate, 21/27 = 0.778, and it misses Jupiter and four. On Qwen3.5-0.8B the same setting scores 3/21. Four-bit weights through this path are not currently shippable, so the Core ML build here is unquantised and correspondingly large. The XNNPACK 8da4w file above remains the small one.