InternVL3.5-1B — LiteRT-LM (on-device Vision-Language Model)

OpenGVLab/InternVL3_5-1B converted to the LiteRT-LM (.litertlm) format for on-device image+text inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models, and the same runtime that runs litert-community/FastVLM-0.5B).

InternVL3.5-1B is a compact vision-language model: an InternViT vision encoder + pixel-shuffle + MLP projector feeding a Qwen3-0.6B language decoder (the newer Qwen3 backbone is what distinguishes it from the InternVL3-2B build, which used Qwen2.5-1.5B). This bundle runs it through LiteRT-LM's fast_vlm multimodal path — give it an image and a question, get a grounded answer, fully on-device.


File	`model.litertlm` (~0.82 GB)
Vision	InternViT encoder + pixel-shuffle + MLP projector, int8 weights — single 448×448 image → 256 image tokens
Decoder	Qwen3-0.6B, int4 weights (symmetric, blockwise-32 + OCTAV optimal-clipping); input embedding INT8 (externalized section)
Compute	integer
Context (KV cache)	2048
Image input	resized to 448×448 (ImageNet normalization is baked into the vision encoder)
Base model	OpenGVLab/InternVL3_5-1B (Apache-2.0)

How to use

1. Install the runtime

pip install litert-lm

2. Run it in one command — this downloads the bundle, encodes your image and answers:

litert-lm run --from-huggingface-repo litert-community/InternVL3_5-1B model.litertlm \
  --attachment photo.jpg \
  --prompt "Describe this image in one sentence."

On the COCO sample image huggingface/documentation-images/coco_sample.png (two tabby cats on a pink blanket, remote controls beside them) this prints:

Two tabby cats are lying on a pink couch, with remote controls nearby.

Drop --prompt for an interactive chat, and pass --attachment more than once for several images. litert-lm serve exposes the same bundle as a local OpenAI-compatible API. The same file runs on macOS, Linux and Windows.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine. These figures cover the text path; the vision encoder runs once per image and is not included.

Device	Backend	Prefill (256)	Decode	TTFT
Apple M4 Max (macOS)	CPU	338 tok/s	43.7 tok/s	0.84 s
Apple M4 Max (macOS)	GPU (Metal)	3878 tok/s	176.2 tok/s	0.08 s

Every desktop backend listed above was checked by actually generating a caption on it, not just by reading the benchmark tool's output.

Accuracy note

The vision tower converts bit-faithfully to the reference — float CPU-parity end-to-end corr ≈ 1.0 (max abs diff ~1e-4), with no FLEX/CUSTOM fallback ops; int8 vision weights preserve grounding. The Qwen3-0.6B decoder uses the same blockwise-32 + OCTAV int4 recipe that scores 90.7% GSM8K on the sibling Ministral-3-3B-Reasoning build. On a reference eager run the model describes photos accurately and in detail (e.g. a black-and-white Ansel-Adams-style landscape → "dramatic mountain landscape … snow-capped peaks … a winding river through a forested valley").

On-device performance: decode/load are expected to be in line with the InternVL3-2B build on the same runtime (~20 tok/s CPU, ~45 tok/s GPU on iPhone 17 Pro for single-image VQA). Independent on-device measurement for this specific 2B/Qwe3 build is recommended before quoting exact numbers.

Galaxy S26 — GPU backend

The published bundle runs on the Android GPU backend and generates.

file	GPU backend	delegation	peak
`model.litertlm`	runs	3840 / 3840 ops across 3 subgraphs on `LiteRT GPU`	628 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-25.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in main; the runtime accepts that split.

The gate prompt carries no image, so this covers engine creation and the text path. The vision path on the GPU is not measured here.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

⚠️ Known limitation — one image per conversation on the GPU backend

Single-image VQA — the primary use case — works on GPU. But on the GPU (Metal) backend, a second image in the same conversation truncates the answer — ask about one image per chat (start a new conversation for a different image). This is GPU-delegate-specific, not a model/bundle issue: on the CPU backend, multi-image works. The same GPU truncation reproduces with Apple's litert-community/FastVLM-0.5B, so it is general to the runtime's GPU fast_vlm path, not specific to this model. For reliable multi-image, run on the CPU backend.

Run on iPhone / macOS

Use the LiteRT-LM Swift runtime (swift-litert-lm / the LiteRTDemo sample). Load model.litertlm with the image (vision) tower enabled (modalities [.vision]), attach a photo, and ask a question.

Note for app integrators: this is a vision-only bundle (no audio tower). Bring up the engine with the vision modality only (Modality.textImage / [.vision]) — requesting the audio tower (.all) on a bundle with no audio section fails at session creation.

Run on Android — Google AI Edge Gallery

Install a recent Google AI Edge Gallery (1.0.16+ can import .litertlm directly from Hugging Face), download model.litertlm, import it (tap +), attach an image and ask. The bundle already carries the tokenizer and prompt template.

Conversion notes

LiteRT-LM fast_vlm bundle: VISION_ENCODER ([1,448,448,3]→[1,256,4096]) + VISION_ADAPTER ([1,256,4096]→[1,256,1024], matched to the Qwen3-0.6B hidden size) + single-token EMBEDDER + PREFILL_DECODE (embeddings-input).
The vision encoder bakes InternVL's ImageNet normalization and the NCHW transpose into the graph (the runtime feeds a [0,1] NHWC image).
The InternViT attention is rewritten 4D-clean (qkv split before the head reshape, avoiding a 5D intermediate) for the GPU delegate.
Decoder extracted from the InternVLChat wrapper as a standalone Qwen3ForCausalLM (dynamic rope_scaling stripped; exported with cache ≤ base max so base RoPE is exact).

2026-08-28 — start_token fix (weights unchanged)

The bundle's LlmMetadata start_token held the literal string "None". This tokenizer has no BOS, and the LiteRT-LM engine resolved that string to a real vocabulary token — so every prompt began with the word None, which the model was never trained on. The start token has been removed.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.