litert-community/wav2vec2-base-960h-LiteRT

wav2vec2-base-960h CTC — LiteRT (GPU)

English speech recognition with wav2vec2-base-960h running fully on the LiteRT CompiledModel GPU (ML Drift) — and with zero FFT anywhere: the raw 16 kHz waveform goes straight into the 1D-conv feature extractor, so there is no mel/fbank step even on the host. Character-level CTC (29 chars + specials), greedy decode, no language model.

Real model output: char-CTC word onsets for J.F. Kennedy's 1961 inaugural address (U.S. National Archives recording, public domain).

Ships as two GPU graphs — the fused graph exceeds the Mali whole-graph shader-compile limit (a graph can be op-clean and still fail to compile when fused; each half compiles and runs fully delegated):

File	Size	Input	Output	API
w2v2_asr_frontend_fp16.tflite	9 MB	waveform [1, 256000]	features [1, 799, 768]	CompiledModel GPU
w2v2_asr_head_fp16.tflite	180 MB	features [1, 799, 768]	CTC logits [1, 799, 32]	CompiledModel GPU

Pipeline

16 kHz mono PCM in [-1, 1], zero-padded to the fixed 16 s window → [GPU] conv frontend → [GPU] 12-layer transformer + lm_head → host greedy-CTC over the valid frames

• Valid frames for n samples: run L=(L-k)//s+1 over the conv stack (10,5)(3,2)(3,2)(3,2)(3,2)(2,2)(2,2) — 50 Hz frames (16 s → 799).
• Blank id 0 (<pad>), | = word delimiter (tokens.txt, index-ordered).
• Greedy char-CTC without an LM has the model's known spelling quirks on hard words (e.g. GRAVED/GRAVE); add a beam+LM host-side if you need the last WER point.

Minimal usage — Python

import numpy as np, torch, torchaudio
from ai_edge_litert.interpreter import Interpreter

wave, sr = torchaudio.load("speech.wav")             # 16 kHz mono, [-1,1]
x = torch.zeros(1, 256000); n = min(wave.shape[1], 256000)
x[0, :n] = wave[0, :n]

def run(path, inp):
    it = Interpreter(model_path=path); it.allocate_tensors()
    d = it.get_input_details()[0]
    it.set_tensor(d["index"], inp.astype(np.float32)); it.invoke()
    return it.get_tensor(it.get_output_details()[0]["index"])

feat = run("w2v2_asr_frontend_fp16.tflite", x.numpy())
logits = run("w2v2_asr_head_fp16.tflite", feat)[0]    # [799, 32]

L = n
for k, s in [(10,5),(3,2),(3,2),(3,2),(3,2),(2,2),(2,2)]:
    L = (L - k) // s + 1
tokens = open("tokens.txt").read().splitlines()
out, prev = [], -1
for i in logits[:L].argmax(-1):
    if i != prev and i != 0: out.append(tokens[int(i)])
    prev = i
print("".join(out).replace("|", " ").strip())

Minimal usage — Kotlin (Android)

val frontend = CompiledModel.create(frontendPath, CompiledModel.Options(Accelerator.GPU), null)
val head = CompiledModel.create(headPath, CompiledModel.Options(Accelerator.GPU), null)
val fIn = frontend.createInputBuffers(); val fOut = frontend.createOutputBuffers()
val hIn = head.createInputBuffers(); val hOut = head.createOutputBuffers()

fIn[0].writeFloat(pcm)                        // [-1,1] floats, zero-padded to 256000
frontend.run(fIn, fOut)
hIn[0].writeFloat(fOut[0].readFloat())        // features [1,799,768]
head.run(hIn, hOut)
val logits = hOut[0].readFloat()              // [799 * 32], readback syncs the GPU
// greedy CTC over the valid frames: argmax per frame, drop blanks (id 0) + repeats,
// map through tokens.txt, '|' -> space

On-device performance (Pixel 8a, CompiledModel GPU)

• frontend 448 ms + head 391 ms per 16 s window (RTF ≈ 0.05); GPU compile 0.7 s + 1.5 s.
• Device logits vs desktop float reference: corr 0.9928 (valid region), per-frame argmax agreement 97.0 %; transcript matches the desktop reference.

Conversion notes

Converted with litert-torch, numerically exact (tflite vs PyTorch: corr 1.000000): GELU → tanh-GELU; frontend GroupNorm → 4D-reshape group-norm (avoids GATHER_ND); pos_conv weight-norm folded to a static weight; the all-valid bidirectional attention mask removed (fixed window → plain SDPA). The CTC head is a plain Linear — logits come out raw.

Snapdragon NPU (Hexagon)

• w2v2_asr_frontend_fp16.tflite — the NPU compiles this graph and then fails to run it: LiteRtException: Failed to invoke the compiled model. The GPU row below is the only S26 figure for it. A clean compile is not evidence that a model runs.

• w2v2_asr_head_fp16.tflite — the NPU is 1.37x faster than the GPU (76.18 ms against 104.1 ms) and loads 5.84x faster (226 ms against 1322 ms).

file	backend	inference (median / min)	load
w2v2_asr_frontend_fp16.tflite	GPU (Adreno)	90.41 ms / 88.96 ms	668 ms
w2v2_asr_head_fp16.tflite	NPU (Hexagon v81)	76.18 ms / 74.02 ms	226 ms
w2v2_asr_head_fp16.tflite	GPU (Adreno)	104.1 ms / 103.8 ms	1322 ms

Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16), LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. Every run held thermal status NONE throughout except w2v2_asr_frontend_fp16.tflite on the GPU LIGHT->LIGHT. Headroom 0.65-0.83, where 1.0 is the throttling threshold.

The NPU rows here ran artifacts compiled ahead of time for SM8850 with QAIRT 2.47.0; the GPU rows ran the published files as they are. LiteRT can also compile for the NPU on the device at first load, which is what lets you ship the published file unchanged — that path and the ten runtime libraries it needs are in the NPU recipe, and we did not measure it here. GPU wiring is in the GPU recipe.

Sources & license

• Model: facebook/wav2vec2-base-960h — Apache-2.0.
• Paper: wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations.
• Hero audio: J.F. Kennedy inaugural address (1961), U.S. National Archives — public domain.