llama.cpp: ASR in Gemma 4 and Qwen3

llama.cpp Adds Speech Recognition for Gemma 4 and Qwen3

llama.cpp now supports Gemma 4 models for audio processing. This enables local Automatic Speech Recognition (ASR) and speech translation. The E4B-it (4.5B parameters, 8B with embeddings) and E2B-it (2.3B parameters, 5.1B with embeddings) models are available in GGUF format for quantization.

Support covers 35+ languages, including Russian, with a 128k token context. The models work in LLM mode with reasoning, image recognition, video, and code handling.

Running on GPU

For testing on an RTX 4090D, compile from source since the Docker image hasn't been updated yet. Launch command:

./build/bin/llama-server --host 0.0.0.0 --port 8080 \
  --ctx-size 128000 \
  -b 1024 -ub 1024 \
  -hf ggml-org/gemma-4-E4B-it-GGUF:Q8_0

Initialization output shows CUDA usage with 48 GB VRAM, n_parallel=4, kv_unified=true. The Q8_0 quant takes about 10 GB VRAM.

Without -b 1024 -ub 1024, you might hit assert errors in audio processing.

Prompting Tips

Google recommends 30-second audio clips, but longer segments can work with varying success.

• ASR (Speech Recognition):

> Transcribe the following speech segment in {LANGUAGE} into {LANGUAGE} text. Follow these specific instructions for formatting the answer: _Only output the transcription, with no newlines._ When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three.

• AST (Speech Translation):

> Transcribe the following speech segment in {SOURCE_LANGUAGE}, then translate it into {TARGET_LANGUAGE}.

When formatting the answer, first output the transcription in {SOURCE_LANGUAGE}, then one newline, then output the string '{TARGET_LANGUAGE}: ', then the translation in {TARGET_LANGUAGE}.

Google AdInline article slot

Qwen3 Support

The same day, ASR integration was added for Qwen3:

• Qwen3-ASR-1.7B (GGUF)
• Qwen3-Omni-30B-A3B-Thinking-GGUF
• Qwen3-Omni-30B-A3B-Instruct-GGUF

These models expand local speech recognition capabilities.

The feature is still raw: the model doesn't always interpret prompts correctly. It needs refinement and user feedback.

Key Takeaways

• llama.cpp now supports Gemma 4 E4B/E2B-it for ASR, AST, and multimodal tasks across 35+ languages.
• Use -b 1024 -ub 1024 for GPU stability.
• 30-second audio limit recommended, but longer clips possible with risks.
• Bonus: Qwen3 ASR models in GGUF.
• 128k context, Q8_0 uses ~10 GB VRAM on RTX 4090D.

— Editorial Team