Running MoE GPT-OSS-120B on 6 GB GPU up to 30 t/s

# Optimizing MoE Models on Low-End GPUs: Running GPT-OSS-120B with 6 GB VRAM at up to 30 t/s

The GPT-OSS-120B model with 120B parameters runs on a GPU with 6 GB VRAM at a speed of 30 tokens/s. The key factor is using RAM instead of VRAM thanks to the Mixture of Experts (MoE) architecture and the -cmoe parameter. This allows efficient distribution of tensors between CPU and GPU without hardware upgrades.

Mixture of Experts Architecture

MoE models activate a limited number of experts from a shared pool on each layer. The router (gating network) selects 2–4 experts out of 128 for token generation, reducing the computational load by 24 times compared to dense models.

Each layer contains:

• Attention blocks (shared tensors)
• FFN blocks, replaced by a group of experts
• Router for routing

Experts are isolated mini-FFNs specializing in ranges of input data. During token generation, it passes through all layers (12–100+), alternating MoE and dense layers.

In dense models, FFN is a monolithic block; in MoE, it's a sparse structure. The total parameter volume is large, but active per pass is small. Issues: expert overlap, suboptimal router routing.

Why VRAM is Inefficient for MoE

Offloading entire MoE layers to GPU leads to idle time: out of 128 experts, only 4 are active, the rest wastes VRAM. Attention tensors (shared) are small in volume; offloading them ensures constant GPU utilization.

Configuration comparison:

| Configuration | VRAM Used | Computed | Efficiency |

|--------------|-----------|----------|------------|

| Whole Layers | 16 blocks | 8 blocks | 50% |

| Attn Tensors | 16 blocks | 16 blocks| 100% |

For GPT-OSS-120B with -cmoe, VRAM usage ~3 GB, speed 30 t/s. Dense models have no gaps, VRAM proportional to computations.

Optimization Parameters in llama.cpp

Main Parameter -cmoe / --cpu-moe

Leaves MoE experts (tensors exps) on CPU, offloads:

• Attention tensors of all layers
• Shared layers (without exps in name)
• Shared experts (shexp)

Launch logic:

• -ngl 999 — everything on GPU
• -cmoe — MoE back to CPU

Important: Do not use ik_llama for GPT-OSS-120B.

-ncmoe N / --n-cpu-moe N

Controls layers on CPU when VRAM is abundant. GPT-OSS-120B: 37 layers.

• -cmoe — 3 GB VRAM
• -ncmoe 30 — fills to limit

-ncmoe ignored with -cmoe (latter takes priority).

New Parameter --fit (December 2025)

Automatic VRAM filling:

• --fit — by default, for MoE includes ncmoe
• --fit-ctx 4096 — context limit (default 4096 instead of 128k)
• --fit-target 1024 — free VRAM in MB (default 1 GB)

Automatically sets -fa, -ngl, -t to maximum. Problem: -t = CPU cores - 1, for hybrid CPUs (E/P cores) specify explicitly -t 8.

Additional Memory Optimizations

• -fa 1 — flash attention / swa / mla, auto-detection
• -ub 4096 -b 4096 — KV-cache quantization
• -ctk q8_0 -ctv q8_0 — context quantization

New --models-dir — switch models from UI, --models-max 1 against OOM.

Key Points

• MoE models (GPT-OSS-120B, DeepSeek V3) require RAM > VRAM
• -cmoe offloads only attention tensors to GPU
• --fit automates parameter tuning
• Speed of 30 t/s on 6 GB VRAM achievable
• Avoid ik_llama and auto---t on hybrid CPUs

— Editorial Team