Local 200B models: Bonsai and TurboQuant

1-Bit Weights and KV-Cache Compression: The Path to Local 200B+ Models

Running AI models with 200B+ parameters locally is becoming a realistic goal thanks to PrismML's Bonsai 8B and Google Research's TurboQuant. These technologies drastically reduce memory requirements for weights and KV-cache while maintaining acceptable quality. An analysis of the principles, benchmarks, and scaling potential.

Bonsai 8B: 1-Bit Weight Representation

PrismML has released the open-source Bonsai 8B model under Apache 2.0. The base architecture is Qwen3-8B dense, but all weights (embeddings, attention projections, MLP projections, LM head) are quantized into an end-to-end 1-bit format. The GGUF file size is 1.15 GB compared to 16.38 GB in FP16, a compression ratio of 14.2x.

Performance:

• M4 Pro: ~131 tokens/sec
• RTX 4090: ~368 tokens/sec
• iPhone 17 Pro Max: ~44 tokens/sec

On RTX 4090, generation speed is up to 6.2x faster than FP16, with 4–5x higher energy efficiency. Benchmarks (average across 6 categories): 70.5 (Bonsai) vs 79.3 (Qwen3 8B), 71.0 (Mistral 3 8B), 67.1 (Llama 3.1 8B). Quality remains competitive despite extreme compression.

Quantization Principle:

Each weight is 1 bit (0 ≈ -scale, 1 ≈ +scale), with one FP16 scale per group of 128 weights. Effectively 1.125 bits/weight. Inference uses inline dequantization kernels without materialization to FP16. Requires forks of llama.cpp and MLX with custom kernels.

Limitation: The full training recipe is not disclosed, mentioning a proprietary mathematical framework to preserve reasoning.

TurboQuant: Compressing KV-Cache to 3.5 Bits/Channel

Google Research proposed TurboQuant for extreme quantization of KV-cache without retraining the model. The goal is to minimize memory usage during inference, which grows with context length.

Key results:

• 3.5 bits/channel: quality neutrality (no loss on Google's tests)
• 2.5 bits/channel: minimal degradation
• KV-cache memory reduction of at least 6x

Example for Qwen3-235B-A22B (128k context):

• FP16 KV-cache: 23.5 GiB
• TurboQuant 3.5 bits: ~5.1 GiB

Savings >18 GiB just on the cache.

Mechanism:

• Data rotation into a space convenient for compression.
• Compensation of quantization errors to preserve computations.

The approach is mathematically grounded, tested on KV-cache.

Combining Approaches for 200B+ Models

Combining Bonsai-like 1-bit weight compression (14.2x) and TurboQuant for KV-cache opens prospects for local inference.

Calculation for Qwen3-235B-A22B (bfloat16 weights 437.7 GiB, 128k context):

| Component | FP16/bfloat16 | Compressed |

|-----------|---------------|------------|

| Weights | 437.7 GiB | ~30.8 GiB (14.2x) |

| KV-cache | 23.5 GiB | ~5.1 GiB (3.5 bits) |

| Total | ~461 GiB | ~36 GiB |

Remaining challenges:

• Runtime overhead
• Memory bandwidth
• Scalability of 1-bit to 200B+
• Integration of kernels into upstream frameworks

This combination shifts the boundary: server-grade 235B models are approaching consumer hardware.

Key Takeaways

• Bonsai 8B reduces weights of an 8B model to 1.15 GB (14.2x), speed on RTX 4090 is 368 tokens/sec, quality close to baseline 8B.
• TurboQuant compresses KV-cache to 3.5 bits/channel without quality loss, saving 18+ GiB on 128k context for 235B.
• The combination yields ~36 GiB for a 235B model instead of 461 GiB—a step toward local giants.
• Limitations: proprietary training details, need for custom kernels.
• Outlook: Development of open-source alternatives for middle/senior devs.

— Editorial Team