Articles by tag: float16

LLM Quantization: 160GB Model on a Laptop

Learn how to quantize large LLMs to 4-bit without quality loss. Symmetric/asymmetric quantization, code, benchmarks. Run 80B models locally — developer guide.