LLM Quantization: 160GB Model on a Laptop

LLM Quantization: Running 160GB Models on a Laptop with Minimal Loss

Models like Qwen-3-Coder-Next with 80 billion parameters take up 159.4 GB of memory. Quantization reduces model size by 4x and speeds up inference by 2x, with only a 5–10% drop in quality. This makes running large LLMs on consumer hardware feasible without sacrificing performance.

Parameters and Their Impact on Model Size

Parameters—weights in a neural network—determine an LLM’s memory footprint. Each parameter is stored as a floating-point number. The simplest unit: input multiplied by weight yields output. Real models have hundreds of layers with thousands of nodes, resulting in billions of parameters.

For example, a network with 2 inputs, 3 layers of 4 nodes each, and 2 outputs has 64 parameters. Scale this to hundreds of thousands of nodes, and you get trillions of weights. Histograms of popular models show that 99% of values are close to zero, within the range [-0.5, 0.5].

Number Representation in Memory

Computers store float32: 1 sign bit, 8 exponent bits, 23 mantissa bits. Range ±3.4×10³⁸ with 7 significant digits. Distribution is uneven—denser near zero, sparser at extremes. This suits LLMs well, where weights are small.

Float16: 1+5+10 bits, 3–4 digits precision, range ±65,504. Bfloat16 (1+8+7): wide range, 2–3 digits. Float8/Float4 are experimental, with 3–4 mantissa bits.

Sine wave approximation: float32 is smooth; float4 is step-like with noticeable errors.

Principles of Quantization

Quantization is lossy compression: mapping floats to a smaller value set. Simple rounding (round-to-nearest) from bfloat16 to float4 breaks the model: weights become zero, output is zero. Why? Float4’s range [-3,3] doesn’t match typical weights [-0.89,0.16].

Symmetric Quantization

Scale data into integer ranges. Formula: scale = max_abs / (2^(bits-1) - 1). Quantize: round(value / scale), dequantize: quantized * scale.

Example code in JavaScript:

function quantize({ values, bits }) {
  const vmax = Math.max(...values.map(Math.abs));
  const qmax = 2 ** (bits - 1) - 1;
  const scale = vmax / qmax;
  return {
    values: values.map((v) => Math.round(v / scale)),
    scale,
  };
}

function dequantize({ values, scale }) {
  return values.map((v) => v * scale);
}

For values = [-0.89, 0.16, 0.08, -0.13, 0.16, -0.54], bits=4:

• quantized: [-7,1,1,-1,1,-4], scale≈0.127
• dequantized: [-0.89,0.127,0.127,-0.127,0.127,-0.509]
• average error: 18%

Model output after 4-bit quantization: 30% deviation from original, but 4x smaller memory footprint.

Asymmetric Quantization

Improves symmetric quantization by handling min/max separately. Range [min, max] maps to [qmin, qmax]. Formula:

• offset = min
• scale = (max - min) / (qmax - qmin)
• quantized = round((value - offset) / scale)

This efficiently uses space: for skewed data (more negative values), positive side isn’t wasted. Average error drops to 5–10%.

Apply to tensors: quantize per channel or group (per-group quantization) to minimize activation errors.

Evaluating Quality After Quantization

Measure perplexity on a validation dataset or task-specific metrics (BLEU, ROUGE). Benchmarks:

• Qwen-3-Coder-Next 4-bit: +7% perplexity vs FP16
• Speed: x2 on GPU without tensor cores

| Format | Size (GB) | Perplexity | Speed (tokens/sec) |

|--------|-----------|------------|--------------------|

| FP16 | 159.4 | 1.00 | 1.0 |

| INT8 | 39.8 | 1.05 | 1.8 |

| INT4 | 19.9 | 1.09 | 2.1 |

Key Takeaways

• Quantization reduces model size 4–8x without retraining.
• Use symmetric quantization for symmetric data; asymmetric for skewed distributions.
• Per-group quantization (groups of 128 elements) balances accuracy and speed.
• Supported out-of-the-box in llama.cpp, bitsandbytes: INT4/INT8.
• Test on downstream tasks: coding, QA.

— Editorial Team