LLM Hallucinations: Data Compression Artifacts

LLM Hallucinations as Lossy Compression Artifacts: An Information Theory Perspective

LLMs function as lossy data compressors: predicting the next token is equivalent to compressing a vast text corpus into model parameters. According to Shannon's theorem, prediction quality directly determines the compression ratio. Hallucinations occur during decompression, when the model reconstructs lost information with plausible but inaccurate fragments.

The Mathematical Basis of Prediction and Compression

At a fundamental level, LLMs implement prediction of the next token's distribution based on context. This is mathematically identical to arithmetic coding, where a probabilistic predictor minimizes bits per source entropy.

def predict_next_token(context: str) -> Distribution:
    """Token prediction = decompression of compressed representation"""
    pass

Model weights store compressed knowledge from the training dataset. Increasing model size raises the bitrate, reducing losses but not eliminating them entirely.

Comparison with JPEG:

| JPEG | LLM |

|------|-----|

| Large blocks | General patterns |

| Fine details | Rare facts |

| Edge artifacts | Hallucinations |

| Quality 1–100% | Parameters 7B–405B |

Why LLMs Excel at Code but Struggle with Math

Code compresses efficiently due to strict syntax and repetitive structures. Patterns like for i in range(n) are preserved almost losslessly, similar to large uniform areas in an image.

Math suffers from the loss of precise numbers—fine details without patterns. Example:

> 17 × 38?
GPT: 646  # correct

> 1847 × 9283?
GPT: 17,143,301  # error, correct: 17,143,501

Rare combinations cause artifacts, analogous to blurring numbers in JPEG.

• Code: High compressibility due to limited vocabulary and structures.
• Math: Requires exact storage or computation, unavailable in a lossy approach.
• Facts: Rare events are lost first.

Temperature as an Artifact Regulator

The temperature parameter controls sampling from the distribution:

# temperature=0.0: argmax, sharp artifacts
# temperature=0.7: balanced probabilities
# temperature=1.5: noise, 'creativity'
# temperature→∞: randomness

Low temperature increases determinism but locks in artifacts. High temperature adds variability, blurring the losses.

Techniques to Minimize Artifacts in Compression Terms

LLM techniques are reinterpreted through a compression lens:

• RAG: Inserting lossless data into context, bypassing compressed representations.
• Fine-tuning: Re-encoding with priorities on target domains.
• Prompt Engineering: Guiding the decoder to relevant blocks in latent space.
• RLHF: Optimizing for subjective quality, like psychoacoustic models in audio.
• System Prompt: Configuring the decoder profile.

| Technique | Compression Analogy |

|-----------|---------------------|

| RAG | Lossless insertion |

| Fine-tuning | New codec profile |

| Prompt | Seek in compressed file |

The Impossibility of Fully Eliminating Hallucinations

Artifacts are inevitable when compressing below data entropy. Increasing model size (bitrate) or using external memory reduces losses, but fitting 10 TB into 70 GB will always leave gaps. Promises of a 'complete solution' ignore information theory.

Human memory is analogous: confabulation fills losses with plausible details due to storage limitations.

Key Takeaways

• Hallucinations are not errors but artifacts of lossy decompression per Shannon.
• Increasing parameters reduces but does not eliminate losses.
• RAG and fine-tuning are tools for local artifact compensation.
• Code and patterns are preserved better than exact facts.
• Temperature regulates artifact visibility, not their presence.

LLMs as artificial memory: efficient for general patterns, leaky for details. Engineering with compression nature in mind is key to reliable applications.

— Editorial Team