LLM attention problems in long context

Why LLMs Forget Facts in Long Contexts: Attention and Hallucinations Explained

Transformer-based LLMs suffer from attention degradation in long contexts: primacy and recency bias weaken the processing of the middle of the text. This leads to the Lost-in-the-Middle effect, increased hallucinations, and loss of accuracy even with million-token windows. This article covers the causes of the problem and techniques for production RAG systems.

Problems with Self-Attention on Long Sequences

Self-attention in transformers has quadratic complexity O(n²), which limits context length. Optimizations like FlashAttention, RoPE, and ALiBi reduce costs but introduce positional biases:

• Primacy bias: The beginning of the context is remembered best.
• Recency bias: The end of the context is also accessible.
• The middle gets lost: Tokens in the center receive weakened attention due to causal masking and decay in positional embeddings.

Increasing the context window does not eliminate bias; it expands the 'dead zone' in the middle. The model does not acknowledge gaps but generates plausible yet false facts.

The Lost-in-the-Middle Effect in Numbers

Research by Nelson Liu (Stanford, 2023) showed a U-shaped accuracy curve: a 30–50% drop for information in the middle of 20–30 documents. In 2025–2026, MIT and other labs confirmed the persistence of the effect in models with 1M+ tokens.

Benchmarks have evolved:

• Needle-in-a-Haystack: A test to find a fact in a long text.
• U-NIAH (2026): Compares long-context LLMs and RAG.
• NeedleBench, BABILong, LooGLE: Tests synthesis from scattered parts.

The Vectara Hallucination Leaderboard (HHEM-2.3, 2026) records a rise in hallucinations at 32K+ tokens: the model links fragments A and C, ignoring B, and invents what's missing.

The Mechanism of Hallucinations in Long Contexts

OpenAI (2025, 'When More Becomes Less') noted: larger context increases inference but reduces quality. The model fills gaps with 'plausible' connections. MIT (2025) added: language confidence grows proportionally to hallucinations—the fewer facts, the more convincing the fiction.

The chain: long context → noise → blurred attention → missing fact → RLHF training to 'always respond' → creative hallucination.

Chunking Strategies for RAG

Effective RAG starts with proper document splitting. Avoid default splitters—use targeted approaches:

• RecursiveCharacterTextSplitter (LangChain): Chunks of 400–512 tokens, overlap 10–20%. Reliable for structured texts.
• SemanticChunker: Splitting by embeddings, +15–20% accuracy on complex data.
• HierarchicalNodeParser (LlamaIndex): Hierarchy of 2048 → 512 → 128 tokens for staged retrieval.

Pass the model 3–5 relevant chunks instead of 100K tokens of junk.

Example implementation of chunking with reranking:

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.retrievers import ContextualCompressionRetriever
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS

# Recursive + overlap
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " ", ""]
)

# Semantic splitter
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2")
semantic_splitter = SemanticChunker(embeddings)

# Reranking
compressor = CrossEncoderReranker(model_name="BAAI/bge-reranker-large")
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=vectorstore.as_retriever(search_kwargs={"k": 20})
)

Retrieve 20 chunks, reranker keeps top-5—reduces noise.

Verification and Agentic Workflows

After retrieval, add verification layers:

• Self-Consistency / CoVe: Generate an answer, then questions for self-checking.
• Critic Agent: A separate LLM validates the answer against context.
• Symbolic verification: Knowledge Graph or Pydantic for dates, numbers, names.

Agentic workflow (2026 standard):

• Planner: Breaks down the task.
• Retriever Agent: Searches for subtasks.
• Executor: Generates.
• Verifier: Iterative checking.

Minimal verification loop:

from langchain.agents import create_react_agent, AgentExecutor
from langchain.tools import Tool

def verify_answer(answer: str, context: str) -> str:
    prompt = (
        "Check if the answer matches the context. "
        "Respond YES/NO + explanation.\n"
        f"Context: {context}\n"
        f"Answer: {answer}"
    )
    return llm.invoke(prompt)

tools = [Tool(name="Verifier", func=verify_answer, description="Checks if the answer matches the context")]
agent = create_react_agent(llm, tools, prompt=react_prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, max_iterations=5)

Hybrid Neuro-Symbolic for High-Stakes Domains

In law, medicine, finance, combine LLM + Knowledge Graph + symbolic verifier. Reduces hallucinations by 60–80% (2025 studies), but requires infrastructure.

Key takeaways:

• Long context amplifies Lost-in-the-Middle and hallucinations due to attention bias.
• Chunking (recursive/semantic) + reranking—70% of RAG success.
• Verification layers (CoVe, critic agents) are essential in production.
• Agentic workflows pay off with reputation risks.
• Hybrid approach for regulated domains.

— Editorial Team