Prompt Caching LLM: KV cache 10 times cheaper

Prompt Caching in LLMs: Optimizing the KV Cache to Reduce Costs and Latency

Prompt Caching reduces LLM input token costs by 10x and cuts latency by up to 85% for long prompts. OpenAI and Anthropic don't cache responses; they cache intermediate computations in the transformer attention mechanism—specifically the KV cache. This stores key-value pairs for repeated prompt prefixes, speeding up inference without sacrificing generation quality.

The technology is ideal for tasks with repeated context: RAG, chatbots with conversation history, multi-step pipelines. Tests on API providers show that fully utilizing the cache delivers a significant speed boost to the first token.

LLM Architecture and the Role of Attention

An LLM is a sequence of transformer blocks, each taking embeddings as input and outputting updated representations. Inference runs in a loop: the prompt is tokenized, embeddings pass through attention and feed-forward layers, the next token is generated, and it's added to the context.

prompt = "What is the meaning of life?";
tokens = tokenizer(prompt);
while (true) {
    embeddings = embed(tokens);
    for ([attention, feedforward] of transformers) {
        embeddings = attention(embeddings);
        embeddings = feedforward(embeddings);
    }
    output_token = output(embeddings);
    if (output_token === END_TOKEN) {
        break;
    }
    tokens.push(output_token);
}
print(decode(tokens));

Caching occurs in attention. Self-attention computes a weighted sum of previous tokens for each token based on similarity between the query (current token) and keys (previous tokens). For query_i: attention(Q_i, K, V) = softmax(Q_i K^T / sqrt(d)) V.

When generating each new token, the KV cache for all previous tokens expands, leading to quadratic complexity O(n^2) with respect to sequence length.

Tokenization and Embeddings

The tokenizer breaks text into subword units and assigns IDs. Example for GPT: "Check out ngrok.ai" → [4383, 842, 1657, 17690, 75584]. It's deterministic, case-sensitive, and uses BPE-like algorithms.

Embeddings transform tokens into n-dimensional vectors (n > 10k in large models). The embeddings matrix is fixed after training:

function embed(tokens) {
    return tokens.map((token, i) => {
        const embeddings = EMBEDDINGS[token];
        return encodePosition(embeddings, i);
    });
}

Positional encoding adds information about order. More dimensions better capture semantics: tone, style, similarity.

Attention Mechanism and KV Cache

In multi-head attention:

• Projections: Q = X W_Q, K = X W_K, V = X * W_V (X — embeddings matrix).
• Scores: attention_scores = Q * K^T / sqrt(d_k).
• Softmax and weighted sum: attention_output = softmax(scores) * V.

For a sequence of length n, the KV cache consists of K and V tensors sized [n, d_model]. When generating the t-th token, attention uses the entire past cache plus the current one.

Prompt Caching saves the prompt prefix's KV cache across API calls. If a new prompt starts with a cached prefix (hit), attention computations for those tokens are skipped—the ready KV is used instead. Costs drop because ~90% of compute is skipped for long contexts.

• Cache hit: prefix > 1024 tokens (Anthropic), cost /10.
• Cache miss: full recompute.
• TTL: cache lives ~5-10 min, provider-dependent.

Benefits and Limitations

Latency gains: for prompts with 10k+ tokens, TTFB drops by 50-85%.

Savings: cached input tokens — $0.0001/1k vs $0.001/1k (approx).

Limitations:

• Prefix-only matching (not arbitrary substrings).
• Doesn't cache output embeddings or logits.
• Depends on the provider's tokenizer.

In production: use for session-based chats where history repeats.

Key Takeaways

• Prompt Caching caches attention KV pairs for prefixes, slashing compute by 90%+ for long contexts.
• Time to first token drops by up to 85%, input costs /10.
• Requires exact prefix match and compatible tokenizer.
• Perfect for RAG, chat histories, multi-iteration tasks.
• No impact on quality: generation remains stochastic.

— Editorial Team