The “KV Cache” in Modern AI: Why LLMs Get Slower and Exhaust VRAM During Long Conversations

When engaging in long conversations with Large Language Models (LLMs), you may notice two phenomena: first, the generation speed gradually slows down as the conversation lengthens; second, VRAM usage keeps climbing, even though the model size itself remains unchanged. The core mechanism behind this is the KV Cache (Key-Value Cache).

What is KV Cache?

To understand KV Cache, one must first grasp the Attention mechanism in Transformers. When generating each new token, the model needs to review all previously generated tokens.

During attention computation, each token is transformed into three vectors: Query (Q), Key (K), and Value (V).
- Query: What the current token is “looking for.”
- Key: What information previous tokens “can provide.”
- Value: The actual content contained in previous tokens.

The computation process involves taking the dot product of the current $Q$ with the $K$ vectors of all previous tokens to calculate weights, and then performing a weighted sum of the $V$ vectors based on these weights.

The key point is: For already generated tokens, the $K$ and $V$ vectors remain completely unchanged in subsequent generation steps. If we had to recompute the $K$ and $V$ vectors for all previous tokens every time a new token is generated, the computational cost would grow quadratically with length ($O(n^2)$), leading to massive resource waste.

To optimize this, engineers store these already computed $K$ and $V$ vectors in VRAM. This is the KV Cache. During the next generation step, the system directly reads the old $K$ and $V$ from memory and only computes the $K$ and $V$ for the current new token.

The Cost of KV Cache: The “Invisible Killer” of VRAM

Although KV Cache significantly boosts inference speed (reducing complexity to linear), it introduces a huge spatial cost.

1. VRAM Usage Formula

The size of the KV Cache depends on:
- Number of Layers
- Hidden Dimension
- Number of Attention Heads
- Sequence Length
- Precision (e.g., FP16/BF16)

A simple estimation formula (for standard Transformers):
$\text{Memory} = 2 \times \text{Layers} \times \text{Heads} \times \text{Dim_per_head} \times \text{Seq_len} \times \text{Bytes_per_param}$

For example, consider a Llama-3-8B model (32 layers, 32 heads, 128 dimensions per head) using FP16 (2 bytes):
Cache required per token $\approx 2 \times 32 \times 32 \times 128 \times 2 = 524,288$ bytes $\approx 0.5\text{ MB}$.
If the context reaches 32k tokens, the KV Cache alone requires $\approx 16\text{ GB}$ of VRAM! This does not even include the model's weights.

2. The “Fragmentation” Problem

Traditional KV Cache allocation is like requesting large contiguous blocks of memory. However, since conversation lengths are unpredictable, systems often pre-allocate a buffer for the maximum possible length. This leads to severe memory fragmentation—much of the reserved space is wasted, yet when memory is truly needed, the system may still run out of memory (OOM) due to excessive fragmentation.

How to Optimize? (Industry Solutions)

To address these issues, modern AI systems employ several advanced techniques:

PagedAttention (vLLM)

This is currently the most mainstream solution. Borrowing the concept of “virtual memory” from operating systems, it divides the KV Cache into fixed-size “pages.” Instead of requiring contiguous storage, it uses page tables for mapping. This nearly eliminates memory fragmentation and allows for higher concurrency (throughput).

MQA and GQA (Multi-Query / Grouped-Query Attention)

Since $K$ and $V$ consume so much space, can we store less?
- MQA: All Query heads share a single set of $K$ and $V$ heads. VRAM usage is directly reduced to $1/\text{Heads}$ of the original. However, performance may degrade slightly.
- GQA: Query heads are grouped, with each group sharing a set of $K$ and $V$ heads (e.g., Llama-3 uses GQA). This achieves an excellent balance between performance and VRAM usage.

Quantization

Quantizing the KV Cache from FP16 to INT8 or FP8. This directly halves the cache space with negligible impact on model accuracy.

Summary

KV Cache is a “space-for-time” strategy in LLM inference. It makes real-time conversation possible but has become a bottleneck limiting context length and concurrency. From PagedAttention to GQA and quantization, one of the core goals of AI engineering is to manage this expensive VRAM cache more efficiently.