The "Inference Speed" of Modern AI: The Engineering Truth Behind KV Cache

In LLM brochures, we often see speed metrics like "100 tokens generated per second." However, for developers, the core determinant of this speed is not GPU compute power (TFLOPS), but an extremely critical memory optimization mechanism: KV Cache (Key-Value Cache).

If you notice fast time-to-first-token but slow subsequent generation when calling an API, or if you find that VRAM is rapidly exhausted when deploying models, you are effectively dealing with the KV Cache.

Why Do We Need KV Cache?

To understand KV Cache, one must first understand the autoregressive nature of Transformers.

LLMs generate text token by token. When you input "The weather today," the model predicts "is"; then, taking "The weather today is" as input, it predicts "good."

When computing the $N$-th token, the model needs to calculate the attention between the current token and all previous $N-1$ tokens. This implies:
1. Redundant Computation: Without caching, every time a new token is generated, the $Q$ (Query), $K$ (Key), and $V$ (Value) vectors for all preceding tokens must be recalculated.
2. Complexity Explosion: The computational load grows quadratically with sequence length, $\mathcal{O}(N^2)$.

The core logic of KV Cache is simple: Since previous tokens do not change, their $K$ and $V$ vectors remain identical in each iteration. We simply store them and reuse them directly in the next step.

How Does KV Cache Work?

During inference, the model operates in two phases:

1. Prefill Phase

When you send a prompt, the model processes all input tokens at once. At this stage, it computes the $K$ and $V$ vectors for all input tokens and writes them into the KV Cache area in VRAM. This phase is compute-bound because it can leverage the GPU's parallel processing capabilities.

2. Decoding Phase

When generating each new token, the model only needs to compute the $Q, K, V$ for this single new token. It then appends this new $K, V$ to the cache and uses the historical $K, V$ stored in the cache to calculate attention weights. This phase is memory-bandwidth-bound because the GPU spends most of its time waiting to read the massive KV Cache matrices from VRAM.

The Harsh Engineering Cost: VRAM Pressure

While KV Cache eliminates redundant computation, it introduces significant memory overhead.

The size of the KV Cache depends on: $\text{Layers} \times \text{Heads} \times \text{Dimension} \times \text{Sequence Length} \times \text{Precision}$.
Taking Llama-3-8B (FP16 precision) as an example:
- Each additional token consumes approximately hundreds of KB to several MB of VRAM per request.
- As concurrent users increase or the context window expands to 128K, the KV Cache can quickly exhaust the VRAM of A100/H100 GPUs, leading to OOM (Out of Memory) errors.

How to Optimize KV Cache? (Industry Solutions)

To support longer contexts and higher concurrency without sacrificing performance, the industry has adopted three mainstream approaches:

1. MQA / GQA (Multi-Query / Grouped-Query Attention)

This reduces cache volume at the architectural level. Traditional MHA assigns a corresponding Key/Value head to each Query head; in contrast, GQA allows multiple Query heads to share a single set of KV heads. This directly compresses the KV Cache size by several folds (for instance, Llama-3 utilizes GQA).

2. PagedAttention (vLLM)

This is currently the most prevalent system-level optimization. Traditional KV Cache requires contiguous memory space, leading to severe fragmentation (similar to early operating system memory management). PagedAttention stores the KV Cache in non-contiguous physical blocks using paging, implementing a virtual memory-like management scheme that significantly improves throughput and reduces waste.

3. Quantization

Quantizing the KV Cache from FP16 to INT8 or FP8. This can halve VRAM usage with negligible impact on model accuracy.

Conclusion

KV Cache is the key engineering cornerstone that transformed LLM inference from a "lab toy" into an "industrial product." It converts $\mathcal{O}(N^2)$ redundant computations into an $\mathcal{O}(N)$ space-for-time strategy. When we discuss AI inference costs and latency, we are essentially discussing how to manage this expensive VRAM cache more efficiently.