Modern AI's "Speculative Decoding": How to Double the Speed of Large Models Without Sacrificing Quality?

In the pursuit of faster inference speeds for Large Language Models (LLMs), we typically face a harsh trade-off: either use smaller models (sacrificing quality) or invest in expensive hardware (sacrificing cost). However, a technique known as Speculative Decoding is quietly changing this landscape.

Put simply, the core logic of speculative decoding is: use a "fast but dumb" small model to make predictions, and a "slow but smart" large model to verify them.

Why is LLM Generation Slow?

To understand speculative decoding, we first need to identify the bottleneck in LLM inference.

LLM text generation is "autoregressive"—for every token generated, all previous tokens must be re-input into the model for computation. For large models like Llama-3-70B, generating each character requires loading massive weight matrices into VRAM. At this point, the bottleneck lies not in computational power (TFLOPS), but in Memory Bandwidth. In other words, while the GPU computes quickly, the time spent moving weights from VRAM to the compute units is excessively long.

Regardless of whether you predict the next character to be "the" or "apple," the overhead of moving weights remains exactly the same.

How Speculative Decoding Works

Speculative decoding introduces an auxiliary role: the Draft Model. This is a model with extremely few parameters and very fast inference speed (for example, using Llama-3-8B as a draft model for Llama-3-70B).

The entire process consists of two stages:

1. Speculation Phase

The draft model runs independently for $K$ steps (e.g., $K=5$). It rapidly predicts the next 5 tokens. Since the small model has fewer weights, they can be loaded extremely quickly, allowing these 5 words to be generated in a very short time.
* Draft Prediction: "Today / the / weather / is / nice"

2. Verification Phase

The large model (target model) takes the stage. The key point is that the large model can parallelize the verification of these 5 tokens in a single pass.
The large model performs a probability check on each prediction made by the draft model:
* 1st word "Today" $\rightarrow$ Large model deems it correct $\checkmark$
* 2nd word "the" $\rightarrow$ Large model deems it correct $\checkmark$
* 3rd word "weather" $\rightarrow$ Large model deems it correct $\checkmark$
* 4th word "is" $\rightarrow$ Large model thinks it should be "looks" $\times$ (Reject)

As soon as an inconsistency is detected, the large model immediately discards that word and all subsequent predictions, providing the correct corrected word (in this case, "looks").

Result:

In this cycle, we effectively obtained $3+1=4$ tokens for the cost of one large model inference. If the draft model is sufficiently accurate, the larger the $K$ value, the higher the acceleration ratio.

Why Does This Work? (Mathematical Equivalence)

Many developers worry: Will using a small model for preliminary predictions degrade generation quality?

The answer is: Not at all.

Speculative decoding is mathematically designed to be completely equivalent to sampling directly from the large model. The verification phase employs a special rejection sampling mechanism. If the distribution of the large model differs from that of the small model, it will resample a word based on the probability distribution. The final output token distribution is identical to running the large model directly, with no loss in precision.

Challenges in Practical Application

Although theoretically perfect, there are challenges in engineering implementation:
1. Distribution Alignment: If the draft model is too "dumb" (low prediction accuracy), it leads to frequent rejections and rewrites, potentially making it slower than running the large model alone. Therefore, the draft model needs to be highly aligned with the target model on the dataset.
2. KV Cache Management: When verification fails and rolls back, the truncation and updating of the KV Cache must be managed efficiently.
3. Hardware Scheduling: Execution flows for both small and large models need to be switched efficiently on the same GPU.

Conclusion

Speculative decoding transforms LLM inference from a pure "computational problem" into a "probabilistic game." It leverages the redundancy inherent in language—many simple connecting words and phrases do not require top-tier intelligence to predict. Through this "fast-slow combined" architecture, we can significantly improve AI response speeds without changing weights or compromising quality.