Speculative Decoding

Technical Overview of Speculative Decoding

Speculative Decoding is an inference acceleration technique that speeds up LLM token generation by using a small draft model to generate candidate tokens that are verified in parallel by a larger target model.

LLM inference is memory-bandwidth bound: generating each token requires loading all model parameters into GPU SRAM, regardless of the calculation size. Speculative Decoding addresses this by using a fast, low-parameter draft model to generate a sequence of tokens, which the larger target model validates in a single forward pass, reducing memory transfer overhead.

Key Architecture & Implementation

The process runs in a loop:

1. Draft Generation: A small draft model (e.g., 1B parameters) autoregressively generates $K$ candidate tokens. This step is fast due to the model’s small memory footprint.
2. Parallel Verification: The large target model (e.g., 70B parameters) evaluates all $K$ candidate tokens in a single forward pass, calculating the probability distribution for each position.
3. Acceptance Phase: The system accepts candidate tokens based on probability thresholds. If a token is rejected, the target model’s output at that position is used to generate the corrected token, and the draft model restarts from that point.

Draft Model (1B):  Token 1 --> Token 2 --> Token 3 (Fast Draft)
                                              |
Target Model (70B): Validate [1, 2, 3] in 1 step ---> Accept 1 & 2, Reject 3 (Correct and Restart)

Core Parameters

• Speedup Ratio: Typically yields a 1.5x to 2.5x latency improvement, depending on the similarity of the draft and target models.
• Acceptance Rate (lpha): The average fraction of draft tokens accepted by the target model.

Real-world Applications

• Used in low-latency production applications, such as code generation and chat interfaces.

Why It Speeds Up Generation Without Changing Output

The key insight behind speculative decoding is that verifying several candidate tokens in parallel with the large target model is much cheaper than generating those same tokens one at a time, since generating each token sequentially is bottlenecked by memory bandwidth (loading the model’s weights from memory for each step) rather than raw computation — verifying multiple draft tokens in a single parallel pass uses that same memory bandwidth far more efficiently. When the small draft model’s guesses are accepted, generation effectively jumps forward multiple tokens per target-model pass instead of one; when a guess is rejected, the target model’s own correct token is used instead, guaranteeing the output is mathematically identical to what the target model would have generated on its own, just faster. Speedups of 2-3x are common in practice, with the exact gain depending on how well the draft model’s outputs tend to align with the target model’s.