Writing tokens one by one is slow because each token waits on a full pass through the model. Speculative decoding speeds this up with a second, much smaller draft model from the same family. The draft guesses the next several tokens cheaply, and the big model checks all of the guesses in a single pass. Right guesses are kept; at the first wrong one the big model supplies its own token. The output is the same as the big model alone, only faster.
When it helps
- Long replies and code, where the next tokens are predictable.
- Large models that already fit entirely in fast memory. Reported gains are typically in the range of 20 to 50 percent.
- A draft from the same family as the target, because they predict alike.
When it does not
- Short replies, where there is little to speed up.
- Small target models, which are already quick.
- Heavy offload, where the slow part is moving data from RAM (why).
- Creative output at a high temperature, because the draft guesses wrong more often.
- Prompt reading. Only generation speeds up (two speeds).
Cost and how to try it
The draft model takes a little extra memory, usually under a gigabyte for a small one, so it can tip a tight fit over the edge. LM Studio has a draft-model option in its model settings, and llama.cpp’s server accepts a draft model with -md. Support is app- and version-dependent, and some apps built on llama.cpp do not expose it, so check the app first. Pick a draft at a higher quantization than you would use for the big model, because a poor draft lowers the hit rate.