Speculative decoding accelerates inference by letting a small fast 'draft' model propose several tokens that the large model then verifies in one pass: accepted
An inference acceleration technique where a small fast 'draft' model proposes several tokens that the large model then verifies in one pass: accepted tokens ship, rejected ones fall back to normal generation. Output is identical to the large model's; it just arrives faster.
It typically yields 2-3x speedups on text-heavy generation without any quality trade, since the output is verified against the large model. It is one of the quiet serving optimizations behind 2026's latency improvements.