Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding

TL;DR

This survey reviews speculative decoding techniques for large language models, highlighting how they improve inference efficiency by combining draft generation with verification, and discusses future research directions.

Contribution

It categorizes and analyzes various speculative decoding methods, providing a comprehensive overview and guiding future research in scalable LLM inference.

Findings

Speculative decoding significantly reduces inference time.

Draft-centric and model-centric approaches offer different trade-offs.

Potential for scaling LLM inference in real-world applications.

Abstract

Efficient inference in large language models (LLMs) has become a critical focus as their scale and complexity grow. Traditional autoregressive decoding, while effective, suffers from computational inefficiencies due to its sequential token generation process. Speculative decoding addresses this bottleneck by introducing a two-stage framework: drafting and verification. A smaller, efficient model generates a preliminary draft, which is then refined by a larger, more sophisticated model. This paper provides a comprehensive survey of speculative decoding methods, categorizing them into draft-centric and model-centric approaches. We discuss key ideas associated with each method, highlighting their potential for scaling LLM inference. This survey aims to guide future research in optimizing speculative decoding and its integration into real-world LLM applications.