Speculative Decoding and Beyond: An In-Depth Survey of Techniques

TL;DR

This paper provides a comprehensive survey of generation-refinement techniques for autoregressive models, analyzing their strategies, implementations, and applications to improve efficiency in real-time generation tasks.

Contribution

It offers a detailed taxonomy and systematic analysis of generation-refinement frameworks, highlighting recent advances and deployment strategies for autoregressive decoding.

Findings

Generation-refinement frameworks significantly mitigate quality trade-offs.

Categorization of methods based on generation and refinement strategies.

Analysis of deployment across diverse computing environments.

Abstract

Sequential dependencies present a fundamental bottleneck in deploying large-scale autoregressive models, particularly for real-time applications. While traditional optimization approaches like pruning and quantization often compromise model quality, recent advances in generation-refinement frameworks demonstrate that this trade-off can be significantly mitigated. This survey presents a comprehensive taxonomy of generation-refinement frameworks, analyzing methods across autoregressive sequence tasks. We categorize methods based on their generation strategies (from simple n-gram prediction to sophisticated draft models) and refinement mechanisms (including single-pass verification and iterative approaches). Through systematic analysis of both algorithmic innovations and system-level implementations, we examine deployment strategies across computing environments and explore applications spanning text, images, and speech generation. This systematic examination of both theoretical frameworks and practical implementations provides a foundation for future research in efficient autoregressive decoding.