AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference

TL;DR

AdaSD introduces a hyperparameter-free, adaptive decoding method that dynamically adjusts inference parameters, significantly speeding up large language model processing with minimal accuracy loss.

Contribution

It proposes a novel adaptive speculative decoding scheme that eliminates the need for pre-tuning, enhancing inference efficiency for large language models.

Findings

Achieves up to 49% speedup over standard speculative decoding.

Limits accuracy degradation to under 2%.

Compatible with off-the-shelf models.

Abstract

Large language models (LLMs) have achieved remarkable performance across a wide range of tasks, but their increasing parameter sizes significantly slow down inference. Speculative decoding mitigates this issue by leveraging a smaller draft model to predict candidate tokens, which are then verified by a larger target model. However, existing approaches often require additional training, extensive hyperparameter tuning, or prior analysis of models and tasks before deployment. In this paper, we propose Adaptive Speculative Decoding (AdaSD), a hyperparameter-free decoding scheme that dynamically adjusts generation length and acceptance criteria during inference. AdaSD introduces two adaptive thresholds: one to determine when to stop candidate token generation and another to decide token acceptance, both updated in real time based on token entropy and Jensen-Shannon distance. This approach eliminates the need for pre-analysis or fine-tuning and is compatible with off-the-shelf models. Experiments on benchmark datasets demonstrate that AdaSD achieves up to 49\% speedup over standard speculative decoding while limiting accuracy degradation to under 2\%, making it a practical solution for efficient and adaptive LLM inference.