Cascade Speculative Drafting for Even Faster LLM Inference

TL;DR

This paper proposes Cascade Speculative Drafting, an improved speculative decoding method for large language models that significantly increases inference speed by eliminating autoregressive generation and optimizing token drafting time.

Contribution

It introduces a novel cascade-based speculative decoding algorithm that removes autoregressive generation and optimizes token drafting, achieving faster inference without changing output distribution.

Findings

Achieves greater speedup than baseline methods

Preserves the same output distribution as the target model

Effectively reduces inference time in large language models

Abstract

Introduced to enhance the efficiency of large language model (LLM) inference, speculative decoding operates by having a smaller model generate a draft. A larger target model then reviews this draft to align with its output, and any acceptance by the target model results in a reduction of the number of the target model runs, ultimately improving efficiency. However, the drafting process in speculative decoding includes slow autoregressive generation and allocates equal time to generating tokens, irrespective of their importance. These inefficiencies collectively contribute to the suboptimal performance of speculative decoding. To further improve LLM inference, we introduce Cascade Speculative Drafting (CS Drafting), a speculative execution algorithm that incorporates two types of cascades. The Vertical Cascade eliminates autoregressive generation from neural models, while the Horizontal Cascade optimizes time allocation in drafting for improved efficiency. Combining both cascades, CS Drafting achieves greater speedup compared to the baselines in our experiments, while preserving the same output distribution as the target model.