Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoff

TL;DR

Fuzzy Speculative Decoding (FSD) introduces a tunable decoding method that balances inference speed and generation quality by allowing controlled divergence from the target model, outperforming traditional SD in efficiency.

Contribution

FSD generalizes Speculative Decoding by enabling a flexible trade-off between accuracy and speed through divergence control, and can be integrated into existing decoding frameworks.

Findings

FSD achieves over 5 tokens/sec faster than SD with only 2% accuracy loss.

FSD matches SD accuracy at higher speeds, showing distributional equivalence isn't necessary.

FSD can be integrated into existing extensions like EAGLE-2 to improve efficiency.

Abstract

Speculative Decoding (SD) enforces strict distributional equivalence to the target model when accepting candidate tokens. While it maintains the target model's generation quality, this strict equivalence limits the speedup achievable by SD and prevents users from trading deviations from the target distribution in exchange for further inference speed gains. To address these limitations, we introduce Fuzzy Speculative Decoding (FSD) - a decoding algorithm that generalizes SD by accepting candidate tokens based on the divergences between the target and draft model distributions. By allowing for controlled divergence from the target model, FSD enables users to flexibly trade generation quality for inference speed. Across several benchmarks, our method is able to achieve significant runtime improvements of over 5 tokens per second faster than SD at only an approximate 2% absolute reduction in benchmark accuracy. In many cases, FSD is even able to match SD benchmark accuracy at over 2 tokens per second faster, demonstrating that distributional equivalence is not necessary to maintain target model performance. Furthermore, FSD can be seamlessly integrated into existing SD extensions; we demonstrate this by applying FSD to EAGLE-2, greatly enhancing this existing extension's efficiency while allowing it to leverage FSD's tunable quality-speed trade-off.