Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling

TL;DR

Cactus introduces a new speculative sampling method that accelerates auto-regressive decoding by controlling divergence from the verifier distribution, improving acceptance rates without sacrificing output quality.

Contribution

We formalize speculative sampling as a constrained optimization problem and propose Cactus, which guarantees controlled divergence and higher acceptance rates during decoding.

Findings

Cactus achieves higher acceptance rates in decoding.

Empirical results show improved decoding speed without quality loss.

The method is effective across various benchmarks.

Abstract

Speculative sampling (SpS) has been successful in accelerating the decoding throughput of auto-regressive large language models by leveraging smaller draft models. SpS strictly enforces the generated distribution to match that of the verifier LLM. This is unnecessarily restrictive as slight variations of the verifier's distribution, such as sampling with top-$k$ or temperature, would also be acceptable. Typical acceptance sampling (TAS) alleviates this issue by accepting more tokens using entropy-based heuristics. However, this approach distorts the verifier distribution, potentially degrading output quality when the verifier encodes critical information. In this work, we formalize the speculative sampling algorithm through the lens of constrained optimization. Based on this formulation, we propose Cactus (constrained acceptance speculative sampling), a method that guarantees controlled divergence from the verifier distribution and increasing acceptance rates. Empirical results across a wide range of benchmarks confirm the effectiveness of our approach.