Principled Coarse-Grained Acceptance for Speculative Decoding in Speech

TL;DR

This paper introduces Principled Coarse-Graining (PCG), a novel method for speculative decoding in speech generation that improves speed by verifying proposals at the acoustic similarity group level, maintaining speech quality.

Contribution

The paper proposes PCG, a new group-level acceptance method for speculative decoding in speech LLMs, enhancing speed while preserving speech quality and intelligibility.

Findings

Increased acceptance and throughput on LibriTTS

Maintained speech intelligibility and speaker similarity

Outperformed standard speculative decoding methods

Abstract

Speculative decoding accelerates autoregressive speech generation by letting a fast draft model propose tokens that a larger target model verifies. However, for speech LLMs that generate acoustic tokens, exact token matching is overly restrictive: many discrete tokens are acoustically or semantically interchangeable, reducing acceptance rates and limiting speedups. We introduce Principled Coarse-Graining (PCG), which verifies proposals at the level of Acoustic Similarity Groups (ASGs) derived from the target model's embedding space. By splitting each token's probability mass across the overlapping groups that contain it, we define an overlap-aware coarse-grained distribution and perform rejection sampling on the resulting group variable. This yields an exactness guarantee at the group level while allowing the accepted draft token to stand in for any member of the group in practice. On LibriTTS, PCG increases acceptance and throughput relative to standard speculative decoding and prior speech-specific relaxations while maintaining intelligibility and speaker similarity. These results suggest acoustically aware, group-level acceptance as a simple and general way to accelerate speech token generation while maintaining speech quality.