Self-Speculative Decoding for LLM-based ASR with CTC Encoder Drafts

TL;DR

This paper introduces self-speculative decoding for speech recognition using LLMs and CTC encoders, significantly speeding up inference and reducing word error rate across multiple languages.

Contribution

It presents a novel three-step decoding method combining CTC and LLMs to accelerate ASR inference and improve accuracy, with extensive multi-language experiments.

Findings

Achieves 5.58% WER on HuggingFace Open ASR benchmark

Speeds up decoding by 4.4 times with minimal WER increase

Demonstrates effectiveness across nine corpora and five languages

Abstract

We propose self-speculative decoding for speech-aware LLMs by using the CTC encoder as a draft model to accelerate auto-regressive (AR) inference and improve ASR accuracy. Our three-step procedure works as follows: (1) if the frame entropies of the CTC output distributions are below a threshold, the greedy CTC hypothesis is accepted as final; (2) otherwise, the CTC hypothesis is verified in a single LLM forward pass using a relaxed acceptance criterion based on token likelihoods; (3) if verification fails, AR decoding resumes from the accepted CTC prefix. Experiments on nine corpora and five languages show that this approach can simultaneously accelerate decoding and reduce WER. On the HuggingFace Open ASR benchmark with a 1B parameter LLM and 440M parameter CTC encoder, we achieve a record 5.58% WER and improve the inverse real time factor by a factor of 4.4 with only a 12% relative WER increase over AR search. Code and model weights are publicly available under a permissive license.