dLLM-ASR: A Faster Diffusion LLM-based Framework for Speech Recognition

TL;DR

This paper introduces dLLM-ASR, a novel diffusion LLM-based speech recognition framework that significantly speeds up inference while maintaining high accuracy by using prior-guided, adaptive denoising techniques.

Contribution

It presents a new ASR framework that adapts diffusion LLMs with prior guidance and adaptive denoising, reducing redundancy and inference time.

Findings

Achieves comparable accuracy to autoregressive LLM-based ASR systems.

Provides a 4.44× inference speedup over traditional methods.

Demonstrates effective adaptive computation at token level.

Abstract

Automatic speech recognition (ASR) systems based on large language models (LLMs) achieve superior performance by leveraging pretrained LLMs as decoders, but their token-by-token generation mechanism leads to inference latency that grows linearly with sequence length. Meanwhile, discrete diffusion large language models (dLLMs) offer a promising alternative, enabling high-quality parallel sequence generation with pretrained decoders. However, directly applying native text-oriented dLLMs to ASR leads to a fundamental mismatch between open-ended text generation and the acoustically conditioned transcription paradigm required by ASR. As a result, it introduces unnecessary difficulty and computational redundancy, such as denoising from pure noise, inflexible generation lengths, and fixed denoising steps. We propose dLLM-ASR, an efficient dLLM-based ASR framework that formulates dLLM's decoding as a prior-guided and adaptive denoising process. It leverages an ASR prior to initialize the denoising process and provide an anchor for sequence length. Building upon this prior, length-adaptive pruning dynamically removes redundant tokens, while confidence-based denoising allows converged tokens to exit the denoising loop early, enabling token-level adaptive computation. Experiments demonstrate that dLLM-ASR achieves recognition accuracy comparable to autoregressive LLM-based ASR systems and delivers a 4.44$\times$ inference speedup, establishing a practical and efficient paradigm for ASR.