Diffusion Language Models for Speech Recognition

TL;DR

This paper explores the application of diffusion language models in speech recognition, introducing new methods for hypothesis rescoring and joint decoding to improve accuracy.

Contribution

It introduces masked and uniform-state diffusion language models for speech hypothesis rescoring and proposes a novel joint-decoding method combining CTC and USDM.

Findings

USDM and MDLM significantly improve recognition accuracy

The joint-decoding method effectively combines language and acoustic information

All code and recipes are publicly available

Abstract

Diffusion language models have recently emerged as a leading alternative to standard language models, due to their ability for bidirectional attention and parallel text generation. In this work, we explore variants for their use in speech recognition. Specifically, we introduce a comprehensive guide to incorporating masked diffusion language models (MDLM) and uniform-state diffusion models (USDMs) for rescoring ASR hypotheses. Additionally, we design a new joint-decoding method that combines CTC and USDM by integrating the framewise probability distributions derived from CTC with the labelwise probability distributions computed by USDM at each decoding step, thereby generating new candidates that combine strong language knowledge from USDM and acoustic information from CTC. Our findings reveal that USDM, as well as MDLM, can significantly improve the accuracy of recognized text. We publish all our code and recipes.