NLE: Non-autoregressive LLM-based ASR by Transcript Editing

TL;DR

NLE introduces a non-autoregressive speech recognition method using transcript editing with bidirectional LLMs, significantly reducing latency and enabling real-time applications while maintaining high accuracy.

Contribution

The paper presents a novel non-autoregressive ASR approach that formulates recognition as transcript editing, leveraging a bidirectional LLM and a new training strategy for parallel decoding.

Findings

Achieves 5.67% WER on Open ASR leaderboard

Provides 27x speedup over autoregressive models in single-utterance scenarios

Enables real-time speech recognition with high accuracy

Abstract

While autoregressive (AR) LLM-based ASR systems achieve strong accuracy, their sequential decoding limits parallelism and incurs high latency. We propose NLE, a non-autoregressive (NAR) approach that formulates speech recognition as conditional transcript editing, enabling fully parallel prediction. NLE extracts acoustic embeddings and an initial hypothesis from a pretrained speech encoder, then refines the hypothesis using a bidirectional LLM editor trained with a latent alignment objective. An interleaved padding strategy exploits the identity mapping bias of Transformers, allowing the model to focus on corrections rather than full reconstruction. On the Open ASR leaderboard, NLE++ achieves 5.67% average WER with an RTFx (inverse real-time factor) of 1630. In single-utterance scenarios, NLE achieves 27x speedup over the AR baseline, making it suitable for real-time applications.