Efficient Streaming LLM for Speech Recognition

TL;DR

SpeechLLM-XL is a novel linear-scaling, streaming speech recognition model that efficiently processes long audio inputs with minimal quality loss, outperforming previous methods in accuracy and computational efficiency.

Contribution

Introduces SpeechLLM-XL, a streaming speech recognition model with linear attention and chunk-based processing, enabling efficient long-form audio recognition with high accuracy.

Findings

Achieves 2.7%/6.7% WER on LibriSpeech test sets

Maintains quality on audio 10x longer than training data

Reduces computational cost with limited attention window

Abstract

Recent works have shown that prompting large language models with audio encodings can unlock speech recognition capabilities. However, existing techniques do not scale efficiently, especially while handling long form streaming audio inputs -- not only do they extrapolate poorly beyond the audio length seen during training, but they are also computationally inefficient due to the quadratic cost of attention. In this work, we introduce SpeechLLM-XL, a linear scaling decoder-only model for streaming speech recognition. We process audios in configurable chunks using limited attention window for reduced computation, and the text tokens for each audio chunk are generated auto-regressively until an EOS is predicted. During training, the transcript is segmented into chunks, using a CTC forced alignment estimated from encoder output. SpeechLLM-XL with 1.28 seconds chunk size achieves 2.7%/6.7% WER on LibriSpeech test clean/other, and it shows no quality degradation on long form utterances 10x longer than the training utterances.