Streaming Speech Recognition with Decoder-Only Large Language Models and Latency Optimization

TL;DR

This paper introduces a streaming speech recognition method using decoder-only large language models combined with a novel latency-optimized segmentation approach, achieving high accuracy and significantly reduced delay.

Contribution

It presents a new streaming ASR framework integrating a read/write policy with monotonic chunkwise attention, enabling efficient segmentation and low-latency recognition with LLMs.

Findings

Achieves 5.1% CER on AISHELL-1

Reduces token delay by 62.5%

Outperforms recent streaming ASR baselines

Abstract

Recent advances have demonstrated the potential of decoderonly large language models (LLMs) for automatic speech recognition (ASR). However, enabling streaming recognition within this framework remains a challenge. In this work, we propose a novel streaming ASR approach that integrates a read/write policy network with monotonic chunkwise attention (MoChA) to dynamically segment speech embeddings. These segments are interleaved with label sequences during training, enabling seamless integration with the LLM. During inference, the audio stream is buffered until the MoChA module triggers a read signal, at which point the buffered segment together with the previous token is fed into the LLM for the next token prediction. We also introduce a minimal-latency training objective to guide the policy network toward accurate segmentation boundaries. Furthermore, we adopt a joint training strategy in which a non-streaming LLM-ASR model and our streaming model share parameters. Experiments on the AISHELL-1 and AISHELL-2 Mandarin benchmarks demonstrate that our method consistently outperforms recent streaming ASR baselines, achieving character error rates of 5.1% and 5.5%, respectively. The latency optimization results in a 62.5% reduction in average token generation delay with negligible impact on recognition accuracy