Chunk-wise Attention Transducers for Fast and Accurate Streaming Speech-to-Text

TL;DR

The paper introduces Chunk-wise Attention Transducer (CHAT), a streaming speech-to-text model that improves efficiency and accuracy by processing audio in fixed chunks with cross-attention, reducing memory and computation while enhancing performance.

Contribution

CHAT extends RNN-T models with chunk-wise cross-attention, enabling faster training and inference while maintaining streaming capabilities and improving accuracy across languages and tasks.

Findings

Up to 46.2% reduction in peak training memory

Up to 1.36X faster training

Up to 1.69X faster inference

Abstract

We propose Chunk-wise Attention Transducer (CHAT), a novel extension to RNN-T models that processes audio in fixed-size chunks while employing cross-attention within each chunk. This hybrid approach maintains RNN-T's streaming capability while introducing controlled flexibility for local alignment modeling. CHAT significantly reduces the temporal dimension that RNN-T must handle, yielding substantial efficiency improvements: up to 46.2% reduction in peak training memory, up to 1.36X faster training, and up to 1.69X faster inference. Alongside these efficiency gains, CHAT achieves consistent accuracy improvements over RNN-T across multiple languages and tasks -- up to 6.3% relative WER reduction for speech recognition and up to 18.0% BLEU improvement for speech translation. The method proves particularly effective for speech translation, where RNN-T's strict monotonic alignment hurts performance. Our results demonstrate that the CHAT model offers a practical solution for deploying more capable streaming speech models without sacrificing real-time constraints.