MFLA: Monotonic Finite Look-ahead Attention for Streaming Speech Recognition

TL;DR

This paper introduces MFLA, a novel attention mechanism for streaming speech recognition that balances latency and accuracy by combining monotonic alignment with finite look-ahead attention, enabling efficient real-time transcription.

Contribution

It proposes Monotonic Finite Look-ahead Attention and a prefix-to-prefix training framework to improve streaming speech recognition with large pre-trained models like Whisper.

Findings

Achieves a controllable trade-off between latency and recognition quality.

Demonstrates effective quasi-monotonic alignment between speech and text.

Simplifies decoding with wait-k strategy while maintaining accuracy.

Abstract

Applying large pre-trained speech models like Whisper has shown promise in reducing training costs for various speech tasks. However, integrating these models into streaming systems remains a challenge. This paper presents a novel prefix-to-prefix training framework for streaming recognition by fine-tuning the Whisper. We introduce the Continuous Integrate-and-Fire mechanism to establish a quasi-monotonic alignment between continuous speech sequences and discrete text tokens. Additionally, we design Monotonic Finite Look-ahead Attention, allowing each token to attend to infinite left-context and finite right-context from the speech sequences. We also employ the wait-k decoding strategy to simplify the decoding process while ensuring consistency between training and testing. Our theoretical analysis and experiments demonstrate that this approach achieves a controllable trade-off between latency and quality, making it suitable for various streaming applications.