WhisperRT -- Turning Whisper into a Causal Streaming Model

TL;DR

This paper transforms the Whisper ASR model into a low-latency streaming system by making the encoder causal and fine-tuning the alignment, achieving better performance with lower complexity.

Contribution

It introduces a method to adapt transformer-based ASR models for streaming by causal encoding and alignment fine-tuning, with an improved inference mechanism.

Findings

Fine-tuned causal models outperform non-fine-tuned streaming approaches.

The approach achieves low latency (<300 ms) with lower complexity.

The method is validated through experiments on real-time audio chunks.

Abstract

Automatic Speech Recognition (ASR) has seen remarkable progress, with models like OpenAI Whisper and NVIDIA Canary achieving state-of-the-art (SOTA) performance in offline transcription. However, these models are not designed for streaming (online or real-time) transcription, due to limitations in their architecture and training methodology. We propose a method to turn the transformer encoder-decoder model into a low-latency streaming model. The encoder is made causal to process audio incrementally, while the decoder conditions on partial encoder states to generate tokens aligned with the available temporal context. This requires explicit synchronization between encoded input frames and token emissions. Since tokens are produced only after sufficient acoustic evidence is observed, an inherent latency arises, necessitating fine-tuning of the encoder-decoder alignment mechanism. We propose an updated inference mechanism that utilizes the fine-tuned causal encoder and decoder to yield greedy and beam-search decoding, and is shown to be locally optimal. Experiments on low-latency chunk sizes (less than 300 msec) show that our fine-tuned model outperforms existing non-fine-tuned streaming approaches in most cases, while using a lower complexity. We release our training and inference code, along with the fine-tuned models, to support further research and development in streaming ASR.