Adapting Whisper for Streaming Speech Recognition via Two-Pass Decoding

TL;DR

This paper adapts the Whisper speech recognition model for streaming applications by implementing a two-pass decoding approach with a CTC decoder and a reranking decoder, enhancing its real-time capabilities.

Contribution

It introduces a novel fine-tuning method for Whisper to support streaming ASR using a two-pass decoding structure with a hybrid tokenizer approach.

Findings

Effective streaming ASR achieved on LibriSpeech and earnings call datasets.

Hybrid tokenizer improves data efficiency and generalization.

Whisper can be adapted for streaming with adequate fine-tuning.

Abstract

OpenAI Whisper is a family of robust Automatic Speech Recognition (ASR) models trained on 680,000 hours of audio. However, its encoder-decoder architecture, trained with a sequence-to-sequence objective, lacks native support for streaming ASR. In this paper, we fine-tune Whisper for streaming ASR using the WeNet toolkit by adopting a Unified Two-pass (U2) structure. We introduce an additional Connectionist Temporal Classification (CTC) decoder trained with causal attention masks to generate streaming partial transcripts, while the original Whisper decoder reranks these partial outputs. Our experiments on LibriSpeech and an earnings call dataset demonstrate that, with adequate fine-tuning data, Whisper can be adapted into a capable streaming ASR model. We also introduce a hybrid tokenizer approach, which uses a smaller token space for the CTC decoder while retaining Whisper's original token space for the attention decoder, resulting in improved data efficiency and generalization.