Whisper in Medusa's Ear: Multi-head Efficient Decoding for Transformer-based ASR

TL;DR

Whisper-Medusa is a novel multi-head decoding approach that accelerates transformer-based speech recognition models by 50% with minimal WER increase, improving inference speed for speech transcription tasks.

Contribution

It introduces Whisper-Medusa, a multi-head decoding method that extends Whisper architecture to significantly reduce inference latency while maintaining accuracy.

Findings

50% reduction in decoding latency

Minimal impact on Word Error Rate

Effective across various datasets and learning setups

Abstract

Large transformer-based models have significant potential for speech transcription and translation. Their self-attention mechanisms and parallel processing enable them to capture complex patterns and dependencies in audio sequences. However, this potential comes with challenges, as these large and computationally intensive models lead to slow inference speeds. Various optimization strategies have been proposed to improve performance, including efficient hardware utilization and algorithmic enhancements. In this paper, we introduce Whisper-Medusa, a novel approach designed to enhance processing speed with minimal impact on Word Error Rate (WER). The proposed model extends the OpenAI's Whisper architecture by predicting multiple tokens per iteration, resulting in a 50% reduction in latency. We showcase the effectiveness of Whisper-Medusa across different learning setups and datasets.