LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation

TL;DR

LiteASR introduces a low-rank compression scheme for ASR encoders that reduces inference costs by over 50% while maintaining high transcription accuracy, enabling more efficient deployment of models like Whisper.

Contribution

The paper presents a novel low-rank approximation method for ASR encoders that significantly reduces computational costs without sacrificing accuracy.

Findings

Over 50% reduction in encoder size for Whisper large-v3

Achieves better transcription accuracy than Whisper medium

Establishes a new Pareto frontier of accuracy and efficiency

Abstract

Modern automatic speech recognition (ASR) models, such as OpenAI's Whisper, rely on deep encoder-decoder architectures, and their encoders are a critical bottleneck for efficient deployment due to high computational intensity. We introduce LiteASR, a low-rank compression scheme for ASR encoders that significantly reduces inference costs while maintaining transcription accuracy. Our approach leverages the strong low-rank properties observed in intermediate activations: by applying principal component analysis (PCA) with a small calibration dataset, we approximate linear transformations with a chain of low-rank matrix multiplications, and further optimize self-attention to work in reduced dimensionality. Evaluation results show that our method can compress Whisper large-v3's encoder size by over 50%, matching Whisper medium's size with better transcription accuracy, thereby establishing a new Pareto frontier of accuracy and efficiency. The code of LiteASR is available at https://github.com/efeslab/LiteASR.