Effective and Efficient Mixed Precision Quantization of Speech Foundation Models

TL;DR

This paper introduces a unified mixed-precision quantization method for speech foundation models that improves compression efficiency and reduces system time without increasing word error rate, demonstrated on LibriSpeech datasets.

Contribution

It proposes a novel integrated approach for mixed-precision learning and quantization, outperforming traditional two-stage methods in compression ratio and efficiency.

Findings

Increased lossless compression ratios up to 1.9x.

Reduced system compression time by up to 1.9 times.

Achieved 8.6x compression with 3.5-bit quantization.

Abstract

This paper presents a novel mixed-precision quantization approach for speech foundation models that tightly integrates mixed-precision learning and quantized model parameter estimation into one single model compression stage. Experiments conducted on LibriSpeech dataset with fine-tuned wav2vec2.0-base and HuBERT-large models suggest the resulting mixed-precision quantized models increased the lossless compression ratio by factors up to 1.7x and 1.9x over the respective uniform-precision and two-stage mixed-precision quantized baselines that perform precision learning and model parameters quantization in separate and disjointed stages, while incurring no statistically word error rate (WER) increase over the 32-bit full-precision models. The system compression time of wav2vec2.0-base and HuBERT-large models is reduced by up to 1.9 and 1.5 times over the two-stage mixed-precision baselines, while both produce lower WERs. The best-performing 3.5-bit mixed-precision quantized HuBERT-large model produces a lossless compression ratio of 8.6x over the 32-bit full-precision system.