Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates

TL;DR

This paper introduces a one-pass speech model compression method using sparsity-aware self-pinching gates, achieving significant parameter reduction with minimal impact on accuracy and faster operation.

Contribution

The paper proposes a novel integrated pruning and training approach with self-pinching gates for efficient speech model compression.

Findings

Reduces wav2vec2.0-base parameters by 65%

Achieves lowest WER of 7.05% on test-clean

Operates with 25% less compression time

Abstract

This paper presents a novel approach for speech foundation models compression that tightly integrates model pruning and parameter update into a single stage. Highly compact layer-level tied self-pinching gates each containing only a single learnable threshold are jointly trained with uncompressed models and used in fine-grained neuron level pruning. Experiments conducted on the LibriSpeech-100hr corpus suggest that our approach reduces the number of parameters of wav2vec2.0-base and HuBERT-large models by 65% and 60% respectively, while incurring no statistically significant word error rate (WER) increase on the test-clean dataset. Compared to previously published methods on the same task, our approach not only achieves the lowest WER of 7.05% on the test-clean dataset under a comparable model compression ratio of 4.26x, but also operates with at least 25% less model compression time.