Synergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Models

TL;DR

This paper investigates combining knowledge distillation with structured pruning techniques like LRF and l0 regularization to enhance self-supervised speech models, achieving significant reductions in word error rates.

Contribution

It introduces a joint pruning and training strategy for RNN-T ASR models and demonstrates improved performance over traditional sequential pruning methods.

Findings

l0 and KD combination reduces RWER by 8.9%

LRF and KD combination reduces RWER by 13.4%

Joint pruning and training outperforms sequential pruning

Abstract

Traditionally, Knowledge Distillation (KD) is used for model compression, often leading to suboptimal performance. In this paper, we evaluate the impact of combining KD loss with alternative pruning techniques, including Low-Rank Factorization (LRF) and l0 regularization, on a conformer-based pre-trained network under the paradigm of Self-Supervised Learning (SSL). We also propose a strategy to jointly prune and train an RNN-T-based ASR model, demonstrating that this approach yields superior performance compared to pruning a pre-trained network first and then using it for ASR training. This approach led to a significant reduction in word error rate: l0 and KD combination achieves the best non-streaming performance, with a 8.9% Relative Word Error Rate (RWER) improvement over the baseline, while LRF and KD combination yields the best results for streaming ASR, improving RWER by 13.4%.