Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment

TL;DR

This paper explores distillation and pruning techniques to create smaller, scalable speech quality assessment models based on self-supervised representations, achieving comparable accuracy with significantly reduced model size.

Contribution

It introduces a combined approach of distillation and pruning for self-supervised speech quality models, demonstrating effective size reduction while maintaining performance.

Findings

Distillation halves the gap in correlation with ground-truth MOS.

Model size reduced by two orders of magnitude with minimal performance loss.

Distillation is more effective for smaller models, while pruning benefits larger models.

Abstract

In this paper, we investigate distillation and pruning methods to reduce model size for non-intrusive speech quality assessment based on self-supervised representations. Our experiments build on XLS-R-SQA, a speech quality assessment model using wav2vec 2.0 XLS-R embeddings. We retrain this model on a large compilation of mean opinion score datasets, encompassing over 100,000 labeled clips. For distillation, using this model as a teacher, we generate pseudo-labels on unlabeled degraded speech signals and train student models of varying sizes. For pruning, we use a data-driven strategy. While data-driven pruning performs better at larger model sizes, distillation on unlabeled data is more effective for smaller model sizes. Distillation can halve the gap between the baseline's correlation with ground-truth MOS labels and that of the XLS-R-based teacher model, while reducing model size by two orders of magnitude compared to the teacher model.