Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models

TL;DR

This paper empirically analyzes how varying the depth and width of self-supervised speech models affects their internal representations and performance, revealing that different architectures excel at different speech tasks.

Contribution

It provides a systematic study on the impact of model architecture variations on SSL speech models, highlighting the importance of depth and width for different tasks.

Findings

Depth is crucial for content-oriented tasks like speech recognition.

Width is important for speaker-oriented tasks such as speaker identification.

A more compressed model with better performance than previous work is identified.

Abstract

Self-supervised learning (SSL) is seen as a very promising approach with high performance for several speech downstream tasks. Since the parameters of SSL models are generally so large that training and inference require a lot of memory and computational cost, it is desirable to produce compact SSL models without a significant performance degradation by applying compression methods such as knowledge distillation (KD). Although the KD approach is able to shrink the depth and/or width of SSL model structures, there has been little research on how varying the depth and width impacts the internal representation of the small-footprint model. This paper provides an empirical study that addresses the question. We investigate the performance on SUPERB while varying the structure and KD methods so as to keep the number of parameters constant; this allows us to analyze the contribution of the representation introduced by varying the model architecture. Experiments demonstrate that a certain depth is essential for solving content-oriented tasks (e.g. automatic speech recognition) accurately, whereas a certain width is necessary for achieving high performance on several speaker-oriented tasks (e.g. speaker identification). Based on these observations, we identify, for SUPERB, a more compressed model with better performance than previous studies.