An Empirical Analysis of Speech Self-Supervised Learning at Multiple Resolutions

TL;DR

This paper investigates multi-resolution speech SSL models, revealing that performance gains are mainly due to auxiliary losses rather than multi-scale representations, and that downsampling improves efficiency but not representation quality.

Contribution

The study provides an empirical analysis of multi-scale speech SSL architectures, challenging assumptions about their effectiveness in capturing hierarchical speech representations.

Findings

Performance improvements stem from auxiliary low-resolution loss.

Downsampling does not enhance downstream task performance.

Downsampling increases computational efficiency without improving representations.

Abstract

Self-supervised learning (SSL) models have become crucial in speech processing, with recent advancements concentrating on developing architectures that capture representations across multiple timescales. The primary goal of these multi-scale architectures is to exploit the hierarchical nature of speech, where lower-resolution components aim to capture representations that align with increasingly abstract concepts (e.g., from phones to words to sentences). Although multi-scale approaches have demonstrated some improvements over single-scale models, the precise reasons for these enhancements have poor empirical support. In this study, we present an initial analysis of layer-wise representations in multi-scale architectures, with a focus on Canonical Correlation Analysis (CCA) and Mutual Information (MI). We apply this analysis to Multi-Resolution HuBERT (MR-HuBERT) and find that (1) the improved performance on SUPERB tasks is primarily due to the auxiliary low-resolution loss rather than the downsampling itself, and (2) downsampling to lower resolutions neither improves downstream performance nor correlates with higher-level information (e.g., words), though it does improve computational efficiency. These findings challenge assumptions about the multi-scale nature of MR-HuBERT and motivate the importance of disentangling computational efficiency from learning better representations.