Similarity Analysis of Self-Supervised Speech Representations

TL;DR

This paper provides a comparative analysis of self-supervised speech representations, examining their similarities, properties, and the impact of training objectives versus architecture on their effectiveness.

Contribution

It introduces a systematic comparison of prominent self-supervised speech models, highlighting the influence of training objectives over architectural differences.

Findings

Training objectives significantly affect representation similarity.

Pre-training loss correlates with downstream performance.

Architectural choices have less impact than training objectives.

Abstract

Self-supervised speech representation learning has recently been a prosperous research topic. Many algorithms have been proposed for learning useful representations from large-scale unlabeled data, and their applications to a wide range of speech tasks have also been investigated. However, there has been little research focusing on understanding the properties of existing approaches. In this work, we aim to provide a comparative study of some of the most representative self-supervised algorithms. Specifically, we quantify the similarities between different self-supervised representations using existing similarity measures. We also design probing tasks to study the correlation between the models' pre-training loss and the amount of specific speech information contained in their learned representations. In addition to showing how various self-supervised models behave differently given the same input, our study also finds that the training objective has a higher impact on representation similarity than architectural choices such as building blocks (RNN/Transformer/CNN) and directionality (uni/bidirectional). Our results also suggest that there exists a strong correlation between pre-training loss and downstream performance for some self-supervised algorithms.