MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets

TL;DR

This paper introduces MT4SSL, a multi-task self-supervised speech learning framework that integrates offline and online targets, leading to improved performance and convergence on speech benchmarks.

Contribution

It proposes a novel multi-task learning framework for self-supervised speech models using combined offline and online target extractors, enhancing performance and training efficiency.

Findings

Outperforms previous SSL methods on LibriSpeech benchmark.

Using both target extractors improves convergence during pre-training.

Achieves comparable or better results with less data.

Abstract

In this paper, we provide a new perspective on self-supervised speech models from how the training targets are obtained. We generalize the targets extractor into Offline Targets Extractor (Off-TE) and Online Targets Extractor (On-TE). Based on this, we propose a new multi-tasking learning framework for self-supervised learning, MT4SSL, which stands for Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets. MT4SSL uses the K-means algorithm as an Off-TE and a teacher network without gradients as an On-TE, respectively. Our model outperforms previous SSL methods by nontrivial margins on the LibriSpeech benchmark, and is comparable to or even better than the best-performing models with fewer data. Furthermore, we find that using both Off-TE and On-TE results in better convergence in the pre-training phase. With both effectiveness and efficiency, we think doing multi-task learning on self-supervised speech models from our perspective is a promising trend.