WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

TL;DR

WavLM is a large-scale self-supervised model trained on 94k hours of speech data, designed to improve a wide range of speech processing tasks by jointly learning speech prediction and denoising.

Contribution

The paper introduces WavLM, a novel pre-trained model that enhances full-stack speech processing by combining masked prediction, denoising, and a new positional bias mechanism.

Findings

WavLM Large achieves state-of-the-art results on the SUPERB benchmark.

WavLM significantly improves performance across various speech tasks.

Scaling up training data enhances model effectiveness.

Abstract

Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks. The code and pre-trained models are available at https://aka.ms/wavlm.