Masked Autoencoders as Universal Speech Enhancer

TL;DR

This paper introduces a self-supervised masked autoencoder approach for universal speech enhancement that effectively handles multiple distortions and improves downstream speech tasks, outperforming existing methods.

Contribution

The work presents a novel self-supervised masked autoencoder model that is distortion-agnostic and capable of enhancing speech across various types of noise and reverberation.

Findings

Outperforms baseline methods in speech enhancement tasks.

Achieves state-of-the-art results on in-domain and out-of-domain datasets.

Effective for both denoising and dereverberation applications.

Abstract

Supervised speech enhancement methods have been very successful. However, in practical scenarios, there is a lack of clean speech, and self-supervised learning-based (SSL) speech enhancement methods that offer comparable enhancement performance and can be applied to other speech-related downstream applications are desired. In this work, we develop a masked autoencoder based universal speech enhancer that is agnostic to the type of distortion affecting speech, can handle multiple distortions simultaneously, and is trained in a self-supervised manner. An augmentation stack adds further distortions to the noisy input data. The masked autoencoder model learns to remove the added distortions along with reconstructing the masked regions of the spectrogram during pre-training. The pre-trained embeddings are then used by fine-tuning models trained on a small amount of paired data for specific downstream tasks. We evaluate the pre-trained features for denoising and dereverberation downstream tasks. We explore different augmentations (like single or multi-speaker) in the pre-training augmentation stack and the effect of different noisy input feature representations (like $log1p$ compression) on pre-trained embeddings and downstream fine-tuning enhancement performance. We show that the proposed method not only outperforms the baseline but also achieves state-of-the-art performance for both in-domain and out-of-domain evaluation datasets.