Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder

TL;DR

This paper introduces a noise-aware variational autoencoder for speech enhancement that improves robustness to noise and generalizes better to unseen noise conditions compared to standard VAEs and DNNs.

Contribution

The paper proposes a noise-aware encoder for VAE training, enhancing robustness and generalization in speech enhancement tasks.

Findings

Outperforms standard VAE in reducing distortion

Generalizes better to unseen noise conditions

Maintains performance with less training data

Abstract

Recently, a generative variational autoencoder (VAE) has been proposed for speech enhancement to model speech statistics. However, this approach only uses clean speech in the training phase, making the estimation particularly sensitive to noise presence, especially in low signal-to-noise ratios (SNRs). To increase the robustness of the VAE, we propose to include noise information in the training phase by using a noise-aware encoder trained on noisy-clean speech pairs. We evaluate our approach on real recordings of different noisy environments and acoustic conditions using two different noise datasets. We show that our proposed noise-aware VAE outperforms the standard VAE in terms of overall distortion without increasing the number of model parameters. At the same time, we demonstrate that our model is capable of generalizing to unseen noise conditions better than a supervised feedforward deep neural network (DNN). Furthermore, we demonstrate the robustness of the model performance to a reduction of the noisy-clean speech training data size.