Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders

TL;DR

This paper introduces a novel unsupervised speech enhancement method using dynamical variational autoencoders (DVAEs) that models temporal dependencies and outperforms existing approaches, especially on unseen noise types.

Contribution

It extends VAE-based speech enhancement to DVAEs, combining speech dynamics modeling with unsupervised learning for improved noise robustness.

Findings

DVAE-based method outperforms VAE-based and baseline methods

Effective on unseen noise types

Versatile framework with three DVAE models

Abstract

Dynamical variational autoencoders (DVAEs) are a class of deep generative models with latent variables, dedicated to model time series of high-dimensional data. DVAEs can be considered as extensions of the variational autoencoder (VAE) that include temporal dependencies between successive observed and/or latent vectors. Previous work has shown the interest of using DVAEs over the VAE for speech spectrograms modeling. Independently, the VAE has been successfully applied to speech enhancement in noise, in an unsupervised noise-agnostic set-up that requires neither noise samples nor noisy speech samples at training time, but only requires clean speech signals. In this paper, we extend these works to DVAE-based single-channel unsupervised speech enhancement, hence exploiting both speech signals unsupervised representation learning and dynamics modeling. We propose an unsupervised speech enhancement algorithm that combines a DVAE speech prior pre-trained on clean speech signals with a noise model based on nonnegative matrix factorization, and we derive a variational expectation-maximization (VEM) algorithm to perform speech enhancement. The algorithm is presented with the most general DVAE formulation and is then applied with three specific DVAE models to illustrate the versatility of the framework. Experimental results show that the proposed DVAE-based approach outperforms its VAE-based counterpart, as well as several supervised and unsupervised noise-dependent baselines, especially when the noise type is unseen during training.