Training Speech Enhancement Systems with Noisy Speech Datasets

TL;DR

This paper introduces methods to train speech enhancement systems using noisy datasets by modifying loss functions and augmenting noise, enabling effective training without clean speech data and improving speech quality metrics.

Contribution

The paper presents novel loss function modifications and a noise augmentation scheme for training speech enhancement systems on noisy data, expanding training options.

Findings

Improved PESQ scores by up to 0.19 with robust loss functions.

Enhanced PESQ scores by up to 0.27 using noise augmentation in MixIT.

Demonstrated effectiveness on Mozilla Common Voice dataset.

Abstract

Recently, deep neural network (DNN)-based speech enhancement (SE) systems have been used with great success. During training, such systems require clean speech data - ideally, in large quantity with a variety of acoustic conditions, many different speaker characteristics and for a given sampling rate (e.g., 48kHz for fullband SE). However, obtaining such clean speech data is not straightforward - especially, if only considering publicly available datasets. At the same time, a lot of material for automatic speech recognition (ASR) with the desired acoustic/speaker/sampling rate characteristics is publicly available except being clean, i.e., it also contains background noise as this is even often desired in order to have ASR systems that are noise-robust. Hence, using such data to train SE systems is not straightforward. In this paper, we propose two improvements to train SE systems on noisy speech data. First, we propose several modifications of the loss functions, which make them robust against noisy speech targets. In particular, computing the median over the sample axis before averaging over time-frequency bins allows to use such data. Furthermore, we propose a noise augmentation scheme for mixture-invariant training (MixIT), which allows using it also in such scenarios. For our experiments, we use the Mozilla Common Voice dataset and we show that using our robust loss function improves PESQ by up to 0.19 compared to a system trained in the traditional way. Similarly, for MixIT we can see an improvement of up to 0.27 in PESQ when using our proposed noise augmentation.