Language and Noise Transfer in Speech Enhancement Generative Adversarial Network

TL;DR

This paper explores how fine-tuning a speech enhancement GAN with minimal data from different languages and noise types can achieve stable, high-quality performance, reducing data requirements significantly.

Contribution

It demonstrates effective adaptation of a speech enhancement GAN to new languages and noise conditions using very limited data, highlighting minimal data needs for stability.

Findings

10 minutes of fine-tuning data suffices for comparable performance to much larger datasets

Stable performance across different noise types with limited training data

Effective cross-lingual adaptation of speech enhancement models

Abstract

Speech enhancement deep learning systems usually require large amounts of training data to operate in broad conditions or real applications. This makes the adaptability of those systems into new, low resource environments an important topic. In this work, we present the results of adapting a speech enhancement generative adversarial network by finetuning the generator with small amounts of data. We investigate the minimum requirements to obtain a stable behavior in terms of several objective metrics in two very different languages: Catalan and Korean. We also study the variability of test performance to unseen noise as a function of the amount of different types of noise available for training. Results show that adapting a pre-trained English model with 10 min of data already achieves a comparable performance to having two orders of magnitude more data. They also demonstrate the relative stability in test performance with respect to the number of training noise types.