The Effect of Training Dataset Size on Discriminative and Diffusion-Based Speech Enhancement Systems

TL;DR

This study compares how training dataset size affects the performance of discriminative and diffusion-based speech enhancement models, revealing that diffusion models excel with small datasets but are outperformed by discriminative models on larger datasets.

Contribution

It systematically evaluates the impact of dataset size on both diffusion-based and discriminative speech enhancement systems under controlled data diversity conditions.

Findings

Diffusion models perform best with datasets of 10 hours or less.

Discriminative models improve more with larger datasets, surpassing diffusion models at 100 hours.

Diffusion models' performance plateaus regardless of increased dataset size.

Abstract

The performance of deep neural network-based speech enhancement systems typically increases with the training dataset size. However, studies that investigated the effect of training dataset size on speech enhancement performance did not consider recent approaches, such as diffusion-based generative models. Diffusion models are typically trained with massive datasets for image generation tasks, but whether this is also required for speech enhancement is unknown. Moreover, studies that investigated the effect of training dataset size did not control for the data diversity. It is thus unclear whether the performance improvement was due to the increased dataset size or diversity. Therefore, we systematically investigate the effect of training dataset size on the performance of popular state-of-the-art discriminative and diffusion-based speech enhancement systems in matched conditions. We control for the data diversity by using a fixed set of speech utterances, noise segments and binaural room impulse responses to generate datasets of different sizes. We find that the diffusion-based systems perform the best relative to the discriminative systems in terms of objective metrics with datasets of 10 h or less. However, their objective metrics performance does not improve when increasing the training dataset size as much as the discriminative systems, and they are outperformed by the discriminative systems with datasets of 100 h or more.