Scale This, Not That: Investigating Key Dataset Attributes for Efficient Speech Enhancement Scaling

TL;DR

This paper investigates how different dataset attributes like speaker, noise, language, and text influence speech enhancement models, revealing that acoustic attributes are more critical than semantic ones for model performance.

Contribution

It introduces a novel framework using zero-shot TTS to systematically analyze the impact of individual dataset attributes on speech enhancement performance.

Findings

Acoustic attributes significantly affect model performance.

Semantic attributes have a lesser impact.

The framework enables scalable, controlled dataset synthesis.

Abstract

Recent speech enhancement models have shown impressive performance gains by scaling up model complexity and training data. However, the impact of dataset variability (e.g. text, language, speaker, and noise) has been underexplored. Analyzing each attribute individually is often challenging, as multiple attributes are usually entangled in commonly used datasets, posing a significant obstacle in understanding the distinct contributions of each attribute to the model's performance. To address this challenge, we propose a generation-training-evaluation framework that leverages zero-shot text-to-speech systems to investigate the impact of controlled attribute variations on speech enhancement performance. It enables us to synthesize training datasets in a scalable manner while carefully altering each attribute. Based on the proposed framework, we analyze the scaling effects of various dataset attributes on the performance of both discriminative and generative SE models. Extensive experiments on multi-domain corpora imply that acoustic attributes (e.g., speaker and noise) are much more important to current speech enhancement models than semantic attributes (e.g., language and text), offering new insights for future research.