Beyond Performance Plateaus: A Comprehensive Study on Scalability in Speech Enhancement

TL;DR

This paper investigates the scalability of deep learning-based speech enhancement models across architectures, sizes, compute budgets, and datasets, revealing insights beyond current performance plateaus and highlighting future research directions.

Contribution

It provides a comprehensive analysis of how different factors affect the scalability of speech enhancement models, which was previously under-explored.

Findings

Scaling effects differ from speech recognition tasks.

Larger datasets and scalable architectures improve performance.

Insights into multi-domain corpora and efficient architectures.

Abstract

Deep learning-based speech enhancement (SE) models have achieved impressive performance in the past decade. Numerous advanced architectures have been designed to deliver state-of-the-art performance; however, their scalability potential remains unrevealed. Meanwhile, the majority of research focuses on small-sized datasets with restricted diversity, leading to a plateau in performance improvement. In this paper, we aim to provide new insights for addressing the above issues by exploring the scalability of SE models in terms of architectures, model sizes, compute budgets, and dataset sizes. Our investigation involves several popular SE architectures and speech data from different domains. Experiments reveal both similarities and distinctions between the scaling effects in SE and other tasks such as speech recognition. These findings further provide insights into the under-explored SE directions, e.g., larger-scale multi-domain corpora and efficiently scalable architectures.