Single-Microphone Speech Enhancement and Separation Using Deep Learning

TL;DR

This paper explores deep learning techniques for single-microphone speech enhancement and separation, demonstrating improved generalizability, state-of-the-art separation results, and effective joint enhancement without prior noise or speaker information.

Contribution

It introduces uPIT, a novel deep learning algorithm for speech separation, and provides insights into training data design for better generalizability of enhancement algorithms.

Findings

uPIT achieves state-of-the-art multi-talker separation results.

Deep learning enhancement algorithms can be optimized for speech intelligibility.

Carefully designed training data improves generalizability of speech enhancement models.

Abstract

The cocktail party problem comprises the challenging task of understanding a speech signal in a complex acoustic environment, where multiple speakers and background noise signals simultaneously interfere with the speech signal of interest. A signal processing algorithm that can effectively increase the speech intelligibility and quality of speech signals in such complicated acoustic situations is highly desirable. Especially for applications involving mobile communication devices and hearing assistive devices. Due to the re-emergence of machine learning techniques, today, known as deep learning, the challenges involved with such algorithms might be overcome. In this PhD thesis, we study and develop deep learning-based techniques for two sub-disciplines of the cocktail party problem: single-microphone speech enhancement and single-microphone multi-talker speech separation. Specifically, we conduct in-depth empirical analysis of the generalizability capability of modern deep learning-based single-microphone speech enhancement algorithms. We show that performance of such algorithms is closely linked to the training data, and good generalizability can be achieved with carefully designed training data. Furthermore, we propose uPIT, a deep learning-based algorithm for single-microphone speech separation and we report state-of-the-art results on a speaker-independent multi-talker speech separation task. Additionally, we show that uPIT works well for joint speech separation and enhancement without explicit prior knowledge about the noise type or number of speakers. Finally, we show that deep learning-based speech enhancement algorithms designed to minimize the classical short-time spectral amplitude mean squared error leads to enhanced speech signals which are essentially optimal in terms of STOI, a state-of-the-art speech intelligibility estimator.