CochleaNet: A Robust Language-independent Audio-Visual Model for Speech Enhancement

TL;DR

CochleaNet is a novel, robust, language-independent audio-visual deep neural network designed for speech enhancement in noisy environments, leveraging visual cues and trained on limited data to outperform existing methods.

Contribution

The paper introduces CochleaNet, a causal AV speech enhancement model that generalizes across languages, noises, and speakers, challenging the belief that large multi-language AV datasets are necessary.

Findings

CochleaNet outperforms state-of-the-art SE approaches in objective and subjective tests.

The model trained on limited synthetic data generalizes well to real noisy environments.

The approach demonstrates robustness across languages, speaker variations, and noise types.

Abstract

Noisy situations cause huge problems for suffers of hearing loss as hearing aids often make the signal more audible but do not always restore the intelligibility. In noisy settings, humans routinely exploit the audio-visual (AV) nature of the speech to selectively suppress the background noise and to focus on the target speaker. In this paper, we present a causal, language, noise and speaker independent AV deep neural network (DNN) architecture for speech enhancement (SE). The model exploits the noisy acoustic cues and noise robust visual cues to focus on the desired speaker and improve the speech intelligibility. To evaluate the proposed SE framework a first of its kind AV binaural speech corpus, called ASPIRE, is recorded in real noisy environments including cafeteria and restaurant. We demonstrate superior performance of our approach in terms of objective measures and subjective listening tests over the state-of-the-art SE approaches as well as recent DNN based SE models. In addition, our work challenges a popular belief that a scarcity of multi-language large vocabulary AV corpus and wide variety of noises is a major bottleneck to build a robust language, speaker and noise independent SE systems. We show that a model trained on synthetic mixture of Grid corpus (with 33 speakers and a small English vocabulary) and ChiME 3 Noises (consisting of only bus, pedestrian, cafeteria, and street noises) generalise well not only on large vocabulary corpora but also on completely unrelated languages (such as Mandarin), wide variety of speakers and noises.