Towards Intelligibility-Oriented Audio-Visual Speech Enhancement

TL;DR

This paper introduces a novel audio-visual speech enhancement model that uses an intelligibility-oriented loss function based on a modified STOI metric, improving robustness and generalization in noisy environments.

Contribution

It is the first to combine audio-visual information with an intelligibility-oriented loss function for speech enhancement, demonstrating superior performance over traditional methods.

Findings

Outperforms audio-only and conventional AV models on unseen speakers and noises.

Uses a fully convolutional AV model with a modified STOI loss function.

Shows improved speech intelligibility in noisy conditions.

Abstract

Existing deep learning (DL) based speech enhancement approaches are generally optimised to minimise the distance between clean and enhanced speech features. These often result in improved speech quality however they suffer from a lack of generalisation and may not deliver the required speech intelligibility in real noisy situations. In an attempt to address these challenges, researchers have explored intelligibility-oriented (I-O) loss functions and integration of audio-visual (AV) information for more robust speech enhancement (SE). In this paper, we introduce DL based I-O SE algorithms exploiting AV information, which is a novel and previously unexplored research direction. Specifically, we present a fully convolutional AV SE model that uses a modified short-time objective intelligibility (STOI) metric as a training cost function. To the best of our knowledge, this is the first work that exploits the integration of AV modalities with an I-O based loss function for SE. Comparative experimental results demonstrate that our proposed I-O AV SE framework outperforms audio-only (AO) and AV models trained with conventional distance-based loss functions, in terms of standard objective evaluation measures when dealing with unseen speakers and noises.