Robust Audio-Visual Speech Enhancement: Correcting Misassignments in Complex Environments with Advanced Post-Processing

TL;DR

This paper proposes a post-processing classifier to correct misassignments in audio-visual speech enhancement systems, significantly improving performance especially in complex environments with poor video quality.

Contribution

Introduction of a robust post-processing classifier with a mixup training strategy to correct speech misassignments in AVSE systems, enhancing accuracy and robustness.

Findings

Significant performance improvement over baseline models.

Effective correction of misassignments in complex environments.

Enhanced robustness with mixup training strategy.

Abstract

This paper addresses the prevalent issue of incorrect speech output in audio-visual speech enhancement (AVSE) systems, which is often caused by poor video quality and mismatched training and test data. We introduce a post-processing classifier (PPC) to rectify these erroneous outputs, ensuring that the enhanced speech corresponds accurately to the intended speaker. We also adopt a mixup strategy in PPC training to improve its robustness. Experimental results on the AVSE-challenge dataset show that integrating PPC into the AVSE model can significantly improve AVSE performance, and combining PPC with the AVSE model trained with permutation invariant training (PIT) yields the best performance. The proposed method substantially outperforms the baseline model by a large margin. This work highlights the potential for broader applications across various modalities and architectures, providing a promising direction for future research in this field.