Multi-Modal Multi-Correlation Learning for Audio-Visual Speech Separation

TL;DR

This paper introduces a multi-modal multi-correlation learning framework for audio-visual speech separation, leveraging identity and phonetic correlations with contrastive and adversarial training to improve separation accuracy.

Contribution

It defines and exploits identity and phonetic correlations between audio and visual data, using adversarial training to enhance speech separation performance.

Findings

Improved separation accuracy over previous methods

Adversarial training outperforms contrastive learning in this framework

Effective in challenging cases like same gender or similar content

Abstract

In this paper we propose a multi-modal multi-correlation learning framework targeting at the task of audio-visual speech separation. Although previous efforts have been extensively put on combining audio and visual modalities, most of them solely adopt a straightforward concatenation of audio and visual features. To exploit the real useful information behind these two modalities, we define two key correlations which are: (1) identity correlation (between timbre and facial attributes); (2) phonetic correlation (between phoneme and lip motion). These two correlations together comprise the complete information, which shows a certain superiority in separating target speaker's voice especially in some hard cases, such as the same gender or similar content. For implementation, contrastive learning or adversarial training approach is applied to maximize these two correlations. Both of them work well, while adversarial training shows its advantage by avoiding some limitations of contrastive learning. Compared with previous research, our solution demonstrates clear improvement on experimental metrics without additional complexity. Further analysis reveals the validity of the proposed architecture and its good potential for future extension.