Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation

TL;DR

This paper introduces CaffNet, a cross-modal affinity network that learns global and local audio-visual correspondences to improve speech separation from videos, overcoming synchronization issues and enhancing performance in real-world scenarios.

Contribution

The paper proposes a novel cross-modal affinity learning approach that captures global and local audio-visual relationships, addressing synchronization and permutation problems in speech separation.

Findings

Outperforms conventional methods on various datasets

Effectively handles synchronization mismatches

Improves separation in complex spectral domain

Abstract

In this paper, we address the problem of separating individual speech signals from videos using audio-visual neural processing. Most conventional approaches utilize frame-wise matching criteria to extract shared information between co-occurring audio and video. Thus, their performance heavily depends on the accuracy of audio-visual synchronization and the effectiveness of their representations. To overcome the frame discontinuity problem between two modalities due to transmission delay mismatch or jitter, we propose a cross-modal affinity network (CaffNet) that learns global correspondence as well as locally-varying affinities between audio and visual streams. Given that the global term provides stability over a temporal sequence at the utterance-level, this resolves the label permutation problem characterized by inconsistent assignments. By extending the proposed cross-modal affinity on the complex network, we further improve the separation performance in the complex spectral domain. Experimental results verify that the proposed methods outperform conventional ones on various datasets, demonstrating their advantages in real-world scenarios.