Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast

TL;DR

This paper introduces CMPC, a novel unsupervised learning method that improves voice-face representation by addressing false negatives and weak correlations through semantic clustering and prototype comparison.

Contribution

The paper proposes cross-modal prototype contrastive learning (CMPC), enhancing unsupervised voice-face representation by leveraging semantic clustering and dynamic prototype comparison.

Findings

Outperforms state-of-the-art unsupervised methods in voice-face association tasks.

Shows significant improvements in low-shot supervision scenarios.

Effectively resists false negatives and deviant positives in contrastive learning.

Abstract

We present an approach to learn voice-face representations from the talking face videos, without any identity labels. Previous works employ cross-modal instance discrimination tasks to establish the correlation of voice and face. These methods neglect the semantic content of different videos, introducing false-negative pairs as training noise. Furthermore, the positive pairs are constructed based on the natural correlation between audio clips and visual frames. However, this correlation might be weak or inaccurate in a large amount of real-world data, which leads to deviating positives into the contrastive paradigm. To address these issues, we propose the cross-modal prototype contrastive learning (CMPC), which takes advantage of contrastive methods and resists adverse effects of false negatives and deviate positives. On one hand, CMPC could learn the intra-class invariance by constructing semantic-wise positives via unsupervised clustering in different modalities. On the other hand, by comparing the similarities of cross-modal instances from that of cross-modal prototypes, we dynamically recalibrate the unlearnable instances' contribution to overall loss. Experiments show that the proposed approach outperforms state-of-the-art unsupervised methods on various voice-face association evaluation protocols. Additionally, in the low-shot supervision setting, our method also has a significant improvement compared to previous instance-wise contrastive learning.