Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association

TL;DR

This paper introduces an adaptive framework for learning voice-face associations that incorporates global and local modality alignment, along with a dynamic reweighting scheme to handle diverse learning difficulties, improving performance across multiple tasks.

Contribution

It proposes a novel two-level modality alignment loss with global and local information, and a dynamic reweighting scheme to address learning difficulty diversity.

Findings

Outperforms previous methods in voice-face matching, verification, and retrieval.

Incorporates global identity classification to enhance embedding separation.

Effectively filters unlearnable identities, improving learning efficiency.

Abstract

Nowadays, we have witnessed the early progress on learning the association between voice and face automatically, which brings a new wave of studies to the computer vision community. However, most of the prior arts along this line (a) merely adopt local information to perform modality alignment and (b) ignore the diversity of learning difficulty across different subjects. In this paper, we propose a novel framework to jointly address the above-mentioned issues. Targeting at (a), we propose a two-level modality alignment loss where both global and local information are considered. Compared with the existing methods, we introduce a global loss into the modality alignment process. The global component of the loss is driven by the identity classification. Theoretically, we show that minimizing the loss could maximize the distance between embeddings across different identities while minimizing the distance between embeddings belonging to the same identity, in a global sense (instead of a mini-batch). Targeting at (b), we propose a dynamic reweighting scheme to better explore the hard but valuable identities while filtering out the unlearnable identities. Experiments show that the proposed method outperforms the previous methods in multiple settings, including voice-face matching, verification and retrieval.