Learning Branched Fusion and Orthogonal Projection for Face-Voice Association

TL;DR

This paper introduces a novel fusion and orthogonal projection (FOP) framework that enhances face-voice association by creating enriched, discriminative embeddings using a lightweight, two-stream network with orthogonality constraints, outperforming existing methods.

Contribution

The paper proposes a new FOP mechanism that effectively fuses face and voice features and enforces orthogonality, improving discriminative power and efficiency over prior metric learning approaches.

Findings

FOP outperforms state-of-the-art on VoxCeleb1 and MAV-Celeb datasets.

The method is more efficient and effective than existing supervision techniques.

Cross-lingual analysis shows robustness of face-voice association across languages.

Abstract

Recent years have seen an increased interest in establishing association between faces and voices of celebrities leveraging audio-visual information from YouTube. Prior works adopt metric learning methods to learn an embedding space that is amenable for associated matching and verification tasks. Albeit showing some progress, such formulations are, however, restrictive due to dependency on distance-dependent margin parameter, poor run-time training complexity, and reliance on carefully crafted negative mining procedures. In this work, we hypothesize that an enriched representation coupled with an effective yet efficient supervision is important towards realizing a discriminative joint embedding space for face-voice association tasks. To this end, we propose a light-weight, plug-and-play mechanism that exploits the complementary cues in both modalities to form enriched fused embeddings and clusters them based on their identity labels via orthogonality constraints. We coin our proposed mechanism as fusion and orthogonal projection (FOP) and instantiate in a two-stream network. The overall resulting framework is evaluated on VoxCeleb1 and MAV-Celeb datasets with a multitude of tasks, including cross-modal verification and matching. Results reveal that our method performs favourably against the current state-of-the-art methods and our proposed formulation of supervision is more effective and efficient than the ones employed by the contemporary methods. In addition, we leverage cross-modal verification and matching tasks to analyze the impact of multiple languages on face-voice association. Code is available: \url{https://github.com/msaadsaeed/FOP}