Fusion and Orthogonal Projection for Improved Face-Voice Association

TL;DR

This paper introduces a novel fusion and orthogonal projection (FOP) method that enhances face-voice association by creating enriched embeddings through modality fusion and orthogonality constraints, outperforming current state-of-the-art techniques.

Contribution

The paper proposes a lightweight, plug-and-play FOP mechanism that improves face-voice embedding discrimination without relying on complex loss functions or negative mining.

Findings

FOP achieves superior performance on VoxCeleb dataset tasks.

The method is more efficient and effective than existing approaches.

Enriched fused embeddings improve cross-modal verification and matching.

Abstract

We study the problem of learning association between face and voice, which is gaining interest in the computer vision community lately. Prior works adopt pairwise or triplet loss formulations to learn an embedding space amenable for associated matching and verification tasks. Albeit showing some progress, such loss formulations are, however, restrictive due to dependency on distance-dependent margin parameter, poor run-time training complexity, and reliance on carefully crafted negative mining procedures. In this work, we hypothesize that enriched feature representation coupled with an effective yet efficient supervision is necessary in realizing a discriminative joint embedding space for improved face-voice association. To this end, we propose a light-weight, plug-and-play mechanism that exploits the complementary cues in both modalities to form enriched fused embeddings and clusters them based on their identity labels via orthogonality constraints. We coin our proposed mechanism as fusion and orthogonal projection (FOP) and instantiate in a two-stream pipeline. The overall resulting framework is evaluated on a large-scale VoxCeleb dataset with a multitude of tasks, including cross-modal verification and matching. Results show that our method performs favourably against the current state-of-the-art methods and our proposed supervision formulation is more effective and efficient than the ones employed by the contemporary methods.