Contrastive Learning-based Chaining-Cluster for Multilingual Voice-Face Association

TL;DR

This paper presents a contrastive learning-based chaining-cluster method to improve face-voice association across multiple languages, addressing variability and outliers, and achieving high performance in the FAME 2024 challenge.

Contribution

It introduces a novel supervised cross-contrastive learning approach combined with a chaining-cluster post-processing step for multilingual face-voice association.

Findings

Achieved 2nd place in FAME 2024 challenge

Demonstrated robustness across multilingual data

Validated effectiveness through extensive experiments

Abstract

The innate correlation between a person's face and voice has recently emerged as a compelling area of study, especially within the context of multilingual environments. This paper introduces our novel solution to the Face-Voice Association in Multilingual Environments (FAME) 2024 challenge, focusing on a contrastive learning-based chaining-cluster method to enhance face-voice association. This task involves the challenges of building biometric relations between auditory and visual modality cues and modelling the prosody interdependence between different languages while addressing both intrinsic and extrinsic variability present in the data. To handle these non-trivial challenges, our method employs supervised cross-contrastive (SCC) learning to establish robust associations between voices and faces in multi-language scenarios. Following this, we have specifically designed a chaining-cluster-based post-processing step to mitigate the impact of outliers often found in unconstrained in the wild data. We conducted extensive experiments to investigate the impact of language on face-voice association. The overall results were evaluated on the FAME public evaluation platform, where we achieved 2nd place. The results demonstrate the superior performance of our method, and we validate the robustness and effectiveness of our proposed approach. Code is available at https://github.com/colaudiolab/FAME24_solution.