Cross-modal Speaker Verification and Recognition: A Multilingual Perspective

TL;DR

This paper investigates whether face-voice association and speaker recognition are language-independent by introducing a multilingual dataset and conducting experiments to evaluate cross-lingual biometric matching.

Contribution

It introduces a new multilingual audio-visual dataset and explores the challenges of cross-lingual face-voice association and speaker recognition, addressing key questions in multilingual biometric systems.

Findings

Face-voice association shows partial language independence.

Speaker recognition performance varies across languages.

Multilingual challenges significantly impact biometric system effectiveness.

Abstract

Recent years have seen a surge in finding association between faces and voices within a cross-modal biometric application along with speaker recognition. Inspired from this, we introduce a challenging task in establishing association between faces and voices across multiple languages spoken by the same set of persons. The aim of this paper is to answer two closely related questions: "Is face-voice association language independent?" and "Can a speaker be recognised irrespective of the spoken language?". These two questions are very important to understand effectiveness and to boost development of multilingual biometric systems. To answer them, we collected a Multilingual Audio-Visual dataset, containing human speech clips of $154$ identities with $3$ language annotations extracted from various videos uploaded online. Extensive experiments on the three splits of the proposed dataset have been performed to investigate and answer these novel research questions that clearly point out the relevance of the multilingual problem.