Seeing Voices and Hearing Faces: Cross-modal biometric matching

TL;DR

This paper explores the feasibility of cross-modal biometric matching between voices and faces, introducing CNN architectures to identify speakers from audio or face images, and compares machine performance with human baseline in both static and dynamic scenarios.

Contribution

It introduces CNN models for cross-modal face-audio matching, evaluates static and dynamic testing scenarios, and benchmarks machine performance against human accuracy.

Findings

CNN can solve cross-modal matching above chance levels.

Machine performance matches human accuracy on easy cases.

CNN outperforms humans on challenging face similarity cases.

Abstract

We introduce a seemingly impossible task: given only an audio clip of someone speaking, decide which of two face images is the speaker. In this paper we study this, and a number of related cross-modal tasks, aimed at answering the question: how much can we infer from the voice about the face and vice versa? We study this task "in the wild", employing the datasets that are now publicly available for face recognition from static images (VGGFace) and speaker identification from audio (VoxCeleb). These provide training and testing scenarios for both static and dynamic testing of cross-modal matching. We make the following contributions: (i) we introduce CNN architectures for both binary and multi-way cross-modal face and audio matching, (ii) we compare dynamic testing (where video information is available, but the audio is not from the same video) with static testing (where only a single still image is available), and (iii) we use human testing as a baseline to calibrate the difficulty of the task. We show that a CNN can indeed be trained to solve this task in both the static and dynamic scenarios, and is even well above chance on 10-way classification of the face given the voice. The CNN matches human performance on easy examples (e.g. different gender across faces) but exceeds human performance on more challenging examples (e.g. faces with the same gender, age and nationality).