Cross-Modal Perceptionist: Can Face Geometry be Gleaned from Voices?

TL;DR

This paper investigates whether 3D face geometry can be inferred from voices, proposing a new dataset and analysis framework to explore the physiological correlation between voice and face structure.

Contribution

It introduces Voxceleb-3D dataset and a cross-modal analysis framework for understanding voice-to-face geometry relationships, including supervised and unsupervised learning methods.

Findings

Voice and face geometry are correlated, consistent with neuroscience.

Face geometry can be partially reconstructed from voices.

The framework enables explainable cross-modal perception analysis.

Abstract

This work digs into a root question in human perception: can face geometry be gleaned from one's voices? Previous works that study this question only adopt developments in image synthesis and convert voices into face images to show correlations, but working on the image domain unavoidably involves predicting attributes that voices cannot hint, including facial textures, hairstyles, and backgrounds. We instead investigate the ability to reconstruct 3D faces to concentrate on only geometry, which is much more physiologically grounded. We propose our analysis framework, Cross-Modal Perceptionist, under both supervised and unsupervised learning. First, we construct a dataset, Voxceleb-3D, which extends Voxceleb and includes paired voices and face meshes, making supervised learning possible. Second, we use a knowledge distillation mechanism to study whether face geometry can still be gleaned from voices without paired voices and 3D face data under limited availability of 3D face scans. We break down the core question into four parts and perform visual and numerical analyses as responses to the core question. Our findings echo those in physiology and neuroscience about the correlation between voices and facial structures. The work provides future human-centric cross-modal learning with explainable foundations. See our project page: https://choyingw.github.io/works/Voice2Mesh/index.html