OT-Talk: Animating 3D Talking Head with Optimal Transportation

TL;DR

OT-Talk introduces a novel method using optimal transportation and advanced geometric features to improve the accuracy and naturalness of 3D talking head animations driven by audio signals.

Contribution

This work is the first to apply optimal transportation and Chebyshev Graph Convolution in 3D talking head animation, enhancing mesh modeling and lip-sync accuracy.

Findings

Outperforms state-of-the-art methods in quantitative metrics

Achieves more natural and coherent facial animations

Validated by user perception study with 20 volunteers

Abstract

Animating 3D head meshes using audio inputs has significant applications in AR/VR, gaming, and entertainment through 3D avatars. However, bridging the modality gap between speech signals and facial dynamics remains a challenge, often resulting in incorrect lip syncing and unnatural facial movements. To address this, we propose OT-Talk, the first approach to leverage optimal transportation to optimize the learning model in talking head animation. Building on existing learning frameworks, we utilize a pre-trained Hubert model to extract audio features and a transformer model to process temporal sequences. Unlike previous methods that focus solely on vertex coordinates or displacements, we introduce Chebyshev Graph Convolution to extract geometric features from triangulated meshes. To measure mesh dissimilarities, we go beyond traditional mesh reconstruction errors and velocity differences between adjacent frames. Instead, we represent meshes as probability measures and approximate their surfaces. This allows us to leverage the sliced Wasserstein distance for modeling mesh variations. This approach facilitates the learning of smooth and accurate facial motions, resulting in coherent and natural facial animations. Our experiments on two public audio-mesh datasets demonstrate that our method outperforms state-of-the-art techniques both quantitatively and qualitatively in terms of mesh reconstruction accuracy and temporal alignment. In addition, we conducted a user perception study with 20 volunteers to further assess the effectiveness of our approach.