EmoVOCA: Speech-Driven Emotional 3D Talking Heads

TL;DR

This paper introduces EmoVOCA, a synthetic dataset for emotional 3D talking head generation, and presents a new generator that produces speech-driven, expressive facial animations with improved realism.

Contribution

The work creates a novel synthetic dataset combining expressive and inexpressive 3D heads and develops a generator that synthesizes emotionally expressive speech-driven 3D talking heads.

Findings

The dataset enables better modeling of speech and expression dynamics.

The generator produces more convincing and expressive 3D talking head animations.

Experimental results outperform existing methods in realism and expressiveness.

Abstract

The domain of 3D talking head generation has witnessed significant progress in recent years. A notable challenge in this field consists in blending speech-related motions with expression dynamics, which is primarily caused by the lack of comprehensive 3D datasets that combine diversity in spoken sentences with a variety of facial expressions. Whereas literature works attempted to exploit 2D video data and parametric 3D models as a workaround, these still show limitations when jointly modeling the two motions. In this work, we address this problem from a different perspective, and propose an innovative data-driven technique that we used for creating a synthetic dataset, called EmoVOCA, obtained by combining a collection of inexpressive 3D talking heads and a set of 3D expressive sequences. To demonstrate the advantages of this approach, and the quality of the dataset, we then designed and trained an emotional 3D talking head generator that accepts a 3D face, an audio file, an emotion label, and an intensity value as inputs, and learns to animate the audio-synchronized lip movements with expressive traits of the face. Comprehensive experiments, both quantitative and qualitative, using our data and generator evidence superior ability in synthesizing convincing animations, when compared with the best performing methods in the literature. Our code and pre-trained model will be made available.