Diverse Code Query Learning for Speech-Driven Facial Animation

TL;DR

This paper introduces a novel approach for speech-driven facial animation that generates diverse and controllable lip-synchronized 3D faces by encouraging sample diversity and leveraging a vector-quantized variational auto-encoding framework.

Contribution

It proposes a new method to produce multiple plausible facial motions from the same speech input, explicitly promoting diversity and control in the synthesis process.

Findings

Achieves state-of-the-art diversity in facial animation synthesis.

Effectively models the stochastic nature of facial motions.

Demonstrates superior qualitative and quantitative performance.

Abstract

Speech-driven facial animation aims to synthesize lip-synchronized 3D talking faces following the given speech signal. Prior methods to this task mostly focus on pursuing realism with deterministic systems, yet characterizing the potentially stochastic nature of facial motions has been to date rarely studied. While generative modeling approaches can easily handle the one-to-many mapping by repeatedly drawing samples, ensuring a diverse mode coverage of plausible facial motions on small-scale datasets remains challenging and less explored. In this paper, we propose predicting multiple samples conditioned on the same audio signal and then explicitly encouraging sample diversity to address diverse facial animation synthesis. Our core insight is to guide our model to explore the expressive facial latent space with a diversity-promoting loss such that the desired latent codes for diversification can be ideally identified. To this end, building upon the rich facial prior learned with vector-quantized variational auto-encoding mechanism, our model temporally queries multiple stochastic codes which can be flexibly decoded into a diverse yet plausible set of speech-faithful facial motions. To further allow for control over different facial parts during generation, the proposed model is designed to predict different facial portions of interest in a sequential manner, and compose them to eventually form full-face motions. Our paradigm realizes both diverse and controllable facial animation synthesis in a unified formulation. We experimentally demonstrate that our method yields state-of-the-art performance both quantitatively and qualitatively, especially regarding sample diversity.