FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning

TL;DR

FACIAL is a novel GAN-based method that synthesizes realistic talking face videos from audio and reference images, capturing both speech-related and implicit facial attributes like eye blinks and head movements.

Contribution

The paper introduces FACIAL-GAN, a new generative model that effectively models complex relationships among facial attributes for realistic talking face synthesis.

Findings

Produces highly realistic talking face videos with natural movements.

Outperforms state-of-the-art methods in quality and synchronization.

Generates natural eye blinks and head poses in sync with speech.

Abstract

In this paper, we propose a talking face generation method that takes an audio signal as input and a short target video clip as reference, and synthesizes a photo-realistic video of the target face with natural lip motions, head poses, and eye blinks that are in-sync with the input audio signal. We note that the synthetic face attributes include not only explicit ones such as lip motions that have high correlations with speech, but also implicit ones such as head poses and eye blinks that have only weak correlation with the input audio. To model such complicated relationships among different face attributes with input audio, we propose a FACe Implicit Attribute Learning Generative Adversarial Network (FACIAL-GAN), which integrates the phonetics-aware, context-aware, and identity-aware information to synthesize the 3D face animation with realistic motions of lips, head poses, and eye blinks. Then, our Rendering-to-Video network takes the rendered face images and the attention map of eye blinks as input to generate the photo-realistic output video frames. Experimental results and user studies show our method can generate realistic talking face videos with not only synchronized lip motions, but also natural head movements and eye blinks, with better qualities than the results of state-of-the-art methods.