Emotion-Controllable Generalized Talking Face Generation

TL;DR

This paper introduces a novel one-shot, emotion-controllable talking face generation method that generalizes to unseen faces by leveraging a graph convolutional neural network and a two-branch texture synthesis approach.

Contribution

It presents a new facial geometry-aware framework with emotion control and one-shot adaptation, improving generalization to arbitrary faces in-the-wild.

Findings

Achieves emotion-controllable face generation with high realism.

Generalizes to unseen faces with only a single neutral image.

Outperforms previous methods in naturalness and flexibility.

Abstract

Despite the significant progress in recent years, very few of the AI-based talking face generation methods attempt to render natural emotions. Moreover, the scope of the methods is majorly limited to the characteristics of the training dataset, hence they fail to generalize to arbitrary unseen faces. In this paper, we propose a one-shot facial geometry-aware emotional talking face generation method that can generalize to arbitrary faces. We propose a graph convolutional neural network that uses speech content feature, along with an independent emotion input to generate emotion and speech-induced motion on facial geometry-aware landmark representation. This representation is further used in our optical flow-guided texture generation network for producing the texture. We propose a two-branch texture generation network, with motion and texture branches designed to consider the motion and texture content independently. Compared to the previous emotion talking face methods, our method can adapt to arbitrary faces captured in-the-wild by fine-tuning with only a single image of the target identity in neutral emotion.