Taming Transformer for Emotion-Controllable Talking Face Generation

TL;DR

This paper introduces a novel transformer-based method for emotion-controllable talking face generation, effectively modeling multimodal relationships to synthesize realistic, emotion-specific talking videos while preserving identity.

Contribution

The paper proposes a new approach using pre-training, visual token quantization, and an emotion-anchor representation within an autoregressive transformer for emotion-controllable face synthesis.

Findings

Outperforms existing methods qualitatively and quantitatively

Successfully models multiple emotional states conditioned on audio

Achieves high-quality, identity-preserving talking face videos

Abstract

Talking face generation is a novel and challenging generation task, aiming at synthesizing a vivid speaking-face video given a specific audio. To fulfill emotion-controllable talking face generation, current methods need to overcome two challenges: One is how to effectively model the multimodal relationship related to the specific emotion, and the other is how to leverage this relationship to synthesize identity preserving emotional videos. In this paper, we propose a novel method to tackle the emotion-controllable talking face generation task discretely. Specifically, we employ two pre-training strategies to disentangle audio into independent components and quantize videos into combinations of visual tokens. Subsequently, we propose the emotion-anchor (EA) representation that integrates the emotional information into visual tokens. Finally, we introduce an autoregressive transformer to model the global distribution of the visual tokens under the given conditions and further predict the index sequence for synthesizing the manipulated videos. We conduct experiments on the MEAD dataset that controls the emotion of videos conditioned on multiple emotional audios. Extensive experiments demonstrate the superiorities of our method both qualitatively and quantitatively.