EMOdiffhead: Continuously Emotional Control in Talking Head Generation via Diffusion

TL;DR

EMOdiffhead is a novel diffusion-based method for generating emotionally expressive talking head videos with fine-grained emotion control and one-shot capability, overcoming data limitations.

Contribution

The paper introduces EMOdiffhead, enabling emotion control in talking head generation using expression vectors and diffusion models, with improved realism and diversity.

Findings

Achieves state-of-the-art performance in emotional portrait animation.

Enables fine-grained emotion and intensity control.

Supports one-shot emotional talking head generation.

Abstract

The task of audio-driven portrait animation involves generating a talking head video using an identity image and an audio track of speech. While many existing approaches focus on lip synchronization and video quality, few tackle the challenge of generating emotion-driven talking head videos. The ability to control and edit emotions is essential for producing expressive and realistic animations. In response to this challenge, we propose EMOdiffhead, a novel method for emotional talking head video generation that not only enables fine-grained control of emotion categories and intensities but also enables one-shot generation. Given the FLAME 3D model's linearity in expression modeling, we utilize the DECA method to extract expression vectors, that are combined with audio to guide a diffusion model in generating videos with precise lip synchronization and rich emotional expressiveness. This approach not only enables the learning of rich facial information from emotion-irrelevant data but also facilitates the generation of emotional videos. It effectively overcomes the limitations of emotional data, such as the lack of diversity in facial and background information, and addresses the absence of emotional details in emotion-irrelevant data. Extensive experiments and user studies demonstrate that our approach achieves state-of-the-art performance compared to other emotion portrait animation methods.