DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generation

TL;DR

DREAM-Talk is a diffusion-based framework that generates realistic, emotionally expressive talking face videos from a single image, balancing lip-sync accuracy and expressive diversity.

Contribution

It introduces EmoDiff, a novel diffusion module for dynamic emotional expression generation, and a two-stage process for improved lip-sync and expressiveness in talking face synthesis.

Findings

Outperforms state-of-the-art in expressiveness and lip-sync accuracy

Generates diverse emotional expressions aligned with audio

Achieves high perceptual quality in synthesized videos

Abstract

The generation of emotional talking faces from a single portrait image remains a significant challenge. The simultaneous achievement of expressive emotional talking and accurate lip-sync is particularly difficult, as expressiveness is often compromised for the accuracy of lip-sync. As widely adopted by many prior works, the LSTM network often fails to capture the subtleties and variations of emotional expressions. To address these challenges, we introduce DREAM-Talk, a two-stage diffusion-based audio-driven framework, tailored for generating diverse expressions and accurate lip-sync concurrently. In the first stage, we propose EmoDiff, a novel diffusion module that generates diverse highly dynamic emotional expressions and head poses in accordance with the audio and the referenced emotion style. Given the strong correlation between lip motion and audio, we then refine the dynamics with enhanced lip-sync accuracy using audio features and emotion style. To this end, we deploy a video-to-video rendering module to transfer the expressions and lip motions from our proxy 3D avatar to an arbitrary portrait. Both quantitatively and qualitatively, DREAM-Talk outperforms state-of-the-art methods in terms of expressiveness, lip-sync accuracy and perceptual quality.