Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation

TL;DR

Dimitra is a new audio-driven diffusion framework that generates realistic talking head videos by modeling facial motion with a conditional transformer, using only audio and a reference image, outperforming existing methods.

Contribution

It introduces a novel conditional Motion Diffusion Transformer trained on 3D facial motion sequences, utilizing minimal input signals for enhanced realism in talking head generation.

Findings

Outperforms existing methods on VoxCeleb2 and HDTF datasets.

Improves lip motion, facial expression, and head pose realism.

Utilizes phoneme and transcript features for better video quality.

Abstract

We propose Dimitra, a novel framework for audio-driven talking head generation, streamlined to learn lip motion, facial expression, as well as head pose motion. Specifically, we train a conditional Motion Diffusion Transformer (cMDT) by modeling facial motion sequences with 3D representation. We condition the cMDT with only two input signals, an audio-sequence, as well as a reference facial image. By extracting additional features directly from audio, Dimitra is able to increase quality and realism of generated videos. In particular, phoneme sequences contribute to the realism of lip motion, whereas text transcript to facial expression and head pose realism. Quantitative and qualitative experiments on two widely employed datasets, VoxCeleb2 and HDTF, showcase that Dimitra is able to outperform existing approaches for generating realistic talking heads imparting lip motion, facial expression, and head pose.