Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation

TL;DR

This paper presents a two-stage diffusion model that generates high-quality, temporally coherent talking head videos synchronized with speech by guiding the process with facial landmarks.

Contribution

It introduces a landmark-guided diffusion framework that improves lip synchronization and visual quality in talking head generation, addressing limitations of prior GAN and diffusion models.

Findings

Achieves superior lip synchronization and visual fidelity.

Reduces mouth jitter and enhances temporal coherence.

Outperforms existing methods in extensive experiments.

Abstract

Audio-driven talking head generation is a significant and challenging task applicable to various fields such as virtual avatars, film production, and online conferences. However, the existing GAN-based models emphasize generating well-synchronized lip shapes but overlook the visual quality of generated frames, while diffusion-based models prioritize generating high-quality frames but neglect lip shape matching, resulting in jittery mouth movements. To address the aforementioned problems, we introduce a two-stage diffusion-based model. The first stage involves generating synchronized facial landmarks based on the given speech. In the second stage, these generated landmarks serve as a condition in the denoising process, aiming to optimize mouth jitter issues and generate high-fidelity, well-synchronized, and temporally coherent talking head videos. Extensive experiments demonstrate that our model yields the best performance.