Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors

TL;DR

This paper presents a probabilistic diffusion-based framework for one-shot audio-driven talking head generation that produces diverse, natural facial motions aligned with input audio, outperforming previous auto-regressive methods.

Contribution

The proposed diffusion prior enables diverse facial motion synthesis in talking head generation, improving realism and semantic consistency over prior deterministic approaches.

Findings

Outperforms auto-regressive prior on most metrics

Maintains audio-lip synchronization comparable to prior methods

Generates rich, natural lip-irrelevant facial motions

Abstract

In this paper, we introduce a simple and novel framework for one-shot audio-driven talking head generation. Unlike prior works that require additional driving sources for controlled synthesis in a deterministic manner, we instead probabilistically sample all the holistic lip-irrelevant facial motions (i.e. pose, expression, blink, gaze, etc.) to semantically match the input audio while still maintaining both the photo-realism of audio-lip synchronization and the overall naturalness. This is achieved by our newly proposed audio-to-visual diffusion prior trained on top of the mapping between audio and disentangled non-lip facial representations. Thanks to the probabilistic nature of the diffusion prior, one big advantage of our framework is it can synthesize diverse facial motion sequences given the same audio clip, which is quite user-friendly for many real applications. Through comprehensive evaluations on public benchmarks, we conclude that (1) our diffusion prior outperforms auto-regressive prior significantly on almost all the concerned metrics; (2) our overall system is competitive with prior works in terms of audio-lip synchronization but can effectively sample rich and natural-looking lip-irrelevant facial motions while still semantically harmonized with the audio input.