DiffTalker: Co-driven audio-image diffusion for talking faces via intermediate landmarks

TL;DR

DiffTalker is a novel model that generates realistic talking faces by co-driving audio and landmarks, combining transformer and diffusion networks to produce accurate and articulate speaking faces without extra alignment.

Contribution

It introduces a dual-agent framework with landmark guidance to improve the realism and geometric accuracy of talking face generation using diffusion models.

Findings

Outperforms existing methods in face realism and accuracy

Produces articulate speaking faces without extra feature alignment

Efficiently integrates pre-trained diffusion models for face synthesis

Abstract

Generating realistic talking faces is a complex and widely discussed task with numerous applications. In this paper, we present DiffTalker, a novel model designed to generate lifelike talking faces through audio and landmark co-driving. DiffTalker addresses the challenges associated with directly applying diffusion models to audio control, which are traditionally trained on text-image pairs. DiffTalker consists of two agent networks: a transformer-based landmarks completion network for geometric accuracy and a diffusion-based face generation network for texture details. Landmarks play a pivotal role in establishing a seamless connection between the audio and image domains, facilitating the incorporation of knowledge from pre-trained diffusion models. This innovative approach efficiently produces articulate-speaking faces. Experimental results showcase DiffTalker's superior performance in producing clear and geometrically accurate talking faces, all without the need for additional alignment between audio and image features.