GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer

TL;DR

GLDiTalker is a novel speech-driven 3D facial animation model that uses a graph latent diffusion transformer to improve lip-sync accuracy and motion diversity, addressing modality misalignment issues in previous methods.

Contribution

It introduces a two-stage training pipeline with graph-enhanced quantized space learning and space-time latent diffusion to improve realism and stability in 3D facial animations.

Findings

Outperforms existing methods in lip-sync accuracy

Achieves higher motion diversity in generated animations

Demonstrates superior stability and realism in benchmarks

Abstract

Speech-driven talking head generation is a critical yet challenging task with applications in augmented reality and virtual human modeling. While recent approaches using autoregressive and diffusion-based models have achieved notable progress, they often suffer from modality inconsistencies, particularly misalignment between audio and mesh, leading to reduced motion diversity and lip-sync accuracy. To address this, we propose GLDiTalker, a novel speech-driven 3D facial animation model based on a Graph Latent Diffusion Transformer. GLDiTalker resolves modality misalignment by diffusing signals within a quantized spatiotemporal latent space. It employs a two-stage training pipeline: the Graph-Enhanced Quantized Space Learning Stage ensures lip-sync accuracy, while the Space-Time Powered Latent Diffusion Stage enhances motion diversity. Together, these stages enable GLDiTalker to generate realistic, temporally stable 3D facial animations. Extensive evaluations on standard benchmarks demonstrate that GLDiTalker outperforms existing methods, achieving superior results in both lip-sync accuracy and motion diversity.