Toward Fine-Grained Facial Control in 3D Talking Head Generation

TL;DR

This paper introduces FG-3DGS, a novel framework for fine-grained, high-fidelity 3D talking head generation that improves lip synchronization and facial detail control using frequency-aware modeling and post-rendering refinement.

Contribution

The paper proposes a frequency-aware disentanglement strategy and a high-frequency refinement mechanism to enhance control and realism in 3D talking head generation.

Findings

Outperforms state-of-the-art methods in lip synchronization accuracy.

Achieves high-fidelity facial detail and temporal consistency.

Demonstrates robustness across multiple datasets.

Abstract

Audio-driven talking head generation is a core component of digital avatars, and 3D Gaussian Splatting has shown strong performance in real-time rendering of high-fidelity talking heads. However, achieving precise control over fine-grained facial movements remains a significant challenge, particularly due to lip-synchronization inaccuracies and facial jitter, both of which can contribute to the uncanny valley effect. To address these challenges, we propose Fine-Grained 3D Gaussian Splatting (FG-3DGS), a novel framework that enables temporally consistent and high-fidelity talking head generation. Our method introduces a frequency-aware disentanglement strategy to explicitly model facial regions based on their motion characteristics. Low-frequency regions, such as the cheeks, nose, and forehead, are jointly modeled using a standard MLP, while high-frequency regions, including the eyes and mouth, are captured separately using a dedicated network guided by facial area masks. The predicted motion dynamics, represented as Gaussian deltas, are applied to the static Gaussians to generate the final head frames, which are rendered via a rasterizer using frame-specific camera parameters. Additionally, a high-frequency-refined post-rendering alignment mechanism, learned from large-scale audio-video pairs by a pretrained model, is incorporated to enhance per-frame generation and achieve more accurate lip synchronization. Extensive experiments on widely used datasets for talking head generation demonstrate that our method outperforms recent state-of-the-art approaches in producing high-fidelity, lip-synced talking head videos.