DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering

TL;DR

DFA-NeRF introduces a neural radiance field framework that disentangles lip movements from other facial attributes, enabling high-fidelity, personalized talking head generation synchronized with audio.

Contribution

The paper presents a novel neural radiance field approach that separately models lip movements and personalized attributes, improving realism and synchronization in talking head synthesis.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Achieves high lip synchronization accuracy.

Generates natural head movements and eye blinks.

Abstract

While recent advances in deep neural networks have made it possible to render high-quality images, generating photo-realistic and personalized talking head remains challenging. With given audio, the key to tackling this task is synchronizing lip movement and simultaneously generating personalized attributes like head movement and eye blink. In this work, we observe that the input audio is highly correlated to lip motion while less correlated to other personalized attributes (e.g., head movements). Inspired by this, we propose a novel framework based on neural radiance field to pursue high-fidelity and personalized talking head generation. Specifically, neural radiance field takes lip movements features and personalized attributes as two disentangled conditions, where lip movements are directly predicted from the audio inputs to achieve lip-synchronized generation. In the meanwhile, personalized attributes are sampled from a probabilistic model, where we design a Transformer-based variational autoencoder sampled from Gaussian Process to learn plausible and natural-looking head pose and eye blink. Experiments on several benchmarks demonstrate that our method achieves significantly better results than state-of-the-art methods.