GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expression

TL;DR

GoHD is a novel framework for generating highly realistic, expressive, and controllable portrait videos from any identity and motion, integrating gaze, prosody-aware head poses, and lip synchronization with limited data.

Contribution

It introduces a disentangled animation module with gaze control, a prosody-aware diffusion model for head poses, and a two-stage training strategy for lip synchronization, advancing portrait animation technology.

Findings

Demonstrates superior generalization to unseen subjects.

Produces realistic, expressive talking head videos.

Effectively decouples lip motion from other facial motions.

Abstract

Audio-driven talking head generation necessitates seamless integration of audio and visual data amidst the challenges posed by diverse input portraits and intricate correlations between audio and facial motions. In response, we propose a robust framework GoHD designed to produce highly realistic, expressive, and controllable portrait videos from any reference identity with any motion. GoHD innovates with three key modules: Firstly, an animation module utilizing latent navigation is introduced to improve the generalization ability across unseen input styles. This module achieves high disentanglement of motion and identity, and it also incorporates gaze orientation to rectify unnatural eye movements that were previously overlooked. Secondly, a conformer-structured conditional diffusion model is designed to guarantee head poses that are aware of prosody. Thirdly, to estimate lip-synchronized and realistic expressions from the input audio within limited training data, a two-stage training strategy is devised to decouple frequent and frame-wise lip motion distillation from the generation of other more temporally dependent but less audio-related motions, e.g., blinks and frowns. Extensive experiments validate GoHD's advanced generalization capabilities, demonstrating its effectiveness in generating realistic talking face results on arbitrary subjects.