X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention

TL;DR

X-Portrait introduces a hierarchical diffusion-based framework for expressive, temporally coherent portrait animation from a single image, utilizing novel control signals for fine-grained motion and identity preservation.

Contribution

It presents a new diffusion model with a motion control module that interprets dynamics directly from RGB inputs, enhancing realism and control in portrait animation.

Findings

Effective in diverse portrait styles and expressions

Achieves high fidelity in facial motion and head movements

Maintains identity consistency across animations

Abstract

We propose X-Portrait, an innovative conditional diffusion model tailored for generating expressive and temporally coherent portrait animation. Specifically, given a single portrait as appearance reference, we aim to animate it with motion derived from a driving video, capturing both highly dynamic and subtle facial expressions along with wide-range head movements. As its core, we leverage the generative prior of a pre-trained diffusion model as the rendering backbone, while achieve fine-grained head pose and expression control with novel controlling signals within the framework of ControlNet. In contrast to conventional coarse explicit controls such as facial landmarks, our motion control module is learned to interpret the dynamics directly from the original driving RGB inputs. The motion accuracy is further enhanced with a patch-based local control module that effectively enhance the motion attention to small-scale nuances like eyeball positions. Notably, to mitigate the identity leakage from the driving signals, we train our motion control modules with scaling-augmented cross-identity images, ensuring maximized disentanglement from the appearance reference modules. Experimental results demonstrate the universal effectiveness of X-Portrait across a diverse range of facial portraits and expressive driving sequences, and showcase its proficiency in generating captivating portrait animations with consistently maintained identity characteristics.