FG-Portrait: 3D Flow Guided Editable Portrait Animation

TL;DR

FG-Portrait introduces a geometry-driven, 3D flow-based approach for portrait animation that improves motion transfer accuracy and allows user editing, outperforming existing diffusion-based methods.

Contribution

The paper proposes a novel 3D flow encoding method using parametric head models, integrating geometry priors into diffusion models for improved portrait animation.

Findings

Outperforms existing methods in motion transfer accuracy

Preserves source identity more faithfully

Enables user-controlled facial editing

Abstract

Motion transfer from the driving to the source portrait remains a key challenge in the portrait animation. Current diffusion-based approaches condition only on the driving motion, which fails to capture source-to-driving correspondences and consequently yields suboptimal motion transfer. Although flow estimation provides an alternative, predicting dense correspondences from 2D input is ill-posed and often yields inaccurate animation. We address this problem by introducing 3D flows, a learning-free and geometry-driven motion correspondence directly computed from parametric 3D head models. To integrate this 3D prior into diffusion model, we introduce 3D flow encoding to query potential 3D flows for each target pixel to indicate its displacement back to the source location. To obtain 3D flows aligned with 2D motion changes, we further propose depth-guided sampling to accurately locate the corresponding 3D points for each pixel. Beyond high-fidelity portrait animation, our model further supports user-specified editing of facial expression and head pose. Extensive experiments demonstrate the superiority of our method on consistent driving motion transfer as well as faithful source identity preservation.