MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learning

TL;DR

This paper introduces MIMAFace, a novel face animation method that enhances temporal stability and identity preservation by modulating appearance features at motion and identity levels, addressing quality gaps in public datasets.

Contribution

The paper proposes the MIA and ICA modules that improve appearance feature learning and temporal consistency in face animation, advancing beyond existing diffusion-based methods.

Findings

Achieves precise facial motion control and identity preservation.

Generates animation videos with high intra/inter-clip temporal consistency.

Demonstrates superior performance on various datasets.

Abstract

Current diffusion-based face animation methods generally adopt a ReferenceNet (a copy of U-Net) and a large amount of curated self-acquired data to learn appearance features, as robust appearance features are vital for ensuring temporal stability. However, when trained on public datasets, the results often exhibit a noticeable performance gap in image quality and temporal consistency. To address this issue, we meticulously examine the essential appearance features in the facial animation tasks, which include motion-agnostic (e.g., clothing, background) and motion-related (e.g., facial details) texture components, along with high-level discriminative identity features. Drawing from this analysis, we introduce a Motion-Identity Modulated Appearance Learning Module (MIA) that modulates CLIP features at both motion and identity levels. Additionally, to tackle the semantic/ color discontinuities between clips, we design an Inter-clip Affinity Learning Module (ICA) to model temporal relationships across clips. Our method achieves precise facial motion control (i.e., expressions and gaze), faithful identity preservation, and generates animation videos that maintain both intra/inter-clip temporal consistency. Moreover, it easily adapts to various modalities of driving sources. Extensive experiments demonstrate the superiority of our method.