StableAnimator++: Overcoming Pose Misalignment and Face Distortion for Human Image Animation

TL;DR

StableAnimator++ is a novel video diffusion framework that preserves identity and improves pose alignment in human image animation by integrating learnable transformations, face encoding, and HJB-based face optimization.

Contribution

It introduces a comprehensive ID-preserving diffusion approach with learnable pose alignment and face optimization, addressing pose misalignment and face distortion issues.

Findings

Enhanced identity preservation in generated videos.

Effective pose alignment via learnable similarity transformations.

Improved facial fidelity through HJB-based face optimization.

Abstract

Current diffusion models for human image animation often struggle to maintain identity (ID) consistency, especially when the reference image and driving video differ significantly in body size or position. We introduce StableAnimator++, the first ID-preserving video diffusion framework with learnable pose alignment, capable of generating high-quality videos conditioned on a reference image and a pose sequence without any post-processing. Building upon a video diffusion model, StableAnimator++ contains carefully designed modules for both training and inference, striving for identity consistency. In particular, StableAnimator++ first uses learnable layers to predict the similarity transformation matrices between the reference image and the driven poses via injecting guidance from Singular Value Decomposition (SVD). These matrices align the driven poses with the reference image, mitigating misalignment to a great extent. StableAnimator++ then computes image and face embeddings using off-the-shelf encoders, refining the face embeddings via a global content-aware Face Encoder. To further maintain ID, we introduce a distribution-aware ID Adapter that counteracts interference caused by temporal layers while preserving ID via distribution alignment. During the inference stage, we propose a novel Hamilton-Jacobi-Bellman (HJB) based face optimization integrated into the denoising process, guiding the diffusion trajectory for enhanced facial fidelity. Experiments on benchmarks show the effectiveness of StableAnimator++ both qualitatively and quantitatively.