TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation

TL;DR

This paper introduces Test-time Procrustes Calibration (TPC), a simple, model-agnostic method that improves diffusion-based human image animation by maintaining alignment between reference and target images, especially in real-world scenarios.

Contribution

The paper proposes TPC, a novel test-time calibration technique that enhances diffusion-based human image animation robustness against compositional misalignment without extra training.

Findings

TPC improves animation fidelity in real-world conditions.

The method is simple and applicable to any diffusion-based system.

It maintains high performance even with shape misalignments.

Abstract

Human image animation aims to generate a human motion video from the inputs of a reference human image and a target motion video. Current diffusion-based image animation systems exhibit high precision in transferring human identity into targeted motion, yet they still exhibit irregular quality in their outputs. Their optimal precision is achieved only when the physical compositions (i.e., scale and rotation) of the human shapes in the reference image and target pose frame are aligned. In the absence of such alignment, there is a noticeable decline in fidelity and consistency. Especially, in real-world environments, this compositional misalignment commonly occurs, posing significant challenges to the practical usage of current systems. To this end, we propose Test-time Procrustes Calibration (TPC), which enhances the robustness of diffusion-based image animation systems by maintaining optimal performance even when faced with compositional misalignment, effectively addressing real-world scenarios. The TPC provides a calibrated reference image for the diffusion model, enhancing its capability to understand the correspondence between human shapes in the reference and target images. Our method is simple and can be applied to any diffusion-based image animation system in a model-agnostic manner, improving the effectiveness at test time without additional training.