Can Pose Transfer Models Generate Realistic Human Motion?

TL;DR

This paper evaluates the realism and consistency of three state-of-the-art pose transfer methods in generating human action videos outside their training distribution, revealing significant challenges in accurate action recognition and perceived realism.

Contribution

The study provides a comprehensive evaluation of three leading pose transfer models on out-of-distribution actions, highlighting their limitations in realism and consistency through a participant study.

Findings

Participants correctly identified actions only 42.92% of the time.

Participants found actions consistent with source videos only 36.46% of the time.

ExAvatar was rated more consistent and photorealistic than other methods.

Abstract

Recent pose-transfer methods aim to generate temporally consistent and fully controllable videos of human action where the motion from a reference video is reenacted by a new identity. We evaluate three state-of-the-art pose-transfer methods -- AnimateAnyone, MagicAnimate, and ExAvatar -- by generating videos with actions and identities outside the training distribution and conducting a participant study about the quality of these videos. In a controlled environment of 20 distinct human actions, we find that participants, presented with the pose-transferred videos, correctly identify the desired action only 42.92% of the time. Moreover, the participants find the actions in the generated videos consistent with the reference (source) videos only 36.46% of the time. These results vary by method: participants find the splatting-based ExAvatar more consistent and photorealistic than the diffusion-based AnimateAnyone and MagicAnimate.