AnyAct: Towards Human Reenactment of Character Motion From Video

TL;DR

AnyAct is a novel method that converts videos of non-human characters into plausible, editable human motions by leveraging sparse local articulated motion cues, enabling effective human reenactment from diverse character videos.

Contribution

It introduces a new approach for character video to human motion transfer using sparse local cues and proposes key training techniques to improve stability and control.

Findings

Produces high-fidelity human reenactments from non-human character videos.

Outperforms existing methods in preserving essential dynamics.

Validated through a new diverse non-human character video benchmark.

Abstract

We study the problem of directly deriving an initial human reenactment from a monocular video of a non-human character. Our goal is not to reconstruct the source character itself but to reinterpret its motion as a plausible and editable human performance for downstream animation authoring. This task is challenging because existing video-based motion capture methods are largely restricted to human-centric structural spaces, while motion retargeting methods typically require structured 3D source motions and known source topologies. Our key insight is that sparse local articulated motion cues can preserve essential dynamics across large structural differences, providing a stable bridge from character video to human reenactment. Based on this observation, we propose AnyAct, which formulates character-video-driven human reenactment as conditional human motion generation from transferable sparse local 2D articulated motion. To make this practical, we introduce three key designs: human-motion-only supervision via augmented 3D-to-2D projection, progressive 3D-to-2D training to alleviate conditioning ambiguity, and global-local motion decoupling for reliable local motion control. We further construct a benchmark primarily covering diverse non-human character videos. Experiments on the benchmark show that AnyAct produces high-fidelity initial human reenactments that preserve the essential dynamics of the characters in reference videos, and further ablation studies validate the effectiveness of its core designs.