MoSA: Motion-Coherent Human Video Generation via Structure-Appearance Decoupling

TL;DR

MoSA introduces a novel decoupled approach for human video generation that separately models structure and appearance, enabling more realistic and complex human motions with fine-grained control and improved interaction modeling.

Contribution

The paper presents MoSA, a structure-appearance decoupling framework for human video synthesis, and introduces a large-scale dataset with diverse complex motions.

Findings

MoSA outperforms existing methods on multiple evaluation metrics.

The decoupling approach improves structural coherence and motion realism.

The dataset enables better training and evaluation of human video models.

Abstract

Existing video generation models predominantly emphasize appearance fidelity while exhibiting limited ability to synthesize complex human motions, such as whole-body movements, long-range dynamics, and fine-grained human-environment interactions. This often leads to unrealistic or physically implausible movements with inadequate structural coherence. To conquer these challenges, we propose MoSA, which decouples the process of human video generation into two components, i.e., structure generation and appearance generation. MoSA first employs a 3D structure transformer to generate a human motion sequence from the text prompt. The remaining video appearance is then synthesized under the guidance of this structural sequence. We achieve fine-grained control over the sparse human structures by introducing Human-Aware Dynamic Control modules with a dense tracking constraint during training. The modeling of human-environment interactions is improved through the proposed contact constraint. Those two components work comprehensively to ensure the structural and appearance fidelity across the generated videos. This paper also contributes a large-scale human video dataset, which features more complex and diverse motions than existing human video datasets. We conduct comprehensive comparisons between MoSA and a variety of approaches, including general video generation models, human video generation models, and human animation models. Experiments demonstrate that MoSA substantially outperforms existing approaches across the majority of evaluation metrics.