Toward Rich Video Human-Motion2D Generation

TL;DR

This paper introduces a large-scale dataset and a diffusion-based model for realistic, controllable 2D human motion video generation, especially for multi-character interactions, with improved realism and text alignment.

Contribution

It presents a new dataset with diverse multi-character actions and a novel diffusion-based model with advanced textual conditioning and reinforcement learning for improved motion generation.

Findings

RVHM2D achieves state-of-the-art results on the Motion2D-Video-150K benchmark.

The dataset enables better modeling of multi-character interactions.

The model produces highly realistic and text-aligned human motion videos.

Abstract

Generating realistic and controllable human motions, particularly those involving rich multi-character interactions, remains a significant challenge due to data scarcity and the complexities of modeling inter-personal dynamics. To address these limitations, we first introduce a new large-scale rich video human motion 2D dataset (Motion2D-Video-150K) comprising 150,000 video sequences. Motion2D-Video-150K features a balanced distribution of diverse single-character and, crucially, double-character interactive actions, each paired with detailed textual descriptions. Building upon this dataset, we propose a novel diffusion-based rich video human motion2D generation (RVHM2D) model. RVHM2D incorporates an enhanced textual conditioning mechanism utilizing either dual text encoders (CLIP-L/B) or T5-XXL with both global and local features. We devise a two-stage training strategy: the model is first trained with a standard diffusion objective, and then fine-tuned using reinforcement learning with an FID-based reward to further enhance motion realism and text alignment. Extensive experiments demonstrate that RVHM2D achieves leading performance on the Motion2D-Video-150K benchmark in generating both single and interactive double-character scenarios.