Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades

TL;DR

This paper introduces a two-stage framework that generates complex human motion videos from text by first creating 2D skeleton sequences and then synthesizing videos, addressing the limitations of existing methods.

Contribution

It presents a novel cascaded approach combining text-to-skeleton and pose-conditioned video diffusion, along with a new synthetic dataset for complex human motion video generation.

Findings

Outperforms prior methods on FID, R-precision, and motion diversity.

Achieves state-of-the-art results on temporal consistency and motion smoothness.

Provides a new dataset with 2,000 synthetic videos of complex motions.

Abstract

Generating videos of complex human motions such as flips, cartwheels, and martial arts remains challenging for current video diffusion models. Text-only conditioning is temporally ambiguous for fine-grained motion control, while explicit pose-based controls, though effective, require users to provide complete skeleton sequences that are costly to produce for long and dynamic actions. We propose a two-stage cascaded framework that addresses both limitations. First, an autoregressive text-to-skeleton model generates 2D pose sequences from natural language descriptions by predicting each joint conditioned on previously generated poses. This design captures long-range temporal dependencies and inter-joint coordination required for complex motions. Second, a pose-conditioned video diffusion model synthesizes videos from a reference image and the generated skeleton sequence. It employs DINO-ALF (Adaptive Layer Fusion), a multi-level reference encoder that preserves appearance and clothing details under large pose changes and self-occlusions. To address the lack of publicly available datasets for complex human motion video generation, we introduce a Blender-based synthetic dataset containing 2,000 videos with diverse characters performing acrobatic and stunt-like motions. The dataset provides full control over appearance, motion, and environment. It fills an important gap because existing benchmarks significantly under-represent acrobatic motions while web-collected datasets raise copyright and privacy concerns. Experiments on our synthetic dataset and the Motion-X Fitness benchmark show that our text-to-skeleton model outperforms prior methods on FID, R-precision, and motion diversity. Our pose-to-video model also achieves the best results among all compared methods on VBench metrics for temporal consistency, motion smoothness, and subject preservation.