OpenDance: Multimodal Controllable 3D Dance Generation with Large-scale Internet Data

TL;DR

OpenDance introduces a large-scale, richly annotated dance dataset and a novel multimodal generative model that enables controllable, diverse, and realistic 3D dance synthesis conditioned on music, text, keypoints, or trajectories.

Contribution

The paper presents OpenDanceSet, a comprehensive dance dataset, and OpenDanceNet, a unified framework for multimodal, controllable 3D dance generation with high fidelity and diversity.

Findings

High-fidelity dance synthesis with diverse styles.

Effective control over spatial and stylistic conditions.

Robust cross-modal learning enabled by rich annotations.

Abstract

Music-driven 3D dance generation offers significant creative potential, yet practical applications demand versatile and multimodal control. As the highly dynamic and complex human motion covering various styles and genres, dance generation requires satisfying diverse conditions beyond just music (e.g., spatial trajectories, keyframe gestures, or style descriptions). However, the absence of a large-scale and richly annotated dataset severely hinders progress. In this paper, we build OpenDanceSet, an extensive human dance dataset comprising over 100 hours across 14 genres and 147 subjects. Each sample has rich annotations to facilitate robust cross-modal learning: 3D motion, paired music, 2D keypoints, trajectories, and expert-annotated text descriptions. Furthermore, we propose OpenDanceNet, a unified masked modeling framework for controllable dance generation, including a disentangled auto-encoder and a multimodal joint-prediction Transformer. OpenDanceNet supports generation conditioned on music and arbitrary combinations of text, keypoints, or trajectories. Comprehensive experiments demonstrate that our work achieves high-fidelity synthesis with strong diversity and realistic physical contacts, while also offering flexible control over spatial and stylistic conditions. Project Page: https://open-dance.github.io