MDD: A Dataset for Text-and-Music Conditioned Duet Dance Generation

TL;DR

This paper presents MDD, a comprehensive dataset combining motion capture, music, and text annotations for advancing text-controlled and music-conditioned duet dance generation, along with baseline tasks and evaluations.

Contribution

The introduction of MDD, the first dataset integrating human motion, music, and text for duet dance generation, enabling new research in multimodal dance synthesis.

Findings

MDD contains 620 minutes of motion data with detailed annotations.

Baseline models demonstrate the feasibility of text- and music-conditioned dance generation.

Two novel tasks are proposed and evaluated for future research.

Abstract

We introduce Multimodal DuetDance (MDD), a diverse multimodal benchmark dataset designed for text-controlled and music-conditioned 3D duet dance motion generation. Our dataset comprises 620 minutes of high-quality motion capture data performed by professional dancers, synchronized with music, and detailed with over 10K fine-grained natural language descriptions. The annotations capture a rich movement vocabulary, detailing spatial relationships, body movements, and rhythm, making MDD the first dataset to seamlessly integrate human motions, music, and text for duet dance generation. We introduce two novel tasks supported by our dataset: (1) Text-to-Duet, where given music and a textual prompt, both the leader and follower dance motion are generated (2) Text-to-Dance Accompaniment, where given music, textual prompt, and the leader's motion, the follower's motion is generated in a cohesive, text-aligned manner. We include baseline evaluations on both tasks to support future research.