TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integration

TL;DR

This paper introduces TM2D, a method for generating 3D dance movements guided by both music and text, using a novel VQ-VAE and cross-modal transformer to effectively combine modalities and produce realistic dance motions.

Contribution

The paper presents a new approach that integrates text and music for 3D dance generation, overcoming data limitations with a VQ-VAE and novel metrics for evaluation.

Findings

Generated dances are realistic and coherent with both music and text.

The approach maintains performance comparable to single-modality methods.

New metrics effectively evaluate motion quality and freezing issues.

Abstract

We propose a novel task for generating 3D dance movements that simultaneously incorporate both text and music modalities. Unlike existing works that generate dance movements using a single modality such as music, our goal is to produce richer dance movements guided by the instructive information provided by the text. However, the lack of paired motion data with both music and text modalities limits the ability to generate dance movements that integrate both. To alleviate this challenge, we propose to utilize a 3D human motion VQ-VAE to project the motions of the two datasets into a latent space consisting of quantized vectors, which effectively mix the motion tokens from the two datasets with different distributions for training. Additionally, we propose a cross-modal transformer to integrate text instructions into motion generation architecture for generating 3D dance movements without degrading the performance of music-conditioned dance generation. To better evaluate the quality of the generated motion, we introduce two novel metrics, namely Motion Prediction Distance (MPD) and Freezing Score (FS), to measure the coherence and freezing percentage of the generated motion. Extensive experiments show that our approach can generate realistic and coherent dance movements conditioned on both text and music while maintaining comparable performance with the two single modalities. Code is available at https://garfield-kh.github.io/TM2D/.