DanceChat: Large Language Model-Guided Music-to-Dance Generation

TL;DR

DanceChat introduces a novel LLM-guided approach for music-to-dance generation, using textual instructions to improve diversity and alignment with musical styles, addressing the semantic gap and data scarcity issues.

Contribution

The paper proposes a new LLM-guided framework that generates explicit textual dance instructions to enhance diversity and style alignment in music-to-dance synthesis.

Findings

Outperforms state-of-the-art methods in qualitative and quantitative evaluations

Generates more diverse dance movements aligned with musical styles

Effectively incorporates textual guidance to improve dance synthesis

Abstract

Music-to-dance generation aims to synthesize human dance motion conditioned on musical input. Despite recent progress, significant challenges remain due to the semantic gap between music and dance motion, as music offers only abstract cues, such as melody, groove, and emotion, without explicitly specifying the physical movements. Moreover, a single piece of music can produce multiple plausible dance interpretations. This one-to-many mapping demands additional guidance, as music alone provides limited information for generating diverse dance movements. The challenge is further amplified by the scarcity of paired music and dance data, which restricts the model\^a\u{A}\'Zs ability to learn diverse dance patterns. In this paper, we introduce DanceChat, a Large Language Model (LLM)-guided music-to-dance generation approach. We use an LLM as a choreographer that provides textual motion instructions, offering explicit, high-level guidance for dance generation. This approach goes beyond implicit learning from music alone, enabling the model to generate dance that is both more diverse and better aligned with musical styles. Our approach consists of three components: (1) an LLM-based pseudo instruction generation module that produces textual dance guidance based on music style and structure, (2) a multi-modal feature extraction and fusion module that integrates music, rhythm, and textual guidance into a shared representation, and (3) a diffusion-based motion synthesis module together with a multi-modal alignment loss, which ensures that the generated dance is aligned with both musical and textual cues. Extensive experiments on AIST++ and human evaluations show that DanceChat outperforms state-of-the-art methods both qualitatively and quantitatively.