MATHDance: Mamba-Transformer Architecture with Uniform Tokenization for High-Quality 3D Dance Generation

TL;DR

MATHDance introduces a novel two-stage framework combining a kinematic-dynamic quantization and a Mamba-Transformer architecture to generate high-quality, choreographically consistent 3D dance motions from music.

Contribution

It proposes a new hybrid architecture and quantization method that significantly improve music-to-dance generation quality and consistency.

Findings

Achieves state-of-the-art performance on the FineDance dataset.

Effectively encodes dance motions with high fidelity using KDQS.

Demonstrates improved choreographic consistency in generated dances.

Abstract

Music-to-dance generation represents a challenging yet pivotal task at the intersection of choreography, virtual reality, and creative content generation. Despite its significance, existing methods face substantial limitation in achieving choreographic consistency. To address the challenge, we propose MatchDance, a novel framework for music-to-dance generation that constructs a latent representation to enhance choreographic consistency. MatchDance employs a two-stage design: (1) a Kinematic-Dynamic-based Quantization Stage (KDQS), which encodes dance motions into a latent representation by Finite Scalar Quantization (FSQ) with kinematic-dynamic constraints and reconstructs them with high fidelity, and (2) a Hybrid Music-to-Dance Generation Stage(HMDGS), which uses a Mamba-Transformer hybrid architecture to map music into the latent representation, followed by the KDQS decoder to generate 3D dance motions. Additionally, a music-dance retrieval framework and comprehensive metrics are introduced for evaluation. Extensive experiments on the FineDance dataset demonstrate state-of-the-art performance.