Not Like Transformers: Drop the Beat Representation for Dance Generation with Mamba-Based Diffusion Model

TL;DR

This paper introduces MambaDance, a novel dance generation method using a Mamba-based diffusion model and a Gaussian beat representation, effectively capturing dance's rhythmic and sequential nature.

Contribution

It replaces Transformer with Mamba in a diffusion architecture and incorporates a Gaussian beat representation to improve dance sequence generation.

Findings

Effective generation of plausible dance movements across various sequence lengths.

Outperforms previous methods in capturing dance's rhythmic and sequential characteristics.

Demonstrates versatility on multiple datasets with qualitative and quantitative results.

Abstract

Dance is a form of human motion characterized by emotional expression and communication, playing a role in various fields such as music, virtual reality, and content creation. Existing methods for dance generation often fail to adequately capture the inherently sequential, rhythmical, and music-synchronized characteristics of dance. In this paper, we propose \emph{MambaDance}, a new dance generation approach that leverages a Mamba-based diffusion model. Mamba, well-suited to handling long and autoregressive sequences, is integrated into our two-stage diffusion architecture, substituting off-the-shelf Transformer. Additionally, considering the critical role of musical beats in dance choreography, we propose a Gaussian-based beat representation to explicitly guide the decoding of dance sequences. Experiments on AIST++ and FineDance datasets for each sequence length show that our proposed method effectively generates plausible dance movements while reflecting essential characteristics, consistently from short to long dances, compared to the previous methods. Additional qualitative results and demo videos are available at \small{https://vision3d-lab.github.io/mambadance}.