Controllable Dance Generation with Style-Guided Motion Diffusion

TL;DR

This paper introduces Style-Guided Motion Diffusion (SGMD), a novel method for controllable dance generation that aligns dance sequences with music styles and user prompts using a Transformer-based architecture and a spatial-temporal masking mechanism.

Contribution

It presents a new controllable dance generation framework integrating style modulation and a masking mechanism, along with benchmarks for various dance generation tasks.

Findings

Generates realistic, stylistically consistent dances.

Enables flexible control over dance styles and movements.

Outperforms existing methods in alignment with music and style.

Abstract

Dance plays an important role as an artistic form and expression in human culture, yet automatically generating dance sequences is a significant yet challenging endeavor. Existing approaches often neglect the critical aspect of controllability in dance generation. Additionally, they inadequately model the nuanced impact of music styles, resulting in dances that lack alignment with the expressive characteristics inherent in the conditioned music. To address this gap, we propose Style-Guided Motion Diffusion (SGMD), which integrates the Transformer-based architecture with a Style Modulation module. By incorporating music features with user-provided style prompts, the SGMD ensures that the generated dances not only match the musical content but also reflect the desired stylistic characteristics. To enable flexible control over the generated dances, we introduce a spatial-temporal masking mechanism. As controllable dance generation has not been fully studied, we construct corresponding experimental setups and benchmarks for tasks such as trajectory-based dance generation, dance in-betweening, and dance inpainting. Extensive experiments demonstrate that our approach can generate realistic and stylistically consistent dances, while also empowering users to create dances tailored to diverse artistic and practical needs. Code is available on Github: https://github.com/mucunzhuzhu/DGSDP