Music-Aligned Holistic 3D Dance Generation via Hierarchical Motion Modeling

TL;DR

This paper introduces SoulDance, a new high-quality dataset and SoulNet, a hierarchical model that generates synchronized, holistic 3D dance sequences aligned with music, addressing previous challenges in data scarcity and complex motion modeling.

Contribution

The paper presents a novel dataset and a hierarchical generative framework that models detailed motion dependencies and ensures music-dance alignment for holistic 3D dance synthesis.

Findings

SoulNet outperforms existing methods in dance quality and alignment.

The hierarchical motion modeling captures complex interdependent movements.

Cross-modal retrieval enhances temporal synchronization and semantic coherence.

Abstract

Well-coordinated, music-aligned holistic dance enhances emotional expressiveness and audience engagement. However, generating such dances remains challenging due to the scarcity of holistic 3D dance datasets, the difficulty of achieving cross-modal alignment between music and dance, and the complexity of modeling interdependent motion across the body, hands, and face. To address these challenges, we introduce SoulDance, a high-precision music-dance paired dataset captured via professional motion capture systems, featuring meticulously annotated holistic dance movements. Building on this dataset, we propose SoulNet, a framework designed to generate music-aligned, kinematically coordinated holistic dance sequences. SoulNet consists of three principal components: (1) Hierarchical Residual Vector Quantization, which models complex, fine-grained motion dependencies across the body, hands, and face; (2) Music-Aligned Generative Model, which composes these hierarchical motion units into expressive and coordinated holistic dance; (3) Music-Motion Retrieval Module, a pre-trained cross-modal model that functions as a music-dance alignment prior, ensuring temporal synchronization and semantic coherence between generated dance and input music throughout the generation process. Extensive experiments demonstrate that SoulNet significantly surpasses existing approaches in generating high-quality, music-coordinated, and well-aligned holistic 3D dance sequences.