InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalization

TL;DR

This paper introduces a scalable approach for 3D dance generation that generalizes well to in-the-wild scenarios by combining high-quality data reconstruction and advanced model design, including retrieval-augmented generation and rhythm adaptation.

Contribution

It presents a novel pipeline for reconstructing high-fidelity 3D dance data and a scalable LLaMA-based model with retrieval and MoE modules for improved generalization and rhythm adaptation.

Findings

Outperforms existing methods in qualitative evaluations

Produces diverse, physically plausible 3D dances from unseen music

Demonstrates robustness across various dance genres

Abstract

Although existing 3D dance generation methods perform well in controlled scenarios, they often struggle to generalize in the wild. When conditioned on unseen music, existing methods often produce unstructured or physically implausible dance, largely due to limited music-to-dance data and restricted model capacity. This work aims to push the frontier of generalizable 3D dance generation by scaling up both data and model design. (1) On the data side, we develop a fully automated pipeline that reconstructs high-fidelity 3D dance motions from monocular videos. To eliminate the physical artifacts prevalent in existing reconstruction methods, we introduce a Foot Restoration Diffusion Model (FRDM) guided by foot-contact and geometric constraints that enforce physical plausibility while preserving kinematic smoothness and expressiveness, resulting in a diverse, high-quality multimodal 3D dance dataset totaling 100.69 hours. (2) On model design, we propose Choreographic LLaMA (ChoreoLLaMA), a scalable LLaMA-based architecture. To enhance robustness under unfamiliar music conditions, we integrate a retrieval-augmented generation (RAG) module that injects reference dance as a prompt. Additionally, we design a slow/fast-cadence Mixture-of-Experts (MoE) module that enables ChoreoLLaMA to smoothly adapt motion rhythms across varying music tempos. Extensive experiments across diverse dance genres show that our approach surpasses existing methods in both qualitative and quantitative evaluations, marking a step toward scalable, real-world 3D dance generation. Code, models, and data will be released.