MACE-Dance: Motion-Appearance Cascaded Experts for Music-Driven Dance Video Generation

TL;DR

MACE-Dance is a novel framework for music-driven dance video generation that combines a motion expert and an appearance expert to produce high-quality, realistic dance videos with preserved identity and expressive motion.

Contribution

The paper introduces MACE-Dance, a cascaded Mixture-of-Experts framework with state-of-the-art performance in 3D dance generation and pose-driven image animation.

Findings

Achieves SOTA in 3D dance generation with a diffusion model and BiMamba-Transformer architecture.

Attains SOTA in pose-driven image animation through a decoupled kinematic-aesthetic fine-tuning strategy.

Develops a large-scale dataset and evaluation protocol for benchmarking dance video generation.

Abstract

With the rise of online dance-video platforms and rapid advances in AI-generated content (AIGC), music-driven dance generation has emerged as a compelling research direction. Despite substantial progress in related domains such as music-driven 3D dance generation, pose-driven image animation, and audio-driven talking-head synthesis, existing methods cannot be directly adapted to this task. Moreover, the limited studies in this area still struggle to jointly achieve high-quality visual appearance and realistic human motion. Accordingly, we present MACE-Dance, a music-driven dance video generation framework with cascaded Mixture-of-Experts (MoE). The Motion Expert performs music-to-3D motion generation while enforcing kinematic plausibility and artistic expressiveness, whereas the Appearance Expert carries out motion- and reference-conditioned video synthesis, preserving visual identity with spatiotemporal coherence. Specifically, the Motion Expert adopts a diffusion model with a BiMamba-Transformer hybrid architecture and a Guidance-Free Training (GFT) strategy, achieving state-of-the-art (SOTA) performance in 3D dance generation. The Appearance Expert employs a decoupled kinematic-aesthetic fine-tuning strategy, achieving state-of-the-art (SOTA) performance in pose-driven image animation. To better benchmark this task, we curate a large-scale and diverse dataset and design a motion-appearance evaluation protocol. Based on this protocol, MACE-Dance also achieves state-of-the-art performance. Code is available at https://github.com/AMAP-ML/MACE-Dance.