Robust Dancer: Long-term 3D Dance Synthesis Using Unpaired Data

TL;DR

Robust Dancer introduces a novel 3D dance synthesis system that generates realistic long-term dance motions from music using unpaired data, leveraging disentangled beat and style representations with a Transformer-based model and long-history attention.

Contribution

It is the first system to enable long-term 3D dance synthesis trained solely on unpaired data, overcoming limitations of data pairing and error accumulation.

Findings

Achieves comparable results to baseline methods without paired data.

Generates robust long-term dance sequences aligned with music.

Outperforms existing methods in unpaired data scenarios.

Abstract

How to automatically synthesize natural-looking dance movements based on a piece of music is an incrementally popular yet challenging task. Most existing data-driven approaches require hard-to-get paired training data and fail to generate long sequences of motion due to error accumulation of autoregressive structure. We present a novel 3D dance synthesis system that only needs unpaired data for training and could generate realistic long-term motions at the same time. For the unpaired data training, we explore the disentanglement of beat and style, and propose a Transformer-based model free of reliance upon paired data. For the synthesis of long-term motions, we devise a new long-history attention strategy. It first queries the long-history embedding through an attention computation and then explicitly fuses this embedding into the generation pipeline via multimodal adaptation gate (MAG). Objective and subjective evaluations show that our results are comparable to strong baseline methods, despite not requiring paired training data, and are robust when inferring long-term music. To our best knowledge, we are the first to achieve unpaired data training - an ability that enables to alleviate data limitations effectively. Our code is released on https://github.com/BFeng14/RobustDancer