Dance Style Transfer with Cross-modal Transformer

TL;DR

CycleDance is a novel dance style transfer system that combines multimodal transformers and cycle-consistency to generate realistic dance movements in target styles while preserving motion context.

Contribution

It introduces a new architecture extending CycleGAN with multimodal transformers and sequence curriculum learning for improved dance style transfer.

Findings

Outperforms baseline CycleGAN in naturalness and transfer strength

Captures long-term intra-relations in dance motions

Validated by human study with experienced dancers

Abstract

We present CycleDance, a dance style transfer system to transform an existing motion clip in one dance style to a motion clip in another dance style while attempting to preserve motion context of the dance. Our method extends an existing CycleGAN architecture for modeling audio sequences and integrates multimodal transformer encoders to account for music context. We adopt sequence length-based curriculum learning to stabilize training. Our approach captures rich and long-term intra-relations between motion frames, which is a common challenge in motion transfer and synthesis work. We further introduce new metrics for gauging transfer strength and content preservation in the context of dance movements. We perform an extensive ablation study as well as a human study including 30 participants with 5 or more years of dance experience. The results demonstrate that CycleDance generates realistic movements with the target style, significantly outperforming the baseline CycleGAN on naturalness, transfer strength, and content preservation.