Quantized GAN for Complex Music Generation from Dance Videos

TL;DR

This paper introduces D2M-GAN, a novel framework that generates complex, multi-style music conditioned on dance videos using a Vector Quantized audio representation, outperforming existing methods in realism and diversity.

Contribution

The paper presents a new multi-modal GAN framework for dance-conditioned music generation employing VQ audio, with extensive experiments and a new TikTok dataset demonstrating its effectiveness.

Findings

Quantitative results show high music consistency and beat correspondence.

The method generates diverse music styles including pop and breaking.

The TikTok dataset enables real-world application testing.

Abstract

We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal framework that generates complex musical samples conditioned on dance videos. Our proposed framework takes dance video frames and human body motions as input, and learns to generate music samples that plausibly accompany the corresponding input. Unlike most existing conditional music generation works that generate specific types of mono-instrumental sounds using symbolic audio representations (e.g., MIDI), and that usually rely on pre-defined musical synthesizers, in this work we generate dance music in complex styles (e.g., pop, breaking, etc.) by employing a Vector Quantized (VQ) audio representation, and leverage both its generality and high abstraction capacity of its symbolic and continuous counterparts. By performing an extensive set of experiments on multiple datasets, and following a comprehensive evaluation protocol, we assess the generative qualities of our proposal against alternatives. The attained quantitative results, which measure the music consistency, beats correspondence, and music diversity, demonstrate the effectiveness of our proposed method. Last but not least, we curate a challenging dance-music dataset of in-the-wild TikTok videos, which we use to further demonstrate the efficacy of our approach in real-world applications -- and which we hope to serve as a starting point for relevant future research.