Music-oriented Dance Video Synthesis with Pose Perceptual Loss

TL;DR

This paper introduces a novel learning-based method for automatic music video generation that produces realistic dance videos synchronized with music using pose perceptual loss and cross-modal evaluation.

Contribution

It proposes a new pose perceptual loss and a cross-modal evaluation method, advancing the realism and synchronization in music-driven dance video synthesis.

Findings

Generated dance videos are highly realistic and synchronized with music.

The pose perceptual loss improves naturalness of generated dances.

Cross-modal evaluation effectively measures dance quality.

Abstract

We present a learning-based approach with pose perceptual loss for automatic music video generation. Our method can produce a realistic dance video that conforms to the beats and rhymes of almost any given music. To achieve this, we firstly generate a human skeleton sequence from music and then apply the learned pose-to-appearance mapping to generate the final video. In the stage of generating skeleton sequences, we utilize two discriminators to capture different aspects of the sequence and propose a novel pose perceptual loss to produce natural dances. Besides, we also provide a new cross-modal evaluation to evaluate the dance quality, which is able to estimate the similarity between two modalities of music and dance. Finally, a user study is conducted to demonstrate that dance video synthesized by the presented approach produces surprisingly realistic results. The results are shown in the supplementary video at https://youtu.be/0rMuFMZa_K4