Music-oriented Dance Video Synthesis with Pose Perceptual Loss
Xuanchi Ren, Haoran Li, Zijian Huang, Qifeng Chen

TL;DR
This paper introduces a novel learning-based method for automatic music video generation that produces realistic dance videos synchronized with music using pose perceptual loss and cross-modal evaluation.
Contribution
It proposes a new pose perceptual loss and a cross-modal evaluation method, advancing the realism and synchronization in music-driven dance video synthesis.
Findings
Generated dance videos are highly realistic and synchronized with music.
The pose perceptual loss improves naturalness of generated dances.
Cross-modal evaluation effectively measures dance quality.
Abstract
We present a learning-based approach with pose perceptual loss for automatic music video generation. Our method can produce a realistic dance video that conforms to the beats and rhymes of almost any given music. To achieve this, we firstly generate a human skeleton sequence from music and then apply the learned pose-to-appearance mapping to generate the final video. In the stage of generating skeleton sequences, we utilize two discriminators to capture different aspects of the sequence and propose a novel pose perceptual loss to produce natural dances. Besides, we also provide a new cross-modal evaluation to evaluate the dance quality, which is able to estimate the similarity between two modalities of music and dance. Finally, a user study is conducted to demonstrate that dance video synthesized by the presented approach produces surprisingly realistic results. The results are shown in…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Motion and Animation · Generative Adversarial Networks and Image Synthesis · Video Analysis and Summarization
