Music- and Lyrics-driven Dance Synthesis
Wenjie Yin, Qingyuan Yao, Yi Yu, Hang Yin, Danica Kragic, M{\aa}rten, Bj\"orkman

TL;DR
This paper introduces JustLMD, a novel multimodal dataset combining 3D dance motion, music, and lyrics, and presents a diffusion-based model for generating dance conditioned on both music and lyrics.
Contribution
It provides the first triplet dataset of dance, music, and lyrics and develops a cross-modal diffusion network for semantically enriched dance synthesis.
Findings
JustLMD contains 4.6 hours of dance data with lyrics and music.
The diffusion model effectively generates dance conditioned on lyrics and music.
The dataset enables new research in semantically aware dance synthesis.
Abstract
Lyrics often convey information about the songs that are beyond the auditory dimension, enriching the semantic meaning of movements and musical themes. Such insights are important in the dance choreography domain. However, most existing dance synthesis methods mainly focus on music-to-dance generation, without considering the semantic information. To complement it, we introduce JustLMD, a new multimodal dataset of 3D dance motion with music and lyrics. To the best of our knowledge, this is the first dataset with triplet information including dance motion, music, and lyrics. Additionally, we showcase a cross-modal diffusion-based network designed to generate 3D dance motion conditioned on music and lyrics. The proposed JustLMD dataset encompasses 4.6 hours of 3D dance motion in 1867 sequences, accompanied by musical tracks and their corresponding English lyrics.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Motion and Animation · Music and Audio Processing · Music Technology and Sound Studies
MethodsFocus
