Spatial-Temporal Graph Mamba for Music-Guided Dance Video Synthesis
Hao Tang, Ling Shao, Zhenyu Zhang, Luc Van Gool, Nicu Sebe

TL;DR
This paper introduces STG-Mamba, a novel framework for synthesizing dance videos from music by translating music to skeleton sequences and then skeletons to videos, utilizing a new spatial-temporal graph block and self-supervised regularization.
Contribution
The paper presents a new spatial-temporal graph Mamba block and a self-supervised regularization network for music-guided dance video synthesis, along with a large new dataset.
Findings
STG-Mamba outperforms existing methods significantly.
Effective modeling of joint dependencies improves synthesis quality.
The new dataset supports robust training and evaluation.
Abstract
We propose a novel spatial-temporal graph Mamba (STG-Mamba) for the music-guided dance video synthesis task, i.e., to translate the input music to a dance video. STG-Mamba consists of two translation mappings: music-to-skeleton translation and skeleton-to-video translation. In the music-to-skeleton translation, we introduce a novel spatial-temporal graph Mamba (STGM) block to effectively construct skeleton sequences from the input music, capturing dependencies between joints in both the spatial and temporal dimensions. For the skeleton-to-video translation, we propose a novel self-supervised regularization network to translate the generated skeletons, along with a conditional image, into a dance video. Lastly, we collect a new skeleton-to-video translation dataset from the Internet, containing 54,944 video clips. Extensive experiments demonstrate that STG-Mamba achieves significantly…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Human Motion and Animation · Music Technology and Sound Studies
