MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence

TL;DR

MoMu-Diffusion introduces a unified framework for long-term, synchronized motion-music generation using a novel auto-encoder and diffusion model, enabling diverse cross-modal and variable-length synthesis with improved realism.

Contribution

The paper presents a novel BiCoR-VAE for efficient modality-aligned representation learning and a multi-modal diffusion model for synchronized motion-music generation, addressing long-term sequence challenges.

Findings

Outperforms state-of-the-art methods in quality and diversity

Capable of long-term, beat-matched motion and music synthesis

Effective in cross-modal and multi-modal generation tasks

Abstract

Motion-to-music and music-to-motion have been studied separately, each attracting substantial research interest within their respective domains. The interaction between human motion and music is a reflection of advanced human intelligence, and establishing a unified relationship between them is particularly important. However, to date, there has been no work that considers them jointly to explore the modality alignment within. To bridge this gap, we propose a novel framework, termed MoMu-Diffusion, for long-term and synchronous motion-music generation. Firstly, to mitigate the huge computational costs raised by long sequences, we propose a novel Bidirectional Contrastive Rhythmic Variational Auto-Encoder (BiCoR-VAE) that extracts the modality-aligned latent representations for both motion and music inputs. Subsequently, leveraging the aligned latent spaces, we introduce a multi-modal Transformer-based diffusion model and a cross-guidance sampling strategy to enable various generation tasks, including cross-modal, multi-modal, and variable-length generation. Extensive experiments demonstrate that MoMu-Diffusion surpasses recent state-of-the-art methods both qualitatively and quantitatively, and can synthesize realistic, diverse, long-term, and beat-matched music or motion sequences. The generated samples and codes are available at https://momu-diffusion.github.io/