Melody-Guided Music Generation

TL;DR

The paper introduces MG2, a melody-guided text-to-music generation model that outperforms existing models with fewer resources by aligning text, audio, and melody using contrastive pretraining and a retrieval-augmented diffusion process.

Contribution

It proposes a novel contrastive language-music pretraining method and a melody-guided diffusion model for efficient, high-quality text-to-music generation with limited data and parameters.

Findings

MG2 surpasses current open-source models in quality.

Achieves high performance with less than 1/3 parameters of competitors.

Human evaluations confirm practical effectiveness.

Abstract

We present the Melody-Guided Music Generation (MG2) model, a novel approach using melody to guide the text-to-music generation that, despite a simple method and limited resources, achieves excellent performance. Specifically, we first align the text with audio waveforms and their associated melodies using the newly proposed Contrastive Language-Music Pretraining, enabling the learned text representation fused with implicit melody information. Subsequently, we condition the retrieval-augmented diffusion module on both text prompt and retrieved melody. This allows MG2 to generate music that reflects the content of the given text description, meantime keeping the intrinsic harmony under the guidance of explicit melody information. We conducted extensive experiments on two public datasets: MusicCaps and MusicBench. Surprisingly, the experimental results demonstrate that the proposed MG2 model surpasses current open-source text-to-music generation models, achieving this with fewer than 1/3 of the parameters or less than 1/200 of the training data compared to state-of-the-art counterparts. Furthermore, we conducted comprehensive human evaluations involving three types of users and five perspectives, using newly designed questionnaires to explore the potential real-world applications of MG2.