Intelligent Text-Conditioned Music Generation

TL;DR

This paper introduces a novel approach for text-conditioned music generation by training a CLIP-like model to align music with text and then using this alignment to generate music from textual prompts.

Contribution

It is the first to apply contrastive learning for text-music alignment and integrate it with a music decoder for text-conditioned music synthesis.

Findings

Successful training of a text-music alignment model using contrastive loss

Demonstration of music generation from text prompts

First implementation of deep text-conditioned music generation

Abstract

CLIP (Contrastive Language-Image Pre-Training) is a multimodal neural network trained on (text, image) pairs to predict the most relevant text caption given an image. It has been used extensively in image generation by connecting its output with a generative model such as VQGAN, with the most notable example being OpenAI's DALLE-2. In this project, we apply a similar approach to bridge the gap between natural language and music. Our model is split into two steps: first, we train a CLIP-like model on pairs of text and music over contrastive loss to align a piece of music with its most probable text caption. Then, we combine the alignment model with a music decoder to generate music. To the best of our knowledge, this is the first attempt at text-conditioned deep music generation. Our experiments show that it is possible to train the text-music alignment model using contrastive loss and train a decoder to generate music from text prompts.