Accompanied Singing Voice Synthesis with Fully Text-controlled Melody

TL;DR

MelodyLM is a novel text-controlled singing voice synthesis model that generates high-quality accompanied songs with minimal user input and maximum control, using a language model approach and latent diffusion for accompaniment.

Contribution

It introduces MelodyLM, the first TTSong system that synthesizes songs with fully text-controlled melodies and minimal input requirements, advancing flexibility and quality in singing voice synthesis.

Findings

Achieves superior objective and subjective performance metrics.

Allows full control through textual prompts or MIDI input.

Requires only lyrics and a reference voice for synthesis.

Abstract

Text-to-song (TTSong) is a music generation task that synthesizes accompanied singing voices. Current TTSong methods, inherited from singing voice synthesis (SVS), require melody-related information that can sometimes be impractical, such as music scores or MIDI sequences. We present MelodyLM, the first TTSong model that generates high-quality song pieces with fully text-controlled melodies, achieving minimal user requirements and maximum control flexibility. MelodyLM explicitly models MIDI as the intermediate melody-related feature and sequentially generates vocal tracks in a language model manner, conditioned on textual and vocal prompts. The accompaniment music is subsequently synthesized by a latent diffusion model with hybrid conditioning for temporal alignment. With minimal requirements, users only need to input lyrics and a reference voice to synthesize a song sample. For full control, just input textual prompts or even directly input MIDI. Experimental results indicate that MelodyLM achieves superior performance in terms of both objective and subjective metrics. Audio samples are available at https://melodylm666.github.io.