ReLyMe: Improving Lyric-to-Melody Generation by Incorporating Lyric-Melody Relationships

TL;DR

ReLyMe enhances lyric-to-melody generation by integrating music theory principles into neural models, improving harmony and coherence between lyrics and melodies across multiple languages.

Contribution

This work introduces a novel approach that incorporates music theory-based relationships into neural lyric-to-melody models to improve musical harmony.

Findings

ReLyMe outperforms baseline models on objective metrics.

Subjective evaluations favor ReLyMe's musical harmony.

Effective across English and Chinese datasets.

Abstract

Lyric-to-melody generation, which generates melody according to given lyrics, is one of the most important automatic music composition tasks. With the rapid development of deep learning, previous works address this task with end-to-end neural network models. However, deep learning models cannot well capture the strict but subtle relationships between lyrics and melodies, which compromises the harmony between lyrics and generated melodies. In this paper, we propose ReLyMe, a method that incorporates Relationships between Lyrics and Melodies from music theory to ensure the harmony between lyrics and melodies. Specifically, we first introduce several principles that lyrics and melodies should follow in terms of tone, rhythm, and structure relationships. These principles are then integrated into neural network lyric-to-melody models by adding corresponding constraints during the decoding process to improve the harmony between lyrics and melodies. We use a series of objective and subjective metrics to evaluate the generated melodies. Experiments on both English and Chinese song datasets show the effectiveness of ReLyMe, demonstrating the superiority of incorporating lyric-melody relationships from the music domain into neural lyric-to-melody generation.