REFFLY: Melody-Constrained Lyrics Editing Model

TL;DR

REFFLY is a novel lyric revision framework that transforms plain text into melody-aligned lyrics, enabling flexible applications like translation and style transfer, and outperforms existing models in musicality and quality.

Contribution

This paper introduces REFFLY, the first revision framework for melody-aligned lyrics, trained on a curated dataset and enhanced with training-free heuristics for semantic and musical consistency.

Findings

REFFLY outperforms Lyra and GPT-4 by 25% in musicality.

Effective in tasks like lyrics generation and song translation.

Demonstrates improved semantic preservation and musical alignment.

Abstract

Automatic melody-to-lyric (M2L) generation aims to create lyrics that align with a given melody. While most previous approaches generate lyrics from scratch, revision, editing plain text draft to fit it into the melody, offers a much more flexible and practical alternative. This enables broad applications, such as generating lyrics from flexible inputs (keywords, themes, or full text that needs refining to be singable), song translation (preserving meaning across languages while keeping the melody intact), or style transfer (adapting lyrics to different genres). This paper introduces REFFLY (REvision Framework For LYrics), the first revision framework for editing and generating melody-aligned lyrics. We train the lyric revision module using our curated synthesized melody-aligned lyrics dataset, enabling it to transform plain text into lyrics that align with a given melody. To further enhance the revision ability, we propose training-free heuristics aimed at preserving both semantic meaning and musical consistency throughout the editing process. Experimental results demonstrate the effectiveness of REFFLY across various tasks (e.g. lyrics generation, song translation), showing that our model outperforms strong baselines, including Lyra (Tian et al., 2023) and GPT-4, by 25% in both musicality and text quality.