Interpretable Melody Generation from Lyrics with Discrete-Valued Adversarial Training

TL;DR

This paper presents an interpretable lyrics-to-melody generation system that enhances consistency and user interaction using mutual information, Gumbel-Softmax, and GANs to generate discrete music attributes.

Contribution

The proposed system introduces a novel approach combining mutual information and Gumbel-Softmax within GANs for more reliable and interpretable melody generation from lyrics.

Findings

Improved lyric-melody consistency through mutual information.

Effective discrete attribute generation using Gumbel-Softmax.

User interaction enables song recreation and attribute selection.

Abstract

Generating melody from lyrics is an interesting yet challenging task in the area of artificial intelligence and music. However, the difficulty of keeping the consistency between input lyrics and generated melody limits the generation quality of previous works. In our proposal, we demonstrate our proposed interpretable lyrics-to-melody generation system which can interact with users to understand the generation process and recreate the desired songs. To improve the reliability of melody generation that matches lyrics, mutual information is exploited to strengthen the consistency between lyrics and generated melodies. Gumbel-Softmax is exploited to solve the non-differentiability problem of generating discrete music attributes by Generative Adversarial Networks (GANs). Moreover, the predicted probabilities output by the generator is utilized to recommend music attributes. Interacting with our lyrics-to-melody generation system, users can listen to the generated AI song as well as recreate a new song by selecting from recommended music attributes.