LyricJam: A system for generating lyrics for live instrumental music

TL;DR

LyricJam is a real-time system that generates contextually matching lyrics for live instrumental music using novel alignment techniques, aiding artists in improvisation and lyric composition.

Contribution

Introduces two innovative methods for aligning audio and text latent spaces to generate lyrics that match live music in real-time.

Findings

System effectively generates contextually relevant lyrics in real-time.

Artists found the system useful for improvisation and lyric creation.

Users preferred the generated lyrics from the proposed methods over baseline models.

Abstract

We describe a real-time system that receives a live audio stream from a jam session and generates lyric lines that are congruent with the live music being played. Two novel approaches are proposed to align the learned latent spaces of audio and text representations that allow the system to generate novel lyric lines matching live instrumental music. One approach is based on adversarial alignment of latent representations of audio and lyrics, while the other approach learns to transfer the topology from the music latent space to the lyric latent space. A user study with music artists using the system showed that the system was useful not only in lyric composition, but also encouraged the artists to improvise and find new musical expressions. Another user study demonstrated that users preferred the lines generated using the proposed methods to the lines generated by a baseline model.