LyricJam: A system for generating lyrics for live instrumental music
Olga Vechtomova, Gaurav Sahu, Dhruv Kumar

TL;DR
LyricJam is a real-time system that generates contextually matching lyrics for live instrumental music using novel alignment techniques, aiding artists in improvisation and lyric composition.
Contribution
Introduces two innovative methods for aligning audio and text latent spaces to generate lyrics that match live music in real-time.
Findings
System effectively generates contextually relevant lyrics in real-time.
Artists found the system useful for improvisation and lyric creation.
Users preferred the generated lyrics from the proposed methods over baseline models.
Abstract
We describe a real-time system that receives a live audio stream from a jam session and generates lyric lines that are congruent with the live music being played. Two novel approaches are proposed to align the learned latent spaces of audio and text representations that allow the system to generate novel lyric lines matching live instrumental music. One approach is based on adversarial alignment of latent representations of audio and lyrics, while the other approach learns to transfer the topology from the music latent space to the lyric latent space. A user study with music artists using the system showed that the system was useful not only in lyric composition, but also encouraged the artists to improvise and find new musical expressions. Another user study demonstrated that users preferred the lines generated using the proposed methods to the lines generated by a baseline model.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMusic and Audio Processing · Music Technology and Sound Studies · Video Analysis and Summarization
