SongSage: A Large Musical Language Model with Lyric Generative Pre-training

TL;DR

SongSage is a large musical language model trained on lyric-focused data, demonstrating strong lyric understanding, query rewriting, and lyric generation capabilities, advancing music AI research.

Contribution

Introduces SongSage, a novel lyric-centric language model trained on LyricBank, with extensive fine-tuning for diverse lyric-related tasks, improving music AI applications.

Findings

Outperforms in lyric rewriting and generation tasks

Achieves strong lyric-centric knowledge understanding

Maintains general knowledge proficiency with competitive MMLU score

Abstract

Large language models have achieved significant success in various domains, yet their understanding of lyric-centric knowledge has not been fully explored. In this work, we first introduce PlaylistSense, a dataset to evaluate the playlist understanding capability of language models. PlaylistSense encompasses ten types of user queries derived from common real-world perspectives, challenging LLMs to accurately grasp playlist features and address diverse user intents. Comprehensive evaluations indicate that current general-purpose LLMs still have potential for improvement in playlist understanding. Inspired by this, we introduce SongSage, a large musical language model equipped with diverse lyric-centric intelligence through lyric generative pretraining. SongSage undergoes continual pretraining on LyricBank, a carefully curated corpus of 5.48 billion tokens focused on lyrical content, followed by fine-tuning with LyricBank-SFT, a meticulously crafted instruction set comprising 775k samples across nine core lyric-centric tasks. Experimental results demonstrate that SongSage exhibits a strong understanding of lyric-centric knowledge, excels in rewriting user queries for zero-shot playlist recommendations, generates and continues lyrics effectively, and performs proficiently across seven additional capabilities. Beyond its lyric-centric expertise, SongSage also retains general knowledge comprehension and achieves a competitive MMLU score. We will keep the datasets inaccessible due to copyright restrictions and release the SongSage and training script to ensure reproducibility and support music AI research and applications, the datasets release plan details are provided in the appendix.