MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response

TL;DR

MusiLingo is a system that connects music audio and text using pre-trained language models, enabling effective music captioning and query responses by aligning music representations with textual contexts.

Contribution

It introduces MusiLingo, a novel approach that aligns music audio representations with language models, and creates the MusicInstruct dataset for music-related question answering.

Findings

Competitive performance in music captioning

Effective music-related query response generation

Creation of the MusicInstruct dataset

Abstract

Large Language Models (LLMs) have shown immense potential in multimodal applications, yet the convergence of textual and musical domains remains not well-explored. To address this gap, we present MusiLingo, a novel system for music caption generation and music-related query responses. MusiLingo employs a single projection layer to align music representations from the pre-trained frozen music audio model MERT with a frozen LLM, bridging the gap between music audio and textual contexts. We train it on an extensive music caption dataset and fine-tune it with instructional data. Due to the scarcity of high-quality music Q&A datasets, we created the MusicInstruct (MI) dataset from captions in the MusicCaps datasets, tailored for open-ended music inquiries. Empirical evaluations demonstrate its competitive performance in generating music captions and composing music-related Q&A pairs. Our introduced dataset enables notable advancements beyond previous ones.