Enriching Music Descriptions with a Finetuned-LLM and Metadata for Text-to-Music Retrieval

TL;DR

This paper introduces TTMR++, a novel text-to-music retrieval model that leverages finetuned large language models and metadata to improve music search based on descriptive and similarity-based queries.

Contribution

The paper presents TTMR++, which enhances text-to-music retrieval by integrating rich text generated with a finetuned LLM and metadata, addressing limitations of previous joint embedding approaches.

Findings

TTMR++ outperforms state-of-the-art models in retrieval accuracy.

Rich text descriptions improve the relevance of retrieved music.

Incorporating metadata enhances the model's ability to handle similarity queries.

Abstract

Text-to-Music Retrieval, finding music based on a given natural language query, plays a pivotal role in content discovery within extensive music databases. To address this challenge, prior research has predominantly focused on a joint embedding of music audio and text, utilizing it to retrieve music tracks that exactly match descriptive queries related to musical attributes (i.e. genre, instrument) and contextual elements (i.e. mood, theme). However, users also articulate a need to explore music that shares similarities with their favorite tracks or artists, such as \textit{I need a similar track to Superstition by Stevie Wonder}. To address these concerns, this paper proposes an improved Text-to-Music Retrieval model, denoted as TTMR++, which utilizes rich text descriptions generated with a finetuned large language model and metadata. To accomplish this, we obtained various types of seed text from several existing music tag and caption datasets and a knowledge graph dataset of artists and tracks. The experimental results show the effectiveness of TTMR++ in comparison to state-of-the-art music-text joint embedding models through a comprehensive evaluation involving various musical text queries.