Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddings

TL;DR

LIVI is a scalable music cover retrieval method that uses lyrics-based audio embeddings, achieving high accuracy with reduced computational complexity by eliminating the need for transcription during inference.

Contribution

LIVI introduces a lyrics-informed approach that balances retrieval accuracy and efficiency, avoiding complex multimodal architectures and transcription steps at inference.

Findings

Achieves retrieval accuracy comparable or superior to harmonic-based systems.

Remains lightweight and efficient by removing transcription at inference.

Balances accuracy with computational efficiency.

Abstract

Music Cover Retrieval, also known as Version Identification, aims to recognize distinct renditions of the same underlying musical work, a task central to catalog management, copyright enforcement, and music retrieval. State-of-the-art approaches have largely focused on harmonic and melodic features, employing increasingly complex audio pipelines designed to be invariant to musical attributes that often vary widely across covers. While effective, these methods demand substantial training time and computational resources. By contrast, lyrics constitute a strong invariant across covers, though their use has been limited by the difficulty of extracting them accurately and efficiently from polyphonic audio. Early methods relied on simple frameworks that limited downstream performance, while more recent systems deliver stronger results but require large models integrated within complex multimodal architectures. We introduce LIVI (Lyrics-Informed Version Identification), an approach that seeks to balance retrieval accuracy with computational efficiency. First, LIVI leverages supervision from state-of-the-art transcription and text embedding models during training to achieve retrieval accuracy on par with--or superior to--harmonic-based systems. Second, LIVI remains lightweight and efficient by removing the transcription step at inference, challenging the dominance of complexity-heavy pipelines.