Interpretable and Perceptually-Aligned Music Similarity with Pretrained Embeddings

TL;DR

This paper demonstrates that pretrained text-audio embeddings can effectively measure music similarity aligning with human perception, and introduces a method to make these embeddings interpretable and controllable for music retrieval tasks.

Contribution

The authors propose a novel approach to align pretrained embeddings with perceptual data, enabling interpretable instrument-wise weighting for music retrieval.

Findings

Pretrained embeddings achieve comparable perceptual alignment without fine-tuning.

The method allows for instrument-wise interpretability and control in music similarity retrieval.

Music producers can retrieve stem-level samples based on mixed references.

Abstract

Perceptual similarity representations enable music retrieval systems to determine which songs sound most similar to listeners. State-of-the-art approaches based on task-specific training via self-supervised metric learning show promising alignment with human judgment, but are difficult to interpret or generalize due to limited dataset availability. We show that pretrained text-audio embeddings (CLAP and MuQ-MuLan) offer comparable perceptual alignment on similarity tasks without any additional fine-tuning. To surpass this baseline, we introduce a novel method to perceptually align pretrained embeddings with source separation and linear optimization on ABX preference data from listening tests. Our model provides interpretable and controllable instrument-wise weights, allowing music producers to retrieve stem-level loops and samples based on mixed reference songs.