Diff4Steer: Steerable Diffusion Prior for Generative Music Retrieval with Semantic Guidance

TL;DR

Diff4Steer introduces a diffusion-based generative framework for music retrieval that models user preference uncertainty and allows flexible, steerable music discovery, outperforming traditional methods.

Contribution

It presents a novel diffusion prior approach for generative music retrieval that captures preference uncertainty and enables steerable, multimodal guidance.

Findings

Outperforms deterministic and LLM-based baselines in retrieval metrics.

Provides diverse and relevant music recommendations.

Enables flexible steering with image or text inputs.

Abstract

Modern music retrieval systems often rely on fixed representations of user preferences, limiting their ability to capture users' diverse and uncertain retrieval needs. To address this limitation, we introduce Diff4Steer, a novel generative retrieval framework that employs lightweight diffusion models to synthesize diverse seed embeddings from user queries that represent potential directions for music exploration. Unlike deterministic methods that map user query to a single point in embedding space, Diff4Steer provides a statistical prior on the target modality (audio) for retrieval, effectively capturing the uncertainty and multi-faceted nature of user preferences. Furthermore, Diff4Steer can be steered by image or text inputs, enabling more flexible and controllable music discovery combined with nearest neighbor search. Our framework outperforms deterministic regression methods and LLM-based generative retrieval baseline in terms of retrieval and ranking metrics, demonstrating its effectiveness in capturing user preferences, leading to more diverse and relevant recommendations. Listening examples are available at tinyurl.com/diff4steer.