GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Models

TL;DR

GD-Retriever introduces a diffusion-based framework that enhances controllability and performance in text-music retrieval by generating and manipulating queries within a latent space, enabling interactive and flexible retrieval results.

Contribution

The paper presents a novel diffusion model-based retrieval framework that improves performance and offers controllability and interactivity in text-music retrieval tasks.

Findings

Outperforms contrastive teacher models in retrieval accuracy

Supports retrieval in audio-only latent spaces with non-joint encoders

Enables post-hoc manipulation of retrieval behavior

Abstract

Multimodal contrastive models have achieved strong performance in text-audio retrieval and zero-shot settings, but improving joint embedding spaces remains an active research area. Less attention has been given to making these systems controllable and interactive for users. In text-music retrieval, the ambiguity of freeform language creates a many-to-many mapping, often resulting in inflexible or unsatisfying results. We introduce Generative Diffusion Retriever (GDR), a novel framework that leverages diffusion models to generate queries in a retrieval-optimized latent space. This enables controllability through generative tools such as negative prompting and denoising diffusion implicit models (DDIM) inversion, opening a new direction in retrieval control. GDR improves retrieval performance over contrastive teacher models and supports retrieval in audio-only latent spaces using non-jointly trained encoders. Finally, we demonstrate that GDR enables effective post-hoc manipulation of retrieval behavior, enhancing interactive control for text-music retrieval tasks.