A Little More Like This: Text-to-Image Retrieval with Vision-Language Models Using Relevance Feedback

TL;DR

This paper introduces relevance feedback mechanisms for vision-language models to improve text-to-image retrieval performance without fine-tuning, demonstrating consistent gains and robustness across different models and settings.

Contribution

It proposes four relevance feedback strategies, including generative and attentive feedback, to enhance retrieval accuracy and robustness in vision-language models at inference time.

Findings

Relevance feedback improves retrieval by 3-5% in MRR@5 for smaller VLMs.

The methods are more robust than traditional pseudo-relevance feedback in multi-turn retrieval.

Relevance feedback enables interactive, adaptive visual search without additional fine-tuning.

Abstract

Large vision-language models (VLMs) enable intuitive visual search using natural language queries. However, improving their performance often requires fine-tuning and scaling to larger model variants. In this work, we propose a mechanism inspired by traditional text-based search to improve retrieval performance at inference time: relevance feedback. While relevance feedback can serve as an alternative to fine-tuning, its model-agnostic design also enables use with fine-tuned VLMs. Specifically, we introduce and evaluate four feedback strategies for VLM-based retrieval. First, we revise classical pseudo-relevance feedback (PRF), which refines query embeddings based on top-ranked results. To address its limitations, we propose generative relevance feedback (GRF), which uses synthetic captions for query refinement. Furthermore, we introduce an attentive feedback summarizer (AFS), a custom transformer-based model that integrates multimodal fine-grained features from relevant items. Finally, we simulate explicit feedback using ground-truth captions as an upper-bound baseline. Experiments on Flickr30k and COCO with the VLM backbones show that GRF, AFS, and explicit feedback improve retrieval performance by 3-5% in MRR@5 for smaller VLMs, and 1-3% for larger ones, compared to retrieval with no feedback. Moreover, AFS, similarly to explicit feedback, mitigates query drift and is more robust than GRF in iterative, multi-turn retrieval settings. Our findings demonstrate that relevance feedback can consistently enhance retrieval across VLMs and open up opportunities for interactive and adaptive visual search.