VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval

TL;DR

VisRet introduces a novel retrieval paradigm that enhances text-to-image retrieval by projecting textual queries into the image domain via T2I generation, significantly improving performance across multiple benchmarks.

Contribution

The paper presents VisRet, a new approach that mitigates cross-modal embedding limitations by transforming text queries into the image modality for more accurate retrieval.

Findings

Outperforms existing methods on four benchmarks with significant nDCG@30 improvements.

Increases downstream question answering accuracy by up to 15.7%.

Demonstrates compatibility with various T2I models and LLMs.

Abstract

Text-to-image retrieval (T2I retrieval) remains challenging because cross-modal embeddings often behave as bags of concepts, underrepresenting structured visual relationships such as pose and viewpoint. We proposeVisualize-then-Retrieve (VisRet), a retrieval paradigm that mitigates this limitation of cross-modal similarity alignment. VisRet first projects textual queries into the image modality via T2I generation, then performs retrieval within the image modality to bypass the weaknesses of cross-modal retrievers in recognizing subtle visual-spatial features. Across four benchmarks (Visual-RAG, INQUIRE-Rerank, Microsoft COCO, and our new Visual-RAG-ME featuring multi-entity comparisons), VisRet substantially outperforms cross-modal similarity matching and baselines that recast T2I retrieval as text-to-text similarity matching, improving nDCG@30 by 0.125 on average with CLIP as the retriever and by 0.121 with E5-V. For downstream question answering, VisRet increases accuracy on Visual-RAG and Visual-RAG-ME by 3.8% and 15.7% in top-1 retrieval, and by 3.9% and 11.1% in top-10 retrieval. Ablation studies show compatibility with different T2I instruction LLMs, T2I generation models, and downstream LLMs. VisRet provides a simple yet effective perspective for advancing in text-image retrieval. Our code and the new benchmark are publicly available at https://github.com/xiaowu0162/Visualize-then-Retrieve.