EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models

TL;DR

EvdCLIP enhances vision-language retrieval by integrating entity visual descriptions generated by large language models into queries, and employs a trainable rewriter to improve retrieval accuracy.

Contribution

The paper introduces EVDs from LLMs and a trainable EVD-aware rewriter to improve retrieval performance, addressing noise issues in query expansion.

Findings

EvdCLIP outperforms existing methods on benchmark datasets.

EVD integration significantly improves retrieval accuracy.

The trainable rewriter effectively reduces noise in queries.

Abstract

Vision-language retrieval (VLR) has attracted significant attention in both academia and industry, which involves using text (or images) as queries to retrieve corresponding images (or text). However, existing methods often neglect the rich visual semantics knowledge of entities, thus leading to incorrect retrieval results. To address this problem, we propose the Entity Visual Description enhanced CLIP (EvdCLIP), designed to leverage the visual knowledge of entities to enrich queries. Specifically, since humans recognize entities through visual cues, we employ a large language model (LLM) to generate Entity Visual Descriptions (EVDs) as alignment cues to complement textual data. These EVDs are then integrated into raw queries to create visually-rich, EVD-enhanced queries. Furthermore, recognizing that EVD-enhanced queries may introduce noise or low-quality expansions, we develop a novel, trainable EVD-aware Rewriter (EaRW) for vision-language retrieval tasks. EaRW utilizes EVD knowledge and the generative capabilities of the language model to effectively rewrite queries. With our specialized training strategy, EaRW can generate high-quality and low-noise EVD-enhanced queries. Extensive quantitative and qualitative experiments on image-text retrieval benchmarks validate the superiority of EvdCLIP on vision-language retrieval tasks.