Language-driven Fine-grained Retrieval

TL;DR

This paper introduces LaFG, a novel framework that uses large language models and vision-language models to generate attribute-level supervision from class names, enhancing fine-grained image retrieval and generalization to unseen categories.

Contribution

LaFG leverages LLMs and VLMs to create rich, attribute-based supervision from class names, improving fine-grained retrieval beyond traditional one-hot label methods.

Findings

Outperforms existing FGIR methods on benchmark datasets.

Enhances generalization to unseen categories.

Effectively captures detailed semantic attributes.

Abstract

Existing fine-grained image retrieval (FGIR) methods learn discriminative embeddings by adopting semantically sparse one-hot labels derived from category names as supervision. While effective on seen classes, such supervision overlooks the rich semantics encoded in category names, hindering the modeling of comparability among cross-category details and, in turn, limiting generalization to unseen categories. To tackle this, we introduce LaFG, a Language-driven framework for Fine-Grained Retrieval that converts class names into attribute-level supervision using large language models (LLMs) and vision-language models (VLMs). Treating each name as a semantic anchor, LaFG prompts an LLM to generate detailed, attribute-oriented descriptions. To mitigate attribute omission in these descriptions, it leverages a frozen VLM to project them into a vision-aligned space, clustering them into a dataset-wide attribute vocabulary while harvesting complementary attributes from related categories. Leveraging this vocabulary, a global prompt template selects category-relevant attributes, which are aggregated into category-specific linguistic prototypes. These prototypes supervise the retrieval model to steer