Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMs

TL;DR

This paper introduces NeaR, a method that fine-tunes CLIP models for vocabulary-free fine-grained visual recognition by leveraging labels generated from multimodal large language models, enabling efficient recognition without prior labels.

Contribution

The paper proposes NeaR, a novel label refinement approach that constructs weakly supervised datasets from unlabeled data using MLLMs, addressing the challenge of vocabulary-free FGVR.

Findings

NeaR effectively handles noisy labels from MLLMs.

NeaR establishes new benchmarks for vocabulary-free FGVR.

The approach reduces reliance on large annotated datasets.

Abstract

Fine-grained Visual Recognition (FGVR) involves distinguishing between visually similar categories, which is inherently challenging due to subtle inter-class differences and the need for large, expert-annotated datasets. In domains like medical imaging, such curated datasets are unavailable due to issues like privacy concerns and high annotation costs. In such scenarios lacking labeled data, an FGVR model cannot rely on a predefined set of training labels, and hence has an unconstrained output space for predictions. We refer to this task as Vocabulary-Free FGVR (VF-FGVR), where a model must predict labels from an unconstrained output space without prior label information. While recent Multimodal Large Language Models (MLLMs) show potential for VF-FGVR, querying these models for each test input is impractical because of high costs and prohibitive inference times. To address these limitations, we introduce \textbf{Nea}rest-Neighbor Label \textbf{R}efinement (NeaR), a novel approach that fine-tunes a downstream CLIP model using labels generated by an MLLM. Our approach constructs a weakly supervised dataset from a small, unlabeled training set, leveraging MLLMs for label generation. NeaR is designed to handle the noise, stochasticity, and open-endedness inherent in labels generated by MLLMs, and establishes a new benchmark for efficient VF-FGVR.