Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models

TL;DR

This paper presents CascadeVLM, a new framework that enhances fine-grained image classification by effectively leveraging large vision-language models, significantly improving zero-shot accuracy over existing CLIP-based methods.

Contribution

Introduces CascadeVLM, a novel approach that overcomes CLIP limitations by utilizing granular knowledge from LVLMs for better fine-grained classification.

Findings

Achieves 85.6% zero-shot accuracy on Stanford Cars dataset.

LVLMs provide more accurate predictions for challenging images.

CascadeVLM outperforms existing models in fine-grained classification.

Abstract

Fine-grained image classification, particularly in zero/few-shot scenarios, presents a significant challenge for vision-language models (VLMs), such as CLIP. These models often struggle with the nuanced task of distinguishing between semantically similar classes due to limitations in their pre-trained recipe, which lacks supervision signals for fine-grained categorization. This paper introduces CascadeVLM, an innovative framework that overcomes the constraints of previous CLIP-based methods by effectively leveraging the granular knowledge encapsulated within large vision-language models (LVLMs). Experiments across various fine-grained image datasets demonstrate that CascadeVLM significantly outperforms existing models, specifically on the Stanford Cars dataset, achieving an impressive 85.6% zero-shot accuracy. Performance gain analysis validates that LVLMs produce more accurate predictions for challenging images that CLIPs are uncertain about, bringing the overall accuracy boost. Our framework sheds light on a holistic integration of VLMs and LVLMs for effective and efficient fine-grained image classification.