# Fine-graind Image Classification via Combining Vision and Language

## TL;DR

This paper introduces a two-stream model combining visual features and natural language descriptions to improve fine-grained image classification accuracy, addressing limitations of existing part detection methods.

## Contribution

The proposed CVL model effectively integrates vision and language streams, providing a flexible way to encode discriminative features and enhance classification performance.

## Key findings

- Achieves state-of-the-art accuracy on CUB-200-2011 dataset.
- Outperforms 12 existing fine-grained classification methods.
- Demonstrates the benefit of combining visual and language information.

## Abstract

Fine-grained image classification is a challenging task due to the large intra-class variance and small inter-class variance, aiming at recognizing hundreds of sub-categories belonging to the same basic-level category. Most existing fine-grained image classification methods generally learn part detection models to obtain the semantic parts for better classification accuracy. Despite achieving promising results, these methods mainly have two limitations: (1) not all the parts which obtained through the part detection models are beneficial and indispensable for classification, and (2) fine-grained image classification requires more detailed visual descriptions which could not be provided by the part locations or attribute annotations. For addressing the above two limitations, this paper proposes the two-stream model combining vision and language (CVL) for learning latent semantic representations. The vision stream learns deep representations from the original visual information via deep convolutional neural network. The language stream utilizes the natural language descriptions which could point out the discriminative parts or characteristics for each image, and provides a flexible and compact way of encoding the salient visual aspects for distinguishing sub-categories. Since the two streams are complementary, combining the two streams can further achieves better classification accuracy. Comparing with 12 state-of-the-art methods on the widely used CUB-200-2011 dataset for fine-grained image classification, the experimental results demonstrate our CVL approach achieves the best performance.