Exploiting LMM-based knowledge for image classification tasks

TL;DR

This paper enhances image classification by leveraging knowledge from Large Multimodal Models, combining image and text embeddings from MiniGPT-4 to improve accuracy across multiple datasets.

Contribution

It introduces a novel approach of using both image and text embeddings from MiniGPT-4 for image classification, surpassing traditional vision-language model methods.

Findings

Improved classification accuracy on three datasets.

Using combined image and text embeddings outperforms image-only methods.

Validates the effectiveness of LMM-based knowledge in image tasks.

Abstract

In this paper we address image classification tasks leveraging knowledge encoded in Large Multimodal Models (LMMs). More specifically, we use the MiniGPT-4 model to extract semantic descriptions for the images, in a multimodal prompting fashion. In the current literature, vision language models such as CLIP, among other approaches, are utilized as feature extractors, using only the image encoder, for solving image classification tasks. In this paper, we propose to additionally use the text encoder to obtain the text embeddings corresponding to the MiniGPT-4-generated semantic descriptions. Thus, we use both the image and text embeddings for solving the image classification task. The experimental evaluation on three datasets validates the improved classification performance achieved by exploiting LMM-based knowledge.