GPT4Image: Large Pre-trained Models Help Vision Models Learn Better on Perception Task

TL;DR

This paper introduces GPT4Image, a framework that leverages large pre-trained multimodal models to enhance the representation learning of traditional vision models like CNNs and ViTs on perception tasks, improving their performance.

Contribution

The paper proposes a novel method to incorporate rich semantic knowledge from pre-trained large models into vision models via text embeddings, boosting their perception capabilities.

Findings

Enhanced image classification accuracy across multiple architectures.

Effective use of text embeddings as additional supervision signals.

Improved generalization on various visual perception benchmarks.

Abstract

The upsurge in pre-trained large models started by ChatGPT has swept across the entire deep learning community. Such powerful models demonstrate advanced generative ability and multimodal understanding capability, which quickly set new state of the arts on a variety of benchmarks. The pre-trained LLM usually plays the role as a universal AI model that can conduct various tasks like article analysis and image comprehension. However, due to the prohibitively high memory and computational cost of implementing such a large model, the conventional models (such as CNN and ViT) are still essential for many visual perception tasks. In this paper, we propose to enhance the representation ability of ordinary vision models on perception tasks (e.g. image classification) by taking advantage of the off-the-shelf large pre-trained models. We present a new learning framework, dubbed GPT4Image, where the knowledge of the large pre-trained models are extracted to help CNNs and ViTs learn better representations and achieve higher performance. Firstly, we curate a high quality description set by prompting a multimodal LLM to generate descriptions for training images. Then, these detailed descriptions are fed into a pre-trained encoder to extract text embeddings that encodes the rich semantics of images. During training, text embeddings will serve as extra supervising signal and be aligned with image representations learned by vision models. The alignment process helps vision models achieve better performance with the aid of pre-trained LLMs. We conduct extensive experiments to verify the effectiveness of the proposed algorithm on various visual perception tasks for heterogeneous model architectures.