Can Synthetic Images Serve as Effective and Efficient Class Prototypes?

TL;DR

This paper introduces LGCLIP, a lightweight framework that uses large language models to generate synthetic images as class prototypes, enabling efficient zero-shot image classification without annotated image-text pairs.

Contribution

LGCLIP leverages LLMs and diffusion models to generate visual prototypes from class labels, reducing dependency on costly annotated datasets and simplifying model architecture.

Findings

LGCLIP achieves competitive zero-shot classification performance.

The framework eliminates the need for annotated image-text pairs.

LGCLIP is more lightweight and efficient than traditional methods.

Abstract

Vision-Language Models (VLMs) have shown strong performance in zero-shot image classification tasks. However, existing methods, including Contrastive Language-Image Pre-training (CLIP), all rely on annotated text-to-image pairs for aligning visual and textual modalities. This dependency introduces substantial cost and accuracy requirement in preparing high-quality datasets. At the same time, processing data from two modes also requires dual-tower encoders for most models, which also hinders their lightweight. To address these limitations, we introduce a ``Contrastive Language-Image Pre-training via Large-Language-Model-based Generation (LGCLIP)" framework. LGCLIP leverages a Large Language Model (LLM) to generate class-specific prompts that guide a diffusion model in synthesizing reference images. Afterwards these generated images serve as visual prototypes, and the visual features of real images are extracted and compared with the visual features of these prototypes to achieve comparative prediction. By optimizing prompt generation through the LLM and employing only a visual encoder, LGCLIP remains lightweight and efficient. Crucially, our framework requires only class labels as input during whole experimental procedure, eliminating the need for manually annotated image-text pairs and extra pre-processing. Experimental results validate the feasibility and efficiency of LGCLIP, demonstrating great performance in zero-shot classification tasks and establishing a novel paradigm for classification.