Improved Few-Shot Image Classification Through Multiple-Choice Questions

TL;DR

This paper introduces a training-free few-shot image classification method that leverages multiple-choice questions with VQA models to extract and combine visual features, outperforming traditional encoders and zero-shot baselines.

Contribution

The proposed approach uses multiple-choice prompts to create prompt-specific latent representations, enhancing few-shot classification without additional training.

Findings

Outperforms pure visual encoders and zero-shot VQA baselines on standard datasets.

Effective in diverse attribute settings like fabric, style, and texture.

Maintains flexibility and dynamic advantages of VQA models.

Abstract

Through a simple multiple choice language prompt a VQA model can operate as a zero-shot image classifier, producing a classification label. Compared to typical image encoders, VQA models offer an advantage: VQA-produced image embeddings can be infused with the most relevant visual information through tailored language prompts. Nevertheless, for most tasks, zero-shot VQA performance is lacking, either because of unfamiliar category names, or dissimilar pre-training data and test data distributions. We propose a simple method to boost VQA performance for image classification using only a handful of labeled examples and a multiple-choice question. This few-shot method is training-free and maintains the dynamic and flexible advantages of the VQA model. Rather than relying on the final language output, our approach uses multiple-choice questions to extract prompt-specific latent representations, which are enriched with relevant visual information. These representations are combined to create a final overall image embedding, which is decoded via reference to latent class prototypes constructed from the few labeled examples. We demonstrate this method outperforms both pure visual encoders and zero-shot VQA baselines to achieve impressive performance on common few-shot tasks including MiniImageNet, Caltech-UCSD Birds, and CIFAR-100. Finally, we show our approach does particularly well in settings with numerous diverse visual attributes such as the fabric, article-style, texture, and view of different articles of clothing, where other few-shot approaches struggle, as we can tailor our image representations only on the semantic features of interest.