PictSure: Pretraining Embeddings Matters for In-Context Learning Image Classifiers

TL;DR

PictSure demonstrates that the choice and training of image embeddings critically influence the effectiveness of in-context learning for few-shot image classification, especially in out-of-domain scenarios.

Contribution

This work systematically analyzes the impact of embedding model pretraining and architecture on in-context learning performance in image classification.

Findings

Embedding pretraining significantly affects out-of-domain performance.

Different visual encoders lead to varied ICL effectiveness.

PictSure outperforms existing models on out-of-domain benchmarks.

Abstract

Building image classification models remains cumbersome in data-scarce domains, where collecting large labeled datasets is impractical. In-context learning (ICL) has emerged as a promising paradigm for few-shot image classification (FSIC), enabling models to generalize across domains without gradient-based adaptation. However, prior work has largely overlooked a critical component of ICL-based FSIC pipelines: the role of image embeddings. In this work, we present PictSure, an ICL framework that places the embedding model -- its architecture, pretraining, and training dynamics -- at the center of analysis. We systematically examine the effects of different visual encoder types, pretraining objectives, and fine-tuning strategies on downstream FSIC performance. Our experiments show that the training success and the out-of-domain performance are highly dependent on how the embedding models are pretrained. Consequently, PictSure manages to outperform existing ICL-based FSIC models on out-of-domain benchmarks that differ significantly from the training distribution, while maintaining comparable results on in-domain tasks. Code can be found at https://github.com/PictSure/pictsure-library.