Vocabulary-free few-shot learning for Vision-Language Models

TL;DR

This paper introduces a vocabulary-free few-shot learning method for Vision-Language Models that classifies images based on similarity to generic prompts, removing the need for class-specific labels or prompts.

Contribution

It proposes the Similarity Mapping (SiM) baseline, enabling efficient, interpretable classification without predefined class names, expanding few-shot learning applicability.

Findings

SiM achieves strong performance on benchmark tasks.

The method operates with high computational efficiency, often under one second.

Provides interpretability by linking classes to generic prompts.

Abstract

Recent advances in few-shot adaptation for Vision-Language Models (VLMs) have greatly expanded their ability to generalize across tasks using only a few labeled examples. However, existing approaches primarily build upon the strong zero-shot priors of these models by leveraging carefully designed, task-specific prompts. This dependence on predefined class names can restrict their applicability, especially in scenarios where exact class names are unavailable or difficult to specify. To address this limitation, we introduce vocabulary-free few-shot learning for VLMs, a setting where target class instances - that is, images - are available but their corresponding names are not. We propose Similarity Mapping (SiM), a simple yet effective baseline that classifies target instances solely based on similarity scores with a set of generic prompts (textual or visual), eliminating the need for carefully handcrafted prompts. Although conceptually straightforward, SiM demonstrates strong performance, operates with high computational efficiency (learning the mapping typically takes less than one second), and provides interpretability by linking target classes to generic prompts. We believe that our approach could serve as an important baseline for future research in vocabulary-free few-shot learning. Code is available at https://github.com/MaxZanella/vocabulary-free-FSL.