Vocabulary-free Image Classification and Semantic Segmentation

TL;DR

This paper introduces Vocabulary-free Image Classification (VIC) and Semantic Segmentation, enabling image understanding without predefined categories by leveraging external databases and pre-trained vision-language models, thus addressing the limitations of fixed vocabularies.

Contribution

The paper proposes CaSED, a training-free method for vocabulary-free classification and segmentation using external databases and vision-language models, advancing flexible image understanding.

Findings

CaSED outperforms complex models on classification benchmarks.

CaSED effectively generates coarse segmentation masks.

The approach requires fewer parameters than existing models.

Abstract

Large vision-language models revolutionized image classification and semantic segmentation paradigms. However, they typically assume a pre-defined set of categories, or vocabulary, at test time for composing textual prompts. This assumption is impractical in scenarios with unknown or evolving semantic context. Here, we address this issue and introduce the Vocabulary-free Image Classification (VIC) task, which aims to assign a class from an unconstrained language-induced semantic space to an input image without needing a known vocabulary. VIC is challenging due to the vastness of the semantic space, which contains millions of concepts, including fine-grained categories. To address VIC, we propose Category Search from External Databases (CaSED), a training-free method that leverages a pre-trained vision-language model and an external database. CaSED first extracts the set of candidate categories from the most semantically similar captions in the database and then assigns the image to the best-matching candidate category according to the same vision-language model. Furthermore, we demonstrate that CaSED can be applied locally to generate a coarse segmentation mask that classifies image regions, introducing the task of Vocabulary-free Semantic Segmentation. CaSED and its variants outperform other more complex vision-language models, on classification and semantic segmentation benchmarks, while using much fewer parameters.