Auto-Vocabulary Semantic Segmentation

TL;DR

Auto-Vocabulary Semantic Segmentation (AVS) autonomously identifies and segments relevant object classes in images without predefined categories, advancing open-ended image understanding and outperforming previous methods on multiple datasets.

Contribution

Introduces AVS, a framework that automatically generates object categories and segments them, removing the need for predefined vocabularies in semantic segmentation.

Findings

Sets new benchmarks on PASCAL VOC, Context, ADE20K, and Cityscapes datasets.

Achieves competitive performance with methods requiring predefined class names.

Develops LAVE for evaluating automatically generated classes and segments.

Abstract

Open-Vocabulary Segmentation (OVS) methods are capable of performing semantic segmentation without relying on a fixed vocabulary, and in some cases, without training or fine-tuning. However, OVS methods typically require a human in the loop to specify the vocabulary based on the task or dataset at hand. In this paper, we introduce Auto-Vocabulary Semantic Segmentation (AVS), advancing open-ended image understanding by eliminating the necessity to predefine object categories for segmentation. Our approach, AutoSeg, presents a framework that autonomously identifies relevant class names using semantically enhanced BLIP embeddings and segments them afterwards. Given that open-ended object category predictions cannot be directly compared with a fixed ground truth, we develop a Large Language Model-based Auto-Vocabulary Evaluator (LAVE) to efficiently evaluate the automatically generated classes and their corresponding segments. With AVS, our method sets new benchmarks on datasets PASCAL VOC, Context, ADE20K, and Cityscapes, while showing competitive performance to OVS methods that require specified class names.