From Open-Vocabulary to Vocabulary-Free Semantic Segmentation

TL;DR

This paper introduces a vocabulary-free semantic segmentation method that automatically recognizes and labels objects without predefined class vocabularies, leveraging vision-language models to improve flexibility and applicability in real-world scenarios.

Contribution

It proposes a novel pipeline that eliminates the need for manual class vocabularies in semantic segmentation by using vision-language models for automatic object recognition and naming.

Findings

Significant improvement in vocabulary-free segmentation accuracy.

The text encoder's role is crucial, especially with generated descriptions.

Sensitivity to false negatives affects segmentation performance.

Abstract

Open-vocabulary semantic segmentation enables models to identify novel object categories beyond their training data. While this flexibility represents a significant advancement, current approaches still rely on manually specified class names as input, creating an inherent bottleneck in real-world applications. This work proposes a Vocabulary-Free Semantic Segmentation pipeline, eliminating the need for predefined class vocabularies. Specifically, we address the chicken-and-egg problem where users need knowledge of all potential objects within a scene to identify them, yet the purpose of segmentation is often to discover these objects. The proposed approach leverages Vision-Language Models to automatically recognize objects and generate appropriate class names, aiming to solve the challenge of class specification and naming quality. Through extensive experiments on several public datasets, we highlight the crucial role of the text encoder in model performance, particularly when the image text classes are paired with generated descriptions. Despite the challenges introduced by the sensitivity of the segmentation text encoder to false negatives within the class tagging process, which adds complexity to the task, we demonstrate that our fully automated pipeline significantly enhances vocabulary-free segmentation accuracy across diverse real-world scenarios.