Training-Free Semantic Segmentation via LLM-Supervision

TL;DR

This paper presents a training-free approach to semantic segmentation that leverages large language models to generate detailed class subclasses, improving segmentation accuracy without additional training.

Contribution

It introduces a novel LLM-based supervision method that generates subclasses for better class representation in zero-shot semantic segmentation.

Findings

Outperforms traditional text-supervised segmentation methods on standard benchmarks.

Uses LLM-generated subclasses to enhance segmentation diversity and accuracy.

Merges multiple segmentation maps for comprehensive test image representation.

Abstract

Recent advancements in open vocabulary models, like CLIP, have notably advanced zero-shot classification and segmentation by utilizing natural language for class-specific embeddings. However, most research has focused on improving model accuracy through prompt engineering, prompt learning, or fine-tuning with limited labeled data, thereby overlooking the importance of refining the class descriptors. This paper introduces a new approach to text-supervised semantic segmentation using supervision by a large language model (LLM) that does not require extra training. Our method starts from an LLM, like GPT-3, to generate a detailed set of subclasses for more accurate class representation. We then employ an advanced text-supervised semantic segmentation model to apply the generated subclasses as target labels, resulting in diverse segmentation results tailored to each subclass's unique characteristics. Additionally, we propose an assembly that merges the segmentation maps from the various subclass descriptors to ensure a more comprehensive representation of the different aspects in the test images. Through comprehensive experiments on three standard benchmarks, our method outperforms traditional text-supervised semantic segmentation methods by a marked margin.