A Lightweight Clustering Framework for Unsupervised Semantic Segmentation

TL;DR

This paper introduces a lightweight clustering framework leveraging self-supervised Vision Transformer features for unsupervised semantic segmentation, achieving state-of-the-art results without neural network training.

Contribution

The authors propose a novel multilevel clustering approach that exploits attention features for effective unsupervised segmentation, reducing computational complexity.

Findings

Achieves state-of-the-art results on PASCAL VOC and MS COCO datasets.

Demonstrates strong foreground-background differentiation in self-supervised Vision Transformer features.

Provides comprehensive analysis comparing DINO and DINOv2 features.

Abstract

Unsupervised semantic segmentation aims to categorize each pixel in an image into a corresponding class without the use of annotated data. It is a widely researched area as obtaining labeled datasets is expensive. While previous works in the field have demonstrated a gradual improvement in model accuracy, most required neural network training. This made segmentation equally expensive, especially when dealing with large-scale datasets. We thus propose a lightweight clustering framework for unsupervised semantic segmentation. We discovered that attention features of the self-supervised Vision Transformer exhibit strong foreground-background differentiability. Therefore, clustering can be employed to effectively separate foreground and background image patches. In our framework, we first perform multilevel clustering across the Dataset-level, Category-level, and Image-level, and maintain consistency throughout. Then, the binary patch-level pseudo-masks extracted are upsampled, refined and finally labeled. Furthermore, we provide a comprehensive analysis of the self-supervised Vision Transformer features and a detailed comparison between DINO and DINOv2 to justify our claims. Our framework demonstrates great promise in unsupervised semantic segmentation and achieves state-of-the-art results on PASCAL VOC and MS COCO datasets.