Segment Anything without Supervision

TL;DR

This paper introduces Unsupervised SAM (UnSAM), a novel method for automatic image segmentation without human labels, leveraging hierarchical clustering to produce pseudo masks that rival supervised models in performance.

Contribution

UnSAM is the first unsupervised approach to promptable segmentation that achieves competitive results and enhances semi-supervised learning by generating effective pseudo masks.

Findings

UnSAM surpasses previous unsupervised segmentation state-of-the-art by 11% in AR.

UnSAM achieves competitive performance with supervised SAM across seven datasets.

Semi-supervised training with UnSAM's pseudo masks improves segmentation metrics over fully supervised SAM.

Abstract

The Segmentation Anything Model (SAM) requires labor-intensive data labeling. We present Unsupervised SAM (UnSAM) for promptable and automatic whole-image segmentation that does not require human annotations. UnSAM utilizes a divide-and-conquer strategy to "discover" the hierarchical structure of visual scenes. We first leverage top-down clustering methods to partition an unlabeled image into instance/semantic level segments. For all pixels within a segment, a bottom-up clustering method is employed to iteratively merge them into larger groups, thereby forming a hierarchical structure. These unsupervised multi-granular masks are then utilized to supervise model training. Evaluated across seven popular datasets, UnSAM achieves competitive results with the supervised counterpart SAM, and surpasses the previous state-of-the-art in unsupervised segmentation by 11% in terms of AR. Moreover, we show that supervised SAM can also benefit from our self-supervised labels. By integrating our unsupervised pseudo masks into SA-1B's ground-truth masks and training UnSAM with only 1% of SA-1B, a lightly semi-supervised UnSAM can often segment entities overlooked by supervised SAM, exceeding SAM's AR by over 6.7% and AP by 3.9% on SA-1B.