Understanding Self-Supervised Features for Learning Unsupervised Instance Segmentation

TL;DR

This paper investigates how self-supervised learning features can be used for unsupervised instance segmentation, revealing differences in instance-awareness among various SSL methods.

Contribution

It provides a comparative analysis of SSL features for instance segmentation, highlighting the varying levels of instance-awareness in different methods.

Findings

DINO features are strong semantic descriptors but less sensitive to separating instances.

MAE features show higher sensitivity for instance separation.

SSL features vary significantly in their suitability for instance segmentation.

Abstract

Self-supervised learning (SSL) can be used to solve complex visual tasks without human labels. Self-supervised representations encode useful semantic information about images, and as a result, they have already been used for tasks such as unsupervised semantic segmentation. In this paper, we investigate self-supervised representations for instance segmentation without any manual annotations. We find that the features of different SSL methods vary in their level of instance-awareness. In particular, DINO features, which are known to be excellent semantic descriptors, lack behind MAE features in their sensitivity for separating instances.