SimSAM: Simple Siamese Representations Based Semantic Affinity Matrix for Unsupervised Image Segmentation

TL;DR

SimSAM introduces a simple, non-contrastive framework leveraging Siamese representations and a Semantic Affinity Matrix for improved unsupervised image segmentation, utilizing pre-trained DINO-ViT features.

Contribution

It presents a novel non-contrastive SSL framework, SimSAM, for computing the Semantic Affinity Matrix to enhance unsupervised image segmentation.

Findings

Effective in object segmentation tasks

Improves semantic segmentation performance

Uses pre-trained DINO-ViT features

Abstract

Recent developments in self-supervised learning (SSL) have made it possible to learn data representations without the need for annotations. Inspired by the non-contrastive SSL approach (SimSiam), we introduce a novel framework SIMSAM to compute the Semantic Affinity Matrix, which is significant for unsupervised image segmentation. Given an image, SIMSAM first extracts features using pre-trained DINO-ViT, then projects the features to predict the correlations of dense features in a non-contrastive way. We show applications of the Semantic Affinity Matrix in object segmentation and semantic segmentation tasks. Our code is available at https://github.com/chandagrover/SimSAM.