Zero-Shot Pseudo Labels Generation Using SAM and CLIP for Semi-Supervised Semantic Segmentation

TL;DR

This paper introduces a novel semi-supervised semantic segmentation approach that leverages zero-shot pseudo labels generated by SAM and CLIP, enhanced with UniMatch, to reduce annotation costs and improve segmentation accuracy.

Contribution

It proposes a new method combining SAM, CLIP, and UniMatch for zero-shot pseudo label generation in semi-supervised segmentation, demonstrating effectiveness on public datasets.

Findings

Improved segmentation accuracy using zero-shot pseudo labels.

Effective on PASCAL and MS COCO datasets.

Reduces reliance on extensive labeled data.

Abstract

Semantic segmentation is a fundamental task in medical image analysis and autonomous driving and has a problem with the high cost of annotating the labels required in training. To address this problem, semantic segmentation methods based on semi-supervised learning with a small number of labeled data have been proposed. For example, one approach is to train a semantic segmentation model using images with annotated labels and pseudo labels. In this approach, the accuracy of the semantic segmentation model depends on the quality of the pseudo labels, and the quality of the pseudo labels depends on the performance of the model to be trained and the amount of data with annotated labels. In this paper, we generate pseudo labels using zero-shot annotation with the Segment Anything Model (SAM) and Contrastive Language-Image Pretraining (CLIP), improve the accuracy of the pseudo labels using the Unified Dual-Stream Perturbations Approach (UniMatch), and use them as enhanced labels to train a semantic segmentation model. The effectiveness of the proposed method is demonstrated through the experiments using the public datasets: PASCAL and MS COCO. The project web page is available at: https://gsisaoki.github.io/ZERO-SHOT-PLG/