Leveraging Out-of-Distribution Unlabeled Images: Semi-Supervised Semantic Segmentation with an Open-Vocabulary Model

TL;DR

This paper introduces SemiOVS, a semi-supervised semantic segmentation framework that leverages out-of-distribution unlabeled images using an open-vocabulary model, significantly improving performance especially with limited labels.

Contribution

The paper proposes a novel semi-supervised segmentation method utilizing open-vocabulary models to effectively incorporate OOD unlabeled images, achieving state-of-the-art results.

Findings

Using additional unlabeled images enhances semi-supervised segmentation performance.

Open-vocabulary pseudo-labeling of OOD images leads to substantial accuracy gains.

SemiOVS outperforms existing methods by +3.5 and +3.0 mIoU on Pascal VOC.

Abstract

In semi-supervised semantic segmentation, existing studies have shown promising results in academic settings with controlled splits of benchmark datasets. However, the potential benefits of leveraging significantly larger sets of unlabeled images remain unexplored. In real-world scenarios, abundant unlabeled images are often available from online sources (web-scraped images) or large-scale datasets. However, these images may have different distributions from those of the target dataset, a situation known as out-of-distribution (OOD). Using these images as unlabeled data in semi-supervised learning can lead to inaccurate pseudo-labels, potentially misguiding network training. In this paper, we propose a new semi-supervised semantic segmentation framework with an open-vocabulary segmentation model (SemiOVS) to effectively utilize unlabeled OOD images. Extensive experiments on Pascal VOC and Context datasets demonstrate two key findings: (1) using additional unlabeled images improves the performance of semi-supervised learners in scenarios with few labels, and (2) using the open-vocabulary segmentation (OVS) model to pseudo-label OOD images leads to substantial performance gains. In particular, SemiOVS outperforms existing PrevMatch and SemiVL methods by +3.5 and +3.0 mIoU, respectively, on Pascal VOC with a 92-label setting, achieving state-of-the-art performance. These findings demonstrate that our approach effectively utilizes abundant unlabeled OOD images for semantic segmentation tasks. We hope this work can inspire future research and real-world applications. The code is available at https://github.com/wooseok-shin/SemiOVS