Masked Image Modeling Boosting Semi-Supervised Semantic Segmentation

TL;DR

This paper introduces a novel class-wise masked image modeling approach that enhances semi-supervised semantic segmentation by improving intra-class feature connections and regularization, achieving state-of-the-art results.

Contribution

The paper proposes a class-wise masked image modeling method with feature aggregation for semi-supervised segmentation, extending masked image modeling to better utilize unlabeled data.

Findings

Achieves state-of-the-art performance on benchmark datasets.

Effectively mitigates semantic confusion through class-wise reconstruction.

Enhances regularization in semantic space.

Abstract

In view of the fact that semi- and self-supervised learning share a fundamental principle, effectively modeling knowledge from unlabeled data, various semi-supervised semantic segmentation methods have integrated representative self-supervised learning paradigms for further regularization. However, the potential of the state-of-the-art generative self-supervised paradigm, masked image modeling, has been scarcely studied. This paradigm learns the knowledge through establishing connections between the masked and visible parts of masked image, during the pixel reconstruction process. By inheriting and extending this insight, we successfully leverage masked image modeling to boost semi-supervised semantic segmentation. Specifically, we introduce a novel class-wise masked image modeling that independently reconstructs different image regions according to their respective classes. In this way, the mask-induced connections are established within each class, mitigating the semantic confusion that arises from plainly reconstructing images in basic masked image modeling. To strengthen these intra-class connections, we further develop a feature aggregation strategy that minimizes the distances between features corresponding to the masked and visible parts within the same class. Additionally, in semantic space, we explore the application of masked image modeling to enhance regularization. Extensive experiments conducted on well-known benchmarks demonstrate that our approach achieves state-of-the-art performance. The code will be available at https://github.com/haoxt/S4MIM.