SSR: Semantic and Spatial Rectification for CLIP-based Weakly Supervised Segmentation

TL;DR

This paper introduces SSR, a novel method that improves CLIP-based weakly supervised segmentation by addressing over-activation issues through semantic and spatial rectification techniques, leading to state-of-the-art results.

Contribution

The paper proposes Semantic and Spatial Rectification (SSR), combining cross-modal prototype alignment and superpixel-guided correction to enhance CLIP-based segmentation accuracy.

Findings

Outperforms existing methods on PASCAL VOC and MS COCO datasets.

Achieves 79.5% and 50.6% mIoU scores respectively.

Effectively reduces over-activation in non-target regions.

Abstract

In recent years, Contrastive Language-Image Pretraining (CLIP) has been widely applied to Weakly Supervised Semantic Segmentation (WSSS) tasks due to its powerful cross-modal semantic understanding capabilities. This paper proposes a novel Semantic and Spatial Rectification (SSR) method to address the limitations of existing CLIP-based weakly supervised semantic segmentation approaches: over-activation in non-target foreground regions and background areas. Specifically, at the semantic level, the Cross-Modal Prototype Alignment (CMPA) establishes a contrastive learning mechanism to enforce feature space alignment across modalities, reducing inter-class overlap while enhancing semantic correlations, to rectify over-activation in non-target foreground regions effectively; at the spatial level, the Superpixel-Guided Correction (SGC) leverages superpixel-based spatial priors to precisely filter out interference from non-target regions during affinity propagation, significantly rectifying background over-activation. Extensive experiments on the PASCAL VOC and MS COCO datasets demonstrate that our method outperforms all single-stage approaches, as well as more complex multi-stage approaches, achieving mIoU scores of 79.5% and 50.6%, respectively.