Activation Modulation and Recalibration Scheme for Weakly Supervised Semantic Segmentation

TL;DR

This paper introduces a novel activation modulation and recalibration scheme for weakly supervised semantic segmentation, improving pseudo label quality and achieving state-of-the-art results on PASCAL VOC 2012.

Contribution

The proposed AMR scheme with attention modulation and cross pseudo supervision enhances segmentation accuracy and can be integrated with other methods for performance gains.

Findings

Achieved new state-of-the-art on PASCAL VOC 2012

Outperformed methods using stronger supervision

Scheme is plug-and-play and versatile

Abstract

Image-level weakly supervised semantic segmentation (WSSS) is a fundamental yet challenging computer vision task facilitating scene understanding and automatic driving. Most existing methods resort to classification-based Class Activation Maps (CAMs) to play as the initial pseudo labels, which tend to focus on the discriminative image regions and lack customized characteristics for the segmentation task. To alleviate this issue, we propose a novel activation modulation and recalibration (AMR) scheme, which leverages a spotlight branch and a compensation branch to obtain weighted CAMs that can provide recalibration supervision and task-specific concepts. Specifically, an attention modulation module (AMM) is employed to rearrange the distribution of feature importance from the channel-spatial sequential perspective, which helps to explicitly model channel-wise interdependencies and spatial encodings to adaptively modulate segmentation-oriented activation responses. Furthermore, we introduce a cross pseudo supervision for dual branches, which can be regarded as a semantic similar regularization to mutually refine two branches. Extensive experiments show that AMR establishes a new state-of-the-art performance on the PASCAL VOC 2012 dataset, surpassing not only current methods trained with the image-level of supervision but also some methods relying on stronger supervision, such as saliency label. Experiments also reveal that our scheme is plug-and-play and can be incorporated with other approaches to boost their performance.