Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation

TL;DR

This paper introduces a self-supervised equivariant attention mechanism that enhances weakly supervised semantic segmentation by improving class activation maps through consistency regularization and pixel correlation, outperforming existing methods.

Contribution

The paper proposes a novel self-supervised attention mechanism with a pixel correlation module to improve CAM quality in weakly supervised segmentation.

Findings

Outperforms state-of-the-art on PASCAL VOC 2012

Uses consistency regularization for CAMs

Refines predictions with pixel correlation

Abstract

Image-level weakly supervised semantic segmentation is a challenging problem that has been deeply studied in recent years. Most of advanced solutions exploit class activation map (CAM). However, CAMs can hardly serve as the object mask due to the gap between full and weak supervisions. In this paper, we propose a self-supervised equivariant attention mechanism (SEAM) to discover additional supervision and narrow the gap. Our method is based on the observation that equivariance is an implicit constraint in fully supervised semantic segmentation, whose pixel-level labels take the same spatial transformation as the input images during data augmentation. However, this constraint is lost on the CAMs trained by image-level supervision. Therefore, we propose consistency regularization on predicted CAMs from various transformed images to provide self-supervision for network learning. Moreover, we propose a pixel correlation module (PCM), which exploits context appearance information and refines the prediction of current pixel by its similar neighbors, leading to further improvement on CAMs consistency. Extensive experiments on PASCAL VOC 2012 dataset demonstrate our method outperforms state-of-the-art methods using the same level of supervision. The code is released online.