Puzzle-CAM: Improved localization via matching partial and full features

TL;DR

Puzzle-CAM enhances weakly-supervised semantic segmentation by matching features from image patches and the whole image, leading to more comprehensive object localization without extra parameters.

Contribution

It introduces a novel puzzle module with regularization to improve object region activation in WSSS using only image-level supervision.

Findings

Outperforms previous state-of-the-art on PASCAL VOC 2012

Activates more complete object regions

Does not require additional parameters

Abstract

Weakly-supervised semantic segmentation (WSSS) is introduced to narrow the gap for semantic segmentation performance from pixel-level supervision to image-level supervision. Most advanced approaches are based on class activation maps (CAMs) to generate pseudo-labels to train the segmentation network. The main limitation of WSSS is that the process of generating pseudo-labels from CAMs that use an image classifier is mainly focused on the most discriminative parts of the objects. To address this issue, we propose Puzzle-CAM, a process that minimizes differences between the features from separate patches and the whole image. Our method consists of a puzzle module and two regularization terms to discover the most integrated region in an object. Puzzle-CAM can activate the overall region of an object using image-level supervision without requiring extra parameters. % In experiments, Puzzle-CAM outperformed previous state-of-the-art methods using the same labels for supervision on the PASCAL VOC 2012 test dataset. In experiments, Puzzle-CAM outperformed previous state-of-the-art methods using the same labels for supervision on the PASCAL VOC 2012 dataset. Code associated with our experiments is available at https://github.com/OFRIN/PuzzleCAM.