Weakly-Supervised Semantic Segmentation with Visual Words Learning and Hybrid Pooling

TL;DR

This paper introduces a novel weakly-supervised semantic segmentation method that uses visual words learning and hybrid pooling to improve object region discovery and reduce background noise, achieving state-of-the-art results.

Contribution

It proposes a visual words learning module and hybrid pooling approach to enhance CAMs in weakly-supervised segmentation, addressing partial focus and background issues.

Findings

Achieved 70.6% mIoU on PASCAL VOC 2012 val set

Achieved 70.7% mIoU on PASCAL VOC 2012 test set

Achieved 36.2% mIoU on MS COCO 2014 val set

Abstract

Weakly-Supervised Semantic Segmentation (WSSS) methods with image-level labels generally train a classification network to generate the Class Activation Maps (CAMs) as the initial coarse segmentation labels. However, current WSSS methods still perform far from satisfactorily because their adopted CAMs 1) typically focus on partial discriminative object regions and 2) usually contain useless background regions. These two problems are attributed to the sole image-level supervision and aggregation of global information when training the classification networks. In this work, we propose the visual words learning module and hybrid pooling approach, and incorporate them in the classification network to mitigate the above problems. In the visual words learning module, we counter the first problem by enforcing the classification network to learn fine-grained visual word labels so that more object extents could be discovered. Specifically, the visual words are learned with a codebook, which could be updated via two proposed strategies, i.e. learning-based strategy and memory-bank strategy. The second drawback of CAMs is alleviated with the proposed hybrid pooling, which incorporates the global average and local discriminative information to simultaneously ensure object completeness and reduce background regions. We evaluated our methods on PASCAL VOC 2012 and MS COCO 2014 datasets. Without any extra saliency prior, our method achieved 70.6% and 70.7% mIoU on the $val$ and $test$ set of PASCAL VOC dataset, respectively, and 36.2% mIoU on the $val$ set of MS COCO dataset, which significantly surpassed the performance of state-of-the-art WSSS methods.