Multi-Granularity Denoising and Bidirectional Alignment for Weakly Supervised Semantic Segmentation

TL;DR

This paper introduces MDBA, an end-to-end model that improves weakly supervised semantic segmentation by denoising pseudo labels and aligning data distributions, achieving state-of-the-art results on PASCAL VOC 2012.

Contribution

The paper proposes a novel multi-granularity denoising and bidirectional alignment framework for WSSS, addressing noisy labels and multi-class generalization challenges.

Findings

Achieves 69.5% mIoU on PASCAL VOC 2012 validation set.

Effective noise filtering and bidirectional alignment improve segmentation accuracy.

Source code and models are publicly available.

Abstract

Weakly supervised semantic segmentation (WSSS) models relying on class activation maps (CAMs) have achieved desirable performance comparing to the non-CAMs-based counterparts. However, to guarantee WSSS task feasible, we need to generate pseudo labels by expanding the seeds from CAMs which is complex and time-consuming, thus hindering the design of efficient end-to-end (single-stage) WSSS approaches. To tackle the above dilemma, we resort to the off-the-shelf and readily accessible saliency maps for directly obtaining pseudo labels given the image-level class labels. Nevertheless, the salient regions may contain noisy labels and cannot seamlessly fit the target objects, and saliency maps can only be approximated as pseudo labels for simple images containing single-class objects. As such, the achieved segmentation model with these simple images cannot generalize well to the complex images containing multi-class objects. To this end, we propose an end-to-end multi-granularity denoising and bidirectional alignment (MDBA) model, to alleviate the noisy label and multi-class generalization issues. Specifically, we propose the online noise filtering and progressive noise detection modules to tackle image-level and pixel-level noise, respectively. Moreover, a bidirectional alignment mechanism is proposed to reduce the data distribution gap at both input and output space with simple-to-complex image synthesis and complex-to-simple adversarial learning. MDBA can reach the mIoU of 69.5\% and 70.2\% on validation and test sets for the PASCAL VOC 2012 dataset. The source codes and models have been made available at \url{https://github.com/NUST-Machine-Intelligence-Laboratory/MDBA}.