DHR: Dual Features-Driven Hierarchical Rebalancing in Inter- and Intra-Class Regions for Weakly-Supervised Semantic Segmentation

TL;DR

This paper introduces a hierarchical rebalancing method using dual feature maps for weakly-supervised semantic segmentation, significantly improving class coverage and accuracy across multiple benchmarks.

Contribution

It proposes a novel hierarchical mask enhancement technique leveraging unsupervised and weakly-supervised features to better restore all classes, especially minor ones.

Findings

Achieved state-of-the-art results on five benchmarks.

Reduced the gap with fully supervised methods by over 84% on VOC.

Effectively restores minor classes in complex images.

Abstract

Weakly-supervised semantic segmentation (WSS) ensures high-quality segmentation with limited data and excels when employed as input seed masks for large-scale vision models such as Segment Anything. However, WSS faces challenges related to minor classes since those are overlooked in images with adjacent multiple classes, a limitation originating from the overfitting of traditional expansion methods like Random Walk. We first address this by employing unsupervised and weakly-supervised feature maps instead of conventional methodologies, allowing for hierarchical mask enhancement. This method distinctly categorizes higher-level classes and subsequently separates their associated lower-level classes, ensuring all classes are correctly restored in the mask without losing minor ones. Our approach, validated through extensive experimentation, significantly improves WSS across five benchmarks (VOC: 79.8\%, COCO: 53.9\%, Context: 49.0\%, ADE: 32.9\%, Stuff: 37.4\%), reducing the gap with fully supervised methods by over 84\% on the VOC validation set. Code is available at https://github.com/shjo-april/DHR.