Weakly Supervised Semantic Segmentation by Knowledge Graph Inference

TL;DR

This paper proposes a graph reasoning-based method to improve weakly supervised semantic segmentation by integrating external knowledge and reasoning about inter-class dependencies, achieving state-of-the-art results.

Contribution

It introduces a novel graph reasoning framework that enhances both classification and segmentation stages using knowledge graphs and textual data, addressing limitations of CNN-based local convolutions.

Findings

Achieved state-of-the-art performance on PASCAL VOC 2012.

Improved pseudo-label completeness through inter-class dependency reasoning.

Enhanced feature representation with the Graph Reasoning Mapping module.

Abstract

Currently, existing efforts in Weakly Supervised Semantic Segmentation (WSSS) based on Convolutional Neural Networks (CNNs) have predominantly focused on enhancing the multi-label classification network stage, with limited attention given to the equally important downstream segmentation network. Furthermore, CNN-based local convolutions lack the ability to model the extensive inter-category dependencies. Therefore, this paper introduces a graph reasoning-based approach to enhance WSSS. The aim is to improve WSSS holistically by simultaneously enhancing both the multi-label classification and segmentation network stages. In the multi-label classification network segment, external knowledge is integrated, coupled with GCNs, to globally reason about inter-class dependencies. This encourages the network to uncover features in non-salient regions of images, thereby refining the completeness of generated pseudo-labels. In the segmentation network segment, the proposed Graph Reasoning Mapping (GRM) module is employed to leverage knowledge obtained from textual databases, facilitating contextual reasoning for class representation within image regions. This GRM module enhances feature representation in high-level semantics of the segmentation network's local convolutions, while dynamically learning semantic coherence for individual samples. Using solely image-level supervision, we have achieved state-of-the-art performance in WSSS on the PASCAL VOC 2012 and MS-COCO datasets. Extensive experimentation on both the multi-label classification and segmentation network stages underscores the effectiveness of the proposed graph reasoning approach for advancing WSSS.