Mining Contextual Information Beyond Image for Semantic Segmentation

TL;DR

This paper introduces a dataset-level feature memory module for semantic segmentation that leverages class-level contextual information beyond individual images, improving performance across multiple benchmarks.

Contribution

It proposes a novel feature memory and class probability aggregation method to enhance pixel representations using dataset-level context, which is integrated into existing segmentation frameworks.

Findings

Achieves state-of-the-art results on ADE20K, LIP, Cityscapes, and COCO-Stuff.

Improves intra-class compactness and inter-class dispersion in segmentation.

Enhances pixel representations by incorporating dataset-level class information.

Abstract

This paper studies the context aggregation problem in semantic image segmentation. The existing researches focus on improving the pixel representations by aggregating the contextual information within individual images. Though impressive, these methods neglect the significance of the representations of the pixels of the corresponding class beyond the input image. To address this, this paper proposes to mine the contextual information beyond individual images to further augment the pixel representations. We first set up a feature memory module, which is updated dynamically during training, to store the dataset-level representations of various categories. Then, we learn class probability distribution of each pixel representation under the supervision of the ground-truth segmentation. At last, the representation of each pixel is augmented by aggregating the dataset-level representations based on the corresponding class probability distribution. Furthermore, by utilizing the stored dataset-level representations, we also propose a representation consistent learning strategy to make the classification head better address intra-class compactness and inter-class dispersion. The proposed method could be effortlessly incorporated into existing segmentation frameworks (e.g., FCN, PSPNet, OCRNet and DeepLabV3) and brings consistent performance improvements. Mining contextual information beyond image allows us to report state-of-the-art performance on various benchmarks: ADE20K, LIP, Cityscapes and COCO-Stuff.