Classifier Enhancement Using Extended Context and Domain Experts for Semantic Segmentation

TL;DR

This paper introduces an Extended Context-Aware Classifier (ECAC) that dynamically adjusts pixel classification using dataset and image-specific context, improving semantic segmentation accuracy especially for minority classes.

Contribution

The paper proposes a novel ECAC method that incorporates global and local context via a memory bank and teacher-student paradigm, enhancing segmentation performance.

Findings

Achieves state-of-the-art results on ADE20K, COCO-Stuff10K, and Pascal-Context datasets.

Improves minority class segmentation accuracy.

Demonstrates effectiveness of dynamic context adjustment in classifiers.

Abstract

Prevalent semantic segmentation methods generally adopt a vanilla classifier to categorize each pixel into specific classes. Although such a classifier learns global information from the training data, this information is represented by a set of fixed parameters (weights and biases). However, each image has a different class distribution, which prevents the classifier from addressing the unique characteristics of individual images. At the dataset level, class imbalance leads to segmentation results being biased towards majority classes, limiting the model's effectiveness in identifying and segmenting minority class regions. In this paper, we propose an Extended Context-Aware Classifier (ECAC) that dynamically adjusts the classifier using global (dataset-level) and local (image-level) contextual information. Specifically, we leverage a memory bank to learn dataset-level contextual information of each class, incorporating the class-specific contextual information from the current image to improve the classifier for precise pixel labeling. Additionally, a teacher-student network paradigm is adopted, where the domain expert (teacher network) dynamically adjusts contextual information with ground truth and transfers knowledge to the student network. Comprehensive experiments illustrate that the proposed ECAC can achieve state-of-the-art performance across several datasets, including ADE20K, COCO-Stuff10K, and Pascal-Context.