Learning Debiased and Disentangled Representations for Semantic Segmentation

TL;DR

This paper introduces a stochastic training scheme for semantic segmentation that encourages models to learn debiased and disentangled features, improving performance especially on under-represented classes.

Contribution

It proposes a model-agnostic, stochastic method to reduce feature entanglement and bias in semantic segmentation models, addressing a gap in complex dense prediction tasks.

Findings

Improved segmentation accuracy on multiple benchmarks.

Significant gains on under-represented classes.

Effective reduction of feature dependencies among classes.

Abstract

Deep neural networks are susceptible to learn biased models with entangled feature representations, which may lead to subpar performances on various downstream tasks. This is particularly true for under-represented classes, where a lack of diversity in the data exacerbates the tendency. This limitation has been addressed mostly in classification tasks, but there is little study on additional challenges that may appear in more complex dense prediction problems including semantic segmentation. To this end, we propose a model-agnostic and stochastic training scheme for semantic segmentation, which facilitates the learning of debiased and disentangled representations. For each class, we first extract class-specific information from the highly entangled feature map. Then, information related to a randomly sampled class is suppressed by a feature selection process in the feature space. By randomly eliminating certain class information in each training iteration, we effectively reduce feature dependencies among classes, and the model is able to learn more debiased and disentangled feature representations. Models trained with our approach demonstrate strong results on multiple semantic segmentation benchmarks, with especially notable performance gains on under-represented classes.