Fairness Continual Learning Approach to Semantic Scene Understanding in Open-World Environments

TL;DR

This paper introduces a fairness-focused continual learning framework for semantic segmentation that addresses catastrophic forgetting and background shift, achieving state-of-the-art results on multiple benchmarks.

Contribution

It proposes a novel fairness continual learning framework with a Prototypical Contrastive Clustering loss and Conditional Structural Consistency loss, enhancing fairness and performance.

Findings

Achieved state-of-the-art results on ADE20K, Cityscapes, and Pascal VOC.

Improved fairness in continual semantic segmentation models.

Effectively mitigated catastrophic forgetting and background shift.

Abstract

Continual semantic segmentation aims to learn new classes while maintaining the information from the previous classes. Although prior studies have shown impressive progress in recent years, the fairness concern in the continual semantic segmentation needs to be better addressed. Meanwhile, fairness is one of the most vital factors in deploying the deep learning model, especially in human-related or safety applications. In this paper, we present a novel Fairness Continual Learning approach to the semantic segmentation problem. In particular, under the fairness objective, a new fairness continual learning framework is proposed based on class distributions. Then, a novel Prototypical Contrastive Clustering loss is proposed to address the significant challenges in continual learning, i.e., catastrophic forgetting and background shift. Our proposed loss has also been proven as a novel, generalized learning paradigm of knowledge distillation commonly used in continual learning. Moreover, the proposed Conditional Structural Consistency loss further regularized the structural constraint of the predicted segmentation. Our proposed approach has achieved State-of-the-Art performance on three standard scene understanding benchmarks, i.e., ADE20K, Cityscapes, and Pascal VOC, and promoted the fairness of the segmentation model.