Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts

TL;DR

This paper proposes a novel generative augmentation and training strategy to improve semantic segmentation models' ability to detect anomalies and generalize across multiple distribution shifts, achieving state-of-the-art results.

Contribution

It introduces a new augmentation method and uncertainty recalibration technique to enhance model robustness against semantic and domain shifts in open-world scenarios.

Findings

Achieves state-of-the-art OOD detection performance.

Improves domain generalization across benchmarks.

Effectively distinguishes semantic shifts from domain shifts.

Abstract

In open-world scenarios, where both novel classes and domains may exist, an ideal segmentation model should detect anomaly classes for safety and generalize to new domains. However, existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts, leading to poor out-of-distribution (OOD) detection or domain generalization performance. In this work, we aim to equip the model to generalize effectively to covariate-shift regions while precisely identifying semantic-shift regions. To achieve this, we design a novel generative augmentation method to produce coherent images that incorporate both anomaly (or novel) objects and various covariate shifts at both image and object levels. Furthermore, we introduce a training strategy that recalibrates uncertainty specifically for semantic shifts and enhances the feature extractor to align features associated with domain shifts. We validate the effectiveness of our method across benchmarks featuring both semantic and domain shifts. Our method achieves state-of-the-art performance across all benchmarks for both OOD detection and domain generalization. Code is available at https://github.com/gaozhitong/MultiShiftSeg.