On the Structures of Representation for the Robustness of Semantic Segmentation to Input Corruption

TL;DR

This paper analyzes how different output structures in semantic segmentation models affect robustness to input corruption and proposes a new combined approach, SCrIBE, that outperforms existing methods across various corruptions.

Contribution

The paper introduces SCrIBE, a novel combination of Sigmoid and Implicit Background Estimation, enhancing robustness in semantic segmentation under corrupted inputs.

Findings

SCrIBE achieves a higher mean IoU of 42.1 across corruptions.

Analysis reveals structural differences influence robustness.

SCrIBE outperforms baseline and IBE methods in robustness tests.

Abstract

Semantic segmentation is a scene understanding task at the heart of safety-critical applications where robustness to corrupted inputs is essential. Implicit Background Estimation (IBE) has demonstrated to be a promising technique to improve the robustness to out-of-distribution inputs for semantic segmentation models for little to no cost. In this paper, we provide analysis comparing the structures learned as a result of optimization objectives that use Softmax, IBE, and Sigmoid in order to improve understanding their relationship to robustness. As a result of this analysis, we propose combining Sigmoid with IBE (SCrIBE) to improve robustness. Finally, we demonstrate that SCrIBE exhibits superior segmentation performance aggregated across all corruptions and severity levels with a mIOU of 42.1 compared to both IBE 40.3 and the Softmax Baseline 37.5.