Detecting and Learning the Unknown in Semantic Segmentation

TL;DR

This paper explores methods for detecting and learning unknown objects in semantic segmentation for automated driving, emphasizing entropy-based detection and unsupervised learning of anomalies to improve safety and adaptability.

Contribution

It introduces an information-theoretic perspective on anomalies, demonstrates the effectiveness of high entropy training for unknown object detection, and proposes unsupervised learning of anomalies for online adaptation.

Findings

High entropy responses improve anomaly detection accuracy.

Unsupervised learning effectively incorporates anomalies into semantic models.

Entropy-based methods outperform recent anomaly detection techniques.

Abstract

Semantic segmentation is a crucial component for perception in automated driving. Deep neural networks (DNNs) are commonly used for this task and they are usually trained on a closed set of object classes appearing in a closed operational domain. However, this is in contrast to the open world assumption in automated driving that DNNs are deployed to. Therefore, DNNs necessarily face data that they have never encountered previously, also known as anomalies, which are extremely safety-critical to properly cope with. In this work, we first give an overview about anomalies from an information-theoretic perspective. Next, we review research in detecting semantically unknown objects in semantic segmentation. We demonstrate that training for high entropy responses on anomalous objects outperforms other recent methods, which is in line with our theoretical findings. Moreover, we examine a method to assess the occurrence frequency of anomalies in order to select anomaly types to include into a model's set of semantic categories. We demonstrate that these anomalies can then be learned in an unsupervised fashion, which is particularly suitable in online applications based on deep learning.