SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation

TL;DR

This paper introduces SegmentMeIfYouCan, a comprehensive benchmark with datasets and evaluation tools for anomaly and obstacle segmentation, addressing the lack of robust benchmarks for detecting unseen objects in safety-critical scenarios.

Contribution

It provides new datasets and a test suite for anomaly and obstacle segmentation, enabling better evaluation of methods on unseen objects in real-world conditions.

Findings

State-of-the-art methods show limited performance on new datasets

Datasets increase diversity and challenge in anomaly segmentation

Component-wise metrics offer new insights into model performance

Abstract

State-of-the-art semantic or instance segmentation deep neural networks (DNNs) are usually trained on a closed set of semantic classes. As such, they are ill-equipped to handle previously-unseen objects. However, detecting and localizing such objects is crucial for safety-critical applications such as perception for automated driving, especially if they appear on the road ahead. While some methods have tackled the tasks of anomalous or out-of-distribution object segmentation, progress remains slow, in large part due to the lack of solid benchmarks; existing datasets either consist of synthetic data, or suffer from label inconsistencies. In this paper, we bridge this gap by introducing the "SegmentMeIfYouCan" benchmark. Our benchmark addresses two tasks: Anomalous object segmentation, which considers any previously-unseen object category; and road obstacle segmentation, which focuses on any object on the road, may it be known or unknown. We provide two corresponding datasets together with a test suite performing an in-depth method analysis, considering both established pixel-wise performance metrics and recent component-wise ones, which are insensitive to object sizes. We empirically evaluate multiple state-of-the-art baseline methods, including several models specifically designed for anomaly / obstacle segmentation, on our datasets and on public ones, using our test suite. The anomaly and obstacle segmentation results show that our datasets contribute to the diversity and difficulty of both data landscapes.