Segment Every Out-of-Distribution Object

TL;DR

The paper presents S2M, a novel method that converts anomaly scores into segmentation masks to effectively detect out-of-distribution objects in semantic segmentation, outperforming existing approaches.

Contribution

S2M introduces a threshold-free framework that transforms anomaly scores into prompts for segmentation, improving OoD detection accuracy in real-world applications.

Findings

S2M achieves approximately 20% higher IoU than state-of-the-art methods.

S2M improves mean F1 score by about 40%.

The method performs well across multiple benchmarks.

Abstract

Semantic segmentation models, while effective for in-distribution categories, face challenges in real-world deployment due to encountering out-of-distribution (OoD) objects. Detecting these OoD objects is crucial for safety-critical applications. Existing methods rely on anomaly scores, but choosing a suitable threshold for generating masks presents difficulties and can lead to fragmentation and inaccuracy. This paper introduces a method to convert anomaly \textbf{S}core \textbf{T}o segmentation \textbf{M}ask, called S2M, a simple and effective framework for OoD detection in semantic segmentation. Unlike assigning anomaly scores to pixels, S2M directly segments the entire OoD object. By transforming anomaly scores into prompts for a promptable segmentation model, S2M eliminates the need for threshold selection. Extensive experiments demonstrate that S2M outperforms the state-of-the-art by approximately 20% in IoU and 40% in mean F1 score, on average, across various benchmarks including Fishyscapes, Segment-Me-If-You-Can, and RoadAnomaly datasets.