FSNet: A Failure Detection Framework for Semantic Segmentation

TL;DR

This paper introduces FSNet, a framework that detects pixel-level failures in semantic segmentation models for autonomous vehicles by leveraging internal features, significantly improving failure detection performance across multiple datasets.

Contribution

The paper presents a novel failure detection framework that exploits internal features of segmentation models, enhancing the ability to identify misclassifications during deployment.

Findings

Achieves over 12% improvement in AUPR-Error on Cityscapes.

Outperforms state-of-the-art methods on BDD100K and Mapillary datasets.

Effectively flags segmentation failures to improve safety in autonomous driving.

Abstract

Semantic segmentation is an important task that helps autonomous vehicles understand their surroundings and navigate safely. During deployment, even the most mature segmentation models are vulnerable to various external factors that can degrade the segmentation performance with potentially catastrophic consequences for the vehicle and its surroundings. To address this issue, we propose a failure detection framework to identify pixel-level misclassification. We do so by exploiting internal features of the segmentation model and training it simultaneously with a failure detection network. During deployment, the failure detector can flag areas in the image where the segmentation model have failed to segment correctly. We evaluate the proposed approach against state-of-the-art methods and achieve 12.30%, 9.46%, and 9.65% performance improvement in the AUPR-Error metric for Cityscapes, BDD100K, and Mapillary semantic segmentation datasets.