A Benchmark for Out of Distribution Detection in Point Cloud 3D Semantic Segmentation

TL;DR

This paper introduces two new datasets to benchmark out-of-distribution detection in 3D semantic segmentation of LiDAR data, evaluating methods like Deep Ensembles and Flipout for safety-critical applications.

Contribution

It presents the first benchmark datasets for OOD detection in 3D point cloud segmentation and compares the effectiveness of ensemble-based methods.

Findings

Deep Ensembles outperform Flipout in OOD detection accuracy.

AUROC scores are higher for Deep Ensembles across datasets.

Proposed datasets enable standardized evaluation of OOD detection methods.

Abstract

Safety-critical applications like autonomous driving use Deep Neural Networks (DNNs) for object detection and segmentation. The DNNs fail to predict when they observe an Out-of-Distribution (OOD) input leading to catastrophic consequences. Existing OOD detection methods were extensively studied for image inputs but have not been explored much for LiDAR inputs. So in this study, we proposed two datasets for benchmarking OOD detection in 3D semantic segmentation. We used Maximum Softmax Probability and Entropy scores generated using Deep Ensembles and Flipout versions of RandLA-Net as OOD scores. We observed that Deep Ensembles out perform Flipout model in OOD detection with greater AUROC scores for both datasets.