On the calibration of underrepresented classes in LiDAR-based semantic segmentation

TL;DR

This paper evaluates how well LiDAR-based semantic segmentation models are calibrated for underrepresented classes, especially vulnerable road users, using a new metric to compare different architectures and probabilistic approaches.

Contribution

It introduces a class-wise calibration evaluation method for LiDAR segmentation models, focusing on safety-critical underrepresented classes, and analyzes the dependency between performance and calibration quality.

Findings

Probabilistic models show improved calibration over deterministic ones.

Calibration quality varies significantly across classes, especially underrepresented ones.

The proposed metric effectively compares calibration performance of different models.

Abstract

The calibration of deep learning-based perception models plays a crucial role in their reliability. Our work focuses on a class-wise evaluation of several model's confidence performance for LiDAR-based semantic segmentation with the aim of providing insights into the calibration of underrepresented classes. Those classes often include VRUs and are thus of particular interest for safety reasons. With the help of a metric based on sparsification curves we compare the calibration abilities of three semantic segmentation models with different architectural concepts, each in a in deterministic and a probabilistic version. By identifying and describing the dependency between the predictive performance of a class and the respective calibration quality we aim to facilitate the model selection and refinement for safety-critical applications.