On the Calibration of Uncertainty Estimation in LiDAR-based Semantic Segmentation

TL;DR

This paper introduces a new metric for assessing confidence calibration in LiDAR-based semantic segmentation, emphasizing class-specific calibration and dataset quality improvement for autonomous driving safety.

Contribution

It proposes a class-wise calibration metric based on sparsification curves and demonstrates its use in evaluating uncertainty methods and identifying annotation issues.

Findings

The metric effectively measures class-specific calibration quality.

Uncertainty estimation methods vary in calibration performance across classes.

The approach can detect label problems in datasets.

Abstract

The confidence calibration of deep learning-based perception models plays a crucial role in their reliability. Especially in the context of autonomous driving, downstream tasks like prediction and planning depend on accurate confidence estimates. In point-wise multiclass classification tasks like sematic segmentation the model has to deal with heavy class imbalances. Due to their underrepresentation, the confidence calibration of classes with smaller instances is challenging but essential, not only for safety reasons. We propose a metric to measure the confidence calibration quality of a semantic segmentation model with respect to individual classes. It is calculated by computing sparsification curves for each class based on the uncertainty estimates. We use the classification calibration metric to evaluate uncertainty estimation methods with respect to their confidence calibration of underrepresented classes. We furthermore suggest a double use for the method to automatically find label problems to improve the quality of hand- or auto-annotated datasets.