Evaluating Uncertainty Calibration for Open-Set Recognition

TL;DR

This paper evaluates how well existing calibration methods for neural networks perform in open-set recognition scenarios, revealing their limitations and emphasizing the need for new calibration techniques.

Contribution

It provides a novel evaluation framework for calibration methods in open-set conditions, demonstrating the inadequacy of closed-set calibration approaches.

Findings

Closed-set calibration methods are less effective in open-set recognition.

Existing calibration techniques do not adequately address out-of-distribution data.

Highlighting the need for developing new calibration methods for open-set scenarios.

Abstract

Despite achieving enormous success in predictive accuracy for visual classification problems, deep neural networks (DNNs) suffer from providing overconfident probabilities on out-of-distribution (OOD) data. Yet, accurate uncertainty estimation is crucial for safe and reliable robot autonomy. In this paper, we evaluate popular calibration techniques for open-set conditions in a way that is distinctly different from the conventional evaluation of calibration methods on OOD data. Our results show that closed-set DNN calibration approaches are much less effective for open-set recognition, which highlights the need to develop new DNN calibration methods to address this problem.