Understanding Softmax Confidence and Uncertainty

TL;DR

This paper investigates why softmax confidence sometimes correlates with uncertainty, revealing biases and factors influencing its reliability, and analyzing how network structure and data overlap affect uncertainty estimation.

Contribution

It identifies implicit biases that cause softmax confidence to reflect epistemic uncertainty and explains why low-dimensional intuitions are misleading.

Findings

Softmax confidence can correlate with uncertainty due to decision boundary structure.

Filtering effects of deep networks influence confidence reliability.

Overlap in representations affects uncertainty estimation more than extrapolation.

Abstract

It is often remarked that neural networks fail to increase their uncertainty when predicting on data far from the training distribution. Yet naively using softmax confidence as a proxy for uncertainty achieves modest success in tasks exclusively testing for this, e.g., out-of-distribution (OOD) detection. This paper investigates this contradiction, identifying two implicit biases that do encourage softmax confidence to correlate with epistemic uncertainty: 1) Approximately optimal decision boundary structure, and 2) Filtering effects of deep networks. It describes why low-dimensional intuitions about softmax confidence are misleading. Diagnostic experiments quantify reasons softmax confidence can fail, finding that extrapolations are less to blame than overlap between training and OOD data in final-layer representations. Pre-trained/fine-tuned networks reduce this overlap.