Explorations of the Softmax Space: Knowing When the Neural Network Doesn't Know

TL;DR

This paper introduces a confidence measure for neural networks based on clustering softmax outputs, enabling the system to identify uncertain predictions and defer decisions, thereby improving reliability in critical applications.

Contribution

It proposes a novel clustering-based confidence measure using softmax vectors to determine when to defer predictions, applicable across different models and datasets.

Findings

Effective in identifying low-confidence predictions

Works consistently across datasets and models

Enables deferral of uncertain predictions to humans

Abstract

Ensuring the reliability of automated decision-making based on neural networks will be crucial as Artificial Intelligence systems are deployed more widely in critical situations. This paper proposes a new approach for measuring confidence in the predictions of any neural network that relies on the predictions of a softmax layer. We identify that a high-accuracy trained network may have certain outputs for which there should be low confidence. In such cases, decisions should be deferred and it is more appropriate for the network to provide a \textit{not known} answer to a corresponding classification task. Our approach clusters the vectors in the softmax layer to measure distances between cluster centroids and network outputs. We show that a cluster with centroid calculated simply as the mean softmax output for all correct predictions can serve as a suitable proxy in the evaluation of confidence. Defining a distance threshold for a class as the smallest distance from an incorrect prediction to the given class centroid offers a simple approach to adding \textit{not known} answers to any network classification falling outside of the threshold. We evaluate the approach on the MNIST and CIFAR-10 datasets using a Convolutional Neural Network and a Vision Transformer, respectively. The results show that our approach is consistent across datasets and network models, and indicate that the proposed distance metric can offer an efficient way of determining when automated predictions are acceptable and when they should be deferred to human operators.