Inhibited Softmax for Uncertainty Estimation in Neural Networks

TL;DR

This paper introduces Inhibited Softmax, a novel method for uncertainty estimation in neural networks that extends the softmax layer with an extra input to effectively gauge uncertainty without extra computational costs.

Contribution

The paper proposes Inhibited Softmax, a simple extension to softmax layers that accurately estimates uncertainty without additional parameters or multiple passes.

Findings

Performs comparably to computationally intensive methods

Outperforms baseline methods in image recognition

Effective in sentiment analysis tasks

Abstract

We present a new method for uncertainty estimation and out-of-distribution detection in neural networks with softmax output. We extend softmax layer with an additional constant input. The corresponding additional output is able to represent the uncertainty of the network. The proposed method requires neither additional parameters nor multiple forward passes nor input preprocessing nor out-of-distribution datasets. We show that our method performs comparably to more computationally expensive methods and outperforms baselines on our experiments from image recognition and sentiment analysis domains.