Towards Robust Deep Learning using Entropic Losses

TL;DR

This paper introduces novel entropic loss functions and detection scores to improve out-of-distribution detection and uncertainty estimation in deep learning, backed by theoretical foundations and extensive experiments.

Contribution

It proposes seamless, principled loss functions and detection scores that enhance robustness and reliability in deep neural networks for out-of-distribution detection.

Findings

Achieved state-of-the-art results in out-of-distribution detection

Provided theoretical foundation based on maximum entropy principles

Demonstrated fast and efficient inference with the proposed methods

Abstract

Current deep learning solutions are well known for not informing whether they can reliably classify an example during inference. One of the most effective ways to build more reliable deep learning solutions is to improve their performance in the so-called out-of-distribution detection task, which essentially consists of "know that you do not know" or "know the unknown". In other words, out-of-distribution detection capable systems may reject performing a nonsense classification when submitted to instances of classes on which the neural network was not trained. This thesis tackles the defiant out-of-distribution detection task by proposing novel loss functions and detection scores. Uncertainty estimation is also a crucial auxiliary task in building more robust deep learning systems. Therefore, we also deal with this robustness-related task, which evaluates how realistic the probabilities presented by the deep neural network are. To demonstrate the effectiveness of our approach, in addition to a substantial set of experiments, which includes state-of-the-art results, we use arguments based on the principle of maximum entropy to establish the theoretical foundation of the proposed approaches. Unlike most current methods, our losses and scores are seamless and principled solutions that produce accurate predictions in addition to fast and efficient inference. Moreover, our approaches can be incorporated into current and future projects simply by replacing the loss used to train the deep neural network and computing a rapid score for detection.