Master's Thesis: Out-of-distribution Detection with Energy-based Models

TL;DR

This thesis explores the use of Energy-based Models for out-of-distribution detection, proposing the Energy-Prior Network to improve uncertainty estimation and detection capabilities without relying on OOD datasets.

Contribution

It introduces the Energy-Prior Network (EPN), a novel method that estimates uncertainties in EBMs and enhances OOD detection, dataset shift, and adversarial example identification.

Findings

EBMs do not inherently outperform other density estimators in OOD detection.

Supervision, dimensionality reduction, and architectural changes impact EBM performance.

EPN effectively detects OOD inputs, dataset shifts, and adversarial examples.

Abstract

Today, deep learning is increasingly applied in security-critical situations such as autonomous driving and medical diagnosis. Despite its success, the behavior and robustness of deep networks are not fully understood yet, posing a significant risk. In particular, researchers recently found that neural networks are overly confident in their predictions, even on data they have never seen before. To tackle this issue, one can differentiate two approaches in the literature. One accounts for uncertainty in the predictions, while the second estimates the underlying density of the training data to decide whether a given input is close to the training data, and thus the network is able to perform as expected.In this thesis, we investigate the capabilities of EBMs at the task of fitting the training data distribution to perform detection of out-of-distribution (OOD) inputs. We find that on most datasets, EBMs do not inherently outperform other density estimators at detecting OOD data despite their flexibility. Thus, we additionally investigate the effects of supervision, dimensionality reduction, and architectural modifications on the performance of EBMs. Further, we propose Energy-Prior Network (EPN) which enables estimation of various uncertainties within an EBM for classification, bridging the gap between two approaches for tackling the OOD detection problem. We identify a connection between the concentration parameters of the Dirichlet distribution and the joint energy in an EBM. Additionally, this allows optimization without a held-out OOD dataset, which might not be available or costly to collect in some applications. Finally, we empirically demonstrate that Energy-Prior Network (EPN) is able to detect OOD inputs, datasets shifts, and adversarial examples. Theoretically, EPN offers favorable properties for the asymptotic case when inputs are far from the training data.