Do Bayesian Variational Autoencoders Know What They Don't Know?

TL;DR

This paper explores Bayesian inference methods applied to Variational Autoencoders to improve Out-of-Distribution detection, addressing over-confidence issues in deep generative models through empirical evaluation of several Bayesian approaches.

Contribution

It compares three Bayesian inference techniques for VAEs and introduces two simple scores that achieve state-of-the-art OoD detection performance.

Findings

Bayesian approaches reduce over-confidence in OoD detection.

Stochastic Weight Averaging-Gaussian performs best among tested methods.

Proposed scores outperform existing benchmarks in OoD detection.

Abstract

The problem of detecting the Out-of-Distribution (OoD) inputs is of paramount importance for Deep Neural Networks. It has been previously shown that even Deep Generative Models that allow estimating the density of the inputs may not be reliable and often tend to make over-confident predictions for OoDs, assigning to them a higher density than to the in-distribution data. This over-confidence in a single model can be potentially mitigated with Bayesian inference over the model parameters that take into account epistemic uncertainty. This paper investigates three approaches to Bayesian inference: stochastic gradient Markov chain Monte Carlo, Bayes by Backpropagation, and Stochastic Weight Averaging-Gaussian. The inference is implemented over the weights of the deep neural networks that parameterize the likelihood of the Variational Autoencoder. We empirically evaluate the approaches against several benchmarks that are often used for OoD detection: estimation of the marginal likelihood utilizing sampled model ensemble, typicality test, disagreement score, and Watanabe-Akaike Information Criterion. Finally, we introduce two simple scores that demonstrate the state-of-the-art performance.