Investigating maximum likelihood based training of infinite mixtures for uncertainty quantification

TL;DR

This paper explores training infinite mixture models for neural network uncertainty quantification using maximum likelihood, showing improved robustness, uncertainty estimation, and out-of-distribution detection over traditional variational methods.

Contribution

It introduces a novel approach to train infinite mixture models with maximum likelihood, enhancing uncertainty quantification without variational inference.

Findings

Increased predictive variance improves uncertainty estimation.

Enhanced robustness against adversarial attacks.

Higher entropy on out-of-distribution data.

Abstract

Uncertainty quantification in neural networks gained a lot of attention in the past years. The most popular approaches, Bayesian neural networks (BNNs), Monte Carlo dropout, and deep ensembles have one thing in common: they are all based on some kind of mixture model. While the BNNs build infinite mixture models and are derived via variational inference, the latter two build finite mixtures trained with the maximum likelihood method. In this work we investigate the effect of training an infinite mixture distribution with the maximum likelihood method instead of variational inference. We find that the proposed objective leads to stochastic networks with an increased predictive variance, which improves uncertainty based identification of miss-classification and robustness against adversarial attacks in comparison to a standard BNN with equivalent network structure. The new model also displays higher entropy on out-of-distribution data.