WAIC, but Why? Generative Ensembles for Robust Anomaly Detection

TL;DR

This paper investigates why WAIC, a Bayesian model selection criterion, performs well in anomaly detection using generative ensembles, despite theoretical limitations of likelihood-based methods.

Contribution

The paper introduces Generative Ensembles to improve density-based OoD detection by estimating epistemic uncertainty, and analyzes the surprising effectiveness of WAIC in practice.

Findings

WAIC performs well despite theoretical limitations.

Generative Ensembles improve robustness of OoD detection.

Likelihood measures alone are insufficient for reliable anomaly detection.

Abstract

Machine learning models encounter Out-of-Distribution (OoD) errors when the data seen at test time are generated from a different stochastic generator than the one used to generate the training data. One proposal to scale OoD detection to high-dimensional data is to learn a tractable likelihood approximation of the training distribution, and use it to reject unlikely inputs. However, likelihood models on natural data are themselves susceptible to OoD errors, and even assign large likelihoods to samples from other datasets. To mitigate this problem, we propose Generative Ensembles, which robustify density-based OoD detection by way of estimating epistemic uncertainty of the likelihood model. We present a puzzling observation in need of an explanation -- although likelihood measures cannot account for the typical set of a distribution, and therefore should not be suitable on their own for OoD detection, WAIC performs surprisingly well in practice.