VIB is Half Bayes

TL;DR

This paper interprets the Variational Information Bottleneck as a hybrid approach that balances Bayesian and empirical methods, reducing sampling risks with less computational effort.

Contribution

It offers a new perspective on VIB, framing it as a compromise between Bayesian and empirical objectives in discriminative tasks.

Findings

VIB minimizes risks related to finite Y sampling.

VIB combines benefits of Bayesian and empirical approaches.

Provides a computationally efficient alternative to full Bayesian methods.

Abstract

In discriminative settings such as regression and classification there are two random variables at play, the inputs X and the targets Y. Here, we demonstrate that the Variational Information Bottleneck can be viewed as a compromise between fully empirical and fully Bayesian objectives, attempting to minimize the risks due to finite sampling of Y only. We argue that this approach provides some of the benefits of Bayes while requiring only some of the work.