Data augmentation in Bayesian neural networks and the cold posterior effect

TL;DR

This paper develops principled Bayesian neural network models with data augmentation, providing exact likelihood computations and bounds, and investigates the persistent cold posterior effect.

Contribution

It introduces a finite orbit setting for exact likelihood computation and demonstrates that the cold posterior effect persists even in these rigorous models.

Findings

Likelihoods can be computed exactly in the finite orbit setting.

Multi-sample bounds are established in the full orbit setting.

The cold posterior effect remains even with principled data augmentation models.

Abstract

Bayesian neural networks that incorporate data augmentation implicitly use a ``randomly perturbed log-likelihood [which] does not have a clean interpretation as a valid likelihood function'' (Izmailov et al. 2021). Here, we provide several approaches to developing principled Bayesian neural networks incorporating data augmentation. We introduce a ``finite orbit'' setting which allows likelihoods to be computed exactly, and give tight multi-sample bounds in the more usual ``full orbit'' setting. These models cast light on the origin of the cold posterior effect. In particular, we find that the cold posterior effect persists even in these principled models incorporating data augmentation. This suggests that the cold posterior effect cannot be dismissed as an artifact of data augmentation using incorrect likelihoods.