The fine print on tempered posteriors

TL;DR

This paper critically examines tempered posteriors, revealing that stochasticity often does not improve test accuracy and that the temperature parameter has a complex role beyond fixing prior misspecification.

Contribution

It provides a comprehensive analysis of tempered posteriors, challenging prior assumptions and clarifying the role of temperature and stochasticity in Bayesian models.

Findings

Stochasticity does not generally improve test accuracy in realistic models.

The coldest temperature often yields the best test performance.

Temperature parameter cannot be simply interpreted as fixing prior misspecification.

Abstract

We conduct a detailed investigation of tempered posteriors and uncover a number of crucial and previously undiscussed points. Contrary to previous results, we first show that for realistic models and datasets and the tightly controlled case of the Laplace approximation to the posterior, stochasticity does not in general improve test accuracy. The coldest temperature is often optimal. One might think that Bayesian models with some stochasticity can at least obtain improvements in terms of calibration. However, we show empirically that when gains are obtained this comes at the cost of degradation in test accuracy. We then discuss how targeting Frequentist metrics using Bayesian models provides a simple explanation of the need for a temperature parameter $\lambda$ in the optimization objective. Contrary to prior works, we finally show through a PAC-Bayesian analysis that the temperature $\lambda$ cannot be seen as simply fixing a misspecified prior or likelihood.