Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It

TL;DR

This paper demonstrates that Bayesian inference can be inconsistent under model misspecification in linear regression and proposes the Safe Bayesian method, which adjusts the likelihood with a learning rate to improve inference accuracy.

Contribution

It introduces the Safe Bayesian method that learns an optimal likelihood exponent to address Bayesian inconsistency under misspecification.

Findings

Bayesian inference becomes inconsistent with heteroskedastic data.

SafeBayes effectively learns a small learning rate to correct inconsistency.

Results show improved inference accuracy with SafeBayes.

Abstract

We empirically show that Bayesian inference can be inconsistent under misspecification in simple linear regression problems, both in a model averaging/selection and in a Bayesian ridge regression setting. We use the standard linear model, which assumes homoskedasticity, whereas the data are heteroskedastic, and observe that the posterior puts its mass on ever more high-dimensional models as the sample size increases. To remedy the problem, we equip the likelihood in Bayes' theorem with an exponent called the learning rate, and we propose the Safe Bayesian method to learn the learning rate from the data. SafeBayes tends to select small learning rates as soon the standard posterior is not `cumulatively concentrated', and its results on our data are quite encouraging.