Performance of Bayesian linear regression in a model with mismatch

TL;DR

This paper analyzes the performance of Bayesian linear regression with model mismatch in high dimensions, revealing that in quadratic loss, it matches ridge regression and is unaffected by a hyperparameter.

Contribution

It provides a straightforward high-dimensional analysis of Bayesian linear regression under model mismatch, connecting it to the Gardner spin glass model.

Findings

Bayesian estimator performance is independent of the temperature hyperparameter.

In quadratic loss, Bayesian regression matches ridge regression.

The analysis uses a leave-one-out approach for mean-square error characterization.

Abstract

In this paper we analyze, for a model of linear regression with gaussian covariates, the performance of a Bayesian estimator given by the mean of a log-concave posterior distribution with gaussian prior, in the high-dimensional limit where the number of samples and the covariates' dimension are large and proportional. Although the high-dimensional analysis of Bayesian estimators has been previously studied for Bayesian-optimal linear regression where the correct posterior is used for inference, much less is known when there is a mismatch. Here we consider a model in which the responses are corrupted by gaussian noise and are known to be generated as linear combinations of the covariates, but the distributions of the ground-truth regression coefficients and of the noise are unknown. This regression task can be rephrased as a statistical mechanics model known as the Gardner spin glass, an analogy which we exploit. Using a leave-one-out approach we characterize the mean-square error for the regression coefficients. We also derive the log-normalizing constant of the posterior. Similar models have been studied by Shcherbina and Tirozzi and by Talagrand, but our arguments are much more straightforward. An interesting consequence of our analysis is that in the quadratic loss case, the performance of the Bayesian estimator is independent of a global "temperature" hyperparameter and matches the ridge estimator: sampling and optimizing are equally good.