Bayesian Restricted Likelihood Methods: Conditioning on Insufficient Statistics in Bayesian Regression

TL;DR

This paper introduces a Bayesian approach that conditions on insufficient statistics to improve robustness against outliers and model misspecification, using a new MCMC algorithm for linear models.

Contribution

It proposes a novel Bayesian method that updates priors with summary statistics instead of full data, enhancing robustness and predictive performance.

Findings

Better predictive accuracy on outlier-contaminated data

Reduced sensitivity to model misspecification

Effective for linear models with various summary statistics

Abstract

Bayesian methods have proven themselves to be successful across a wide range of scientific problems and have many well-documented advantages over competing methods. However, these methods run into difficulties for two major and prevalent classes of problems: handling data sets with outliers and dealing with model misspecification. We outline the drawbacks of previous solutions to both of these problems and propose a new method as an alternative. When working with the new method, the data is summarized through a set of insufficient statistics, targeting inferential quantities of interest, and the prior distribution is updated with the summary statistics rather than the complete data. By careful choice of conditioning statistics, we retain the main benefits of Bayesian methods while reducing the sensitivity of the analysis to features of the data not captured by the conditioning statistics. For reducing sensitivity to outliers, classical robust estimators (e.g., M-estimators) are natural choices for conditioning statistics. A major contribution of this work is the development of a data augmented Markov chain Monte Carlo (MCMC) algorithm for the linear model and a large class of summary statistics. We demonstrate the method on simulated and real data sets containing outliers and subject to model misspecification. Success is manifested in better predictive performance for data points of interest as compared to competing methods.