Reconciliating Bayesian and frequentist approaches to robustness against outliers

TL;DR

This paper bridges Bayesian and frequentist robustness methods against outliers by framing M-estimators as maximum likelihood estimators of heavy-tailed models, and proposes a generalized Bayesian approach for improved robustness and comparable results.

Contribution

It introduces a unified framework reconciling Bayesian and frequentist robustness techniques using improper models within generalized Bayesian inference.

Findings

Bayesian and frequentist methods can be aligned through heavy-tailed models.

Generalized Bayesian approach yields similar results to frequentist methods in real data.

Theoretical analysis confirms robustness and proper uncertainty quantification.

Abstract

Heavy-tailed models are used as a way to gain robustness against outliers in Bayesian analyses. In frequentist analyses, M-estimators are often employed. In this paper, the two approaches are tentatively reconciled by considering M-estimators as maximum likelihood estimators of heavy-tailed models. From this perspective, it is realized that a fundamental difference exists as frequentists, contrarily to Bayesians, do not require these heavy-tailed models to be proper. For instance, a popular robust estimator in linear regression, Tukey's biweight M-estimator, does not correspond to a proper heavy-tailed model. Thus, a Bayesian practitioner does not have access to the same range of tools as a frequentist practitioner. It is shown through two real-data linear regression analyses that the former may in consequence obtain significantly different estimation results than the latter, where the difference is due to a more pronounced influence by the outliers in the former case. It is highlighted that a way to give these practitioners access to the same range of tools is for the Bayesian to adopt the generalized Bayesian framework of Bissiri et al. (2016) which allows the use of improper models (Jewson and Rossell, 2022), in combination with proper prior distributions yielding proper generalized posterior distributions. A complete reconciliation of the Bayesian and frequentist approaches to robustness is then achieved. An extensive theoretical study of the generalized Bayesian counterpart of Tukey's biweight M-estimator is provided, which includes a robustness characterization result and a Bernstein--von Mises result, the latter allowing to calibrate the generalized posterior distribution for meaningful uncertainty quantification. After adopting the generalized Bayesian framework, the Bayesian practitioner obtains similar results as the frequentist practitioner in the aforementioned examples.