From Thomas Bayes to Big Data: On the feasibility of being a subjective Bayesian

TL;DR

This paper critically examines the feasibility of applying the subjective Bayesian paradigm in ultra-high-dimensional models, arguing that natural priors often fail to accurately represent beliefs and that many Bayesian procedures are essentially frequentist methods in disguise.

Contribution

The paper provides a rigorous mathematical critique of the subjective Bayesian approach, highlighting its limitations in high-dimensional settings and clarifying misconceptions about its foundational validity.

Findings

Natural priors often fail in high-dimensional models

Many Bayesian procedures are essentially frequentist methods

Subjective Bayesian paradigm faces fundamental limitations

Abstract

We argue that the Bayesian paradigm, of a prior which represents the beliefs of the statistician before observing the data, is not feasible in ultra-high-dimensional models. We claim that natural priors that represent the a priori beliefs fail in unpredictable ways under values of the parameters that cannot be honestly ignored. We do not claim that the frequentist estimators we present cannot be mimicked by Bayesian procedures, but that these Bayesian procedures do not represent beliefs. They were created with the frequentist analysis in mind, and in most cases, they cannot represent a consistent set of beliefs about the parameters (for example, since they depend on the loss function, the particular functional of interest, and not only on the a priori knowledge, different priors should be used for different analyses of the same data set). In a way, these are frequentist procedures using a Bayesian technique. The paper presents different examples where the subjective point of view fails. It is argued that the arguments based on Wald's and Savage's seminal works are not relevant to the validity of the subjective Bayesian paradigm. The discussion tries to deal with the fundamentals, but the argument is based on a firm mathematical proofs.