Statistical Inference for Bayesian Risk Minimization via Exponentially Tilted Empirical Likelihood

TL;DR

This paper introduces a robust Bayesian inference method using exponentially tilted empirical likelihoods, providing well-calibrated credible regions even under model misspecification, with practical MCMC implementation and superior accuracy.

Contribution

It proposes a novel Bayesian approach replacing likelihoods with empirical likelihoods based on risk minimization conditions, enhancing robustness and calibration.

Findings

Method yields asymptotically normal posteriors centered at empirical risk minimizers.

Provides credible regions with valid frequentist coverage.

Outperforms existing methods in numerical experiments.

Abstract

The celebrated Bernstein von-Mises theorem ensures that credible regions from Bayesian posterior are well-calibrated when the model is correctly-specified, in the frequentist sense that their coverage probabilities tend to the nominal values as data accrue. However, this conventional Bayesian framework is known to lack robustness when the model is misspecified or only partly specified, such as in quantile regression, risk minimization based supervised/unsupervised learning and robust estimation. To overcome this difficulty, we propose a new Bayesian inferential approach that substitutes the (misspecified or partly specified) likelihoods with proper exponentially tilted empirical likelihoods plus a regularization term. Our surrogate empirical likelihood is carefully constructed by using the first order optimality condition of the empirical risk minimization as the moment condition. We show that the Bayesian posterior obtained by combining this surrogate empirical likelihood and the prior is asymptotically close to a normal distribution centering at the empirical risk minimizer with covariance matrix taking an appropriate sandwiched form. Consequently, the resulting Bayesian credible regions are automatically calibrated to deliver valid uncertainty quantification. Computationally, the proposed method can be easily implemented by Markov Chain Monte Carlo sampling algorithms. Our numerical results show that the proposed method tends to be more accurate than existing state-of-the-art competitors.