Calibrating Bayesian Inference

TL;DR

This paper addresses the limitations of standard Bayesian inference by proposing a calibration method for credible regions to ensure frequentist validity, demonstrated through simulations and a real-data example.

Contribution

It introduces a novel calibration procedure for Bayesian credible regions that guarantees frequentist validity regardless of prior-data mismatch.

Findings

Uncalibrated Bayesian inference can be misleading when priors mismatch true data-generating process.

Calibrated Bayesian credible regions maintain validity across different scenarios.

The proposed stochastic approximation algorithm effectively calibrates Bayesian inference.

Abstract

Bayesian statistics has gained popularity in psychological research due to its intuitive uncertainty quantification and convenient information-updating rules. In many applications, however, prior distributions are introduced merely as instruments to facilitate computation, rather than as representations of genuine subjective belief. Consequently, relying on standard Bayesian justifications for inferential procedures becomes conceptually ungrounded. In this paper, we recommend evaluating finite-sample performance over repeated sampling of data and parameters as an alternative justification for "pragmatic Bayes." We demonstrate a key vulnerability in the usual posterior-based inference: when analysts' chosen prior distribution mismatches the true parameter-generating process, Bayesian inference can be misleading. Given that this true process is rarely known in practice, we propose a safer alternative: calibrating Bayesian credible regions to achieve frequentist validity. This latter criterion is stronger and guarantees validity of Bayesian inference regardless of the underlying parameter-generating mechanism. To solve the calibration problem in practice, we propose a novel stochastic approximation algorithm. A Monte Carlo experiment is conducted and reported, in which we observe that uncalibrated Bayesian inference can be liberal under certain parameter-generating scenarios, whereas our calibrated solution consistently maintain validity. We also illustrate the proposed calibration procedure using a real-data example involving location-scale regression.