Geometric Conditions for the Discrepant Posterior Phenomenon and Connections to Simpson's Paradox

TL;DR

This paper investigates the geometric conditions under which the discrepant posterior phenomenon occurs in Bayesian models, revealing its connection to Simpson's paradox and implications for Bayesian inference and computation.

Contribution

The paper derives geometric conditions for DPP in Gaussian and exponential quadratic likelihood models, linking it to Simpson's paradox and Bayesian computational challenges.

Findings

DPP occurs in a nontrivial space of marginal directions.

A geometric interpretation of DPP is provided.

DPP is more common than previously thought, affecting Bayesian analysis.

Abstract

The discrepant posterior phenomenon (DPP) is a counter-intuitive phenomenon that can frequently occur in a Bayesian analysis of multivariate parameters. It refers to the phenomenon that a parameter estimate based on a posterior is more extreme than both of those inferred based on either the prior or the likelihood alone. Inferential claims that exhibit DPP defy the common intuition that the posterior is a prior-data compromise, and the phenomenon can be surprisingly ubiquitous in well-behaved Bayesian models. In this paper we revisit this phenomenon and, using point estimation as an example, derive conditions under which the DPP occurs in Bayesian models with exponential quadratic likelihoods and conjugate multivariate Gaussian priors. The family of exponential quadratic likelihood models includes Gaussian models and those models with local asymptotic normality property. We provide an intuitive geometric interpretation of the phenomenon and show that there exists a nontrivial space of marginal directions such that the DPP occurs. We further relate the phenomenon to the Simpson's paradox and discover their deep-rooted connection that is associated with marginalization. We also draw connections with Bayesian computational algorithms when difficult geometry exists. Our discovery demonstrates that DPP is more prevalent than previously understood and anticipated. Theoretical results are complemented by numerical illustrations. Scenarios covered in this study have implications for parameterization, sensitivity analysis, and prior choice for Bayesian modeling.