Dynamics of Bayesian Updating with Dependent Data and Misspecified Models

TL;DR

This paper establishes conditions for Bayesian posterior convergence with dependent, possibly misspecified data, extending classical results to complex data dependencies and providing insights into model prediction and evolutionary dynamics.

Contribution

It introduces new sufficient conditions for Bayesian posterior convergence under dependent and misspecified models, utilizing information-theoretic properties and capacity control techniques.

Findings

Established posterior convergence conditions for dependent data

Derived large deviations principles for Bayesian posteriors

Discussed advantages of model combination in misspecified settings

Abstract

Much is now known about the consistency of Bayesian updating on infinite-dimensional parameter spaces with independent or Markovian data. Necessary conditions for consistency include the prior putting enough weight on the correct neighborhoods of the data-generating distribution; various sufficient conditions further restrict the prior in ways analogous to capacity control in frequentist nonparametrics. The asymptotics of Bayesian updating with mis-specified models or priors, or non-Markovian data, are far less well explored. Here I establish sufficient conditions for posterior convergence when all hypotheses are wrong, and the data have complex dependencies. The main dynamical assumption is the asymptotic equipartition (Shannon-McMillan-Breiman) property of information theory. This, along with Egorov's Theorem on uniform convergence, lets me build a sieve-like structure for the prior. The main statistical assumption, also a form of capacity control, concerns the compatibility of the prior and the data-generating process, controlling the fluctuations in the log-likelihood when averaged over the sieve-like sets. In addition to posterior convergence, I derive a kind of large deviations principle for the posterior measure, extending in some cases to rates of convergence, and discuss the advantages of predicting using a combination of models known to be wrong. An appendix sketches connections between these results and the replicator dynamics of evolutionary theory.