Identifiability and Falsifiability: Two Challenges for Bayesian Model Expansion

TL;DR

This paper explores the challenges of parameter identifiability and falsifiability in Bayesian model expansion, introducing information-theoretic measures to analyze trade-offs and practical implications for model design.

Contribution

It develops mutual information proxies for identifiability and falsifiability, and proves a trade-off relation in complex model expansions, with practical guidelines.

Findings

Complex model expansions force a trade-off between identifiability and falsifiability.

Lower mutual information indicates higher risk of poor inference and weak model checks.

Constraining priors can mitigate negative effects of increased model complexity.

Abstract

We study the identifiability of parameters and falsifiability of predictions under the process of model expansion in a Bayesian setting. Identifiability is represented by the closeness of the posterior to the prior distribution and falsifiability by the power of posterior predictive tests against alternatives. To study these two concepts formally, we develop information-theoretic proxies, which we term the identifiability and falsifiability mutual information. We argue that these are useful indicators, with lower values indicating a risk of poor parameter inference and underpowered model checks, respectively. Our main result establishes that a sufficiently complex expansion of a base statistical model forces a trade-off between these two mutual information quantities -- at least one of the two must decrease relative to the base model. We illustrate our result in three worked examples and extract implications for model expansion in practice. In particular, we show as an implication of our result that the negative impacts of model expansion can be limited by offsetting complexity in the likelihood with sufficiently constraining prior distributions.