The Dependence of Routine Bayesian Model Selection Methods on Irrelevant Alternatives

TL;DR

This paper investigates how common Bayesian model selection methods like Bayes Factors and BIC depend on irrelevant model alternatives, revealing they may not preserve preferences under model embeddings, especially in complex contexts like phylogenetics.

Contribution

It demonstrates that standard Bayesian model selection procedures can fail to maintain model preferences when models are embedded into larger classes, highlighting a fundamental limitation.

Findings

Bayesian methods do not always preserve model preferences under embeddings.

Common implementations of Bayes Factors and BIC can be inconsistent.

No natural embedding class exists in certain complex model contexts like phylogenetics.

Abstract

Bayesian methods - either based on Bayes Factors or BIC - are now widely used for model selection. One property that might reasonably be demanded of any model selection method is that if a model ${M}_{1}$ is preferred to a model ${M}_{0}$, when these two models are expressed as members of one model class $\mathbb{M}$, this preference is preserved when they are embedded in a different class $\mathbb{M}'$. However, we illustrate in this paper that with the usual implementation of these common Bayesian procedures this property does not hold true even approximately. We therefore contend that to use these methods it is first necessary for there to exist a "natural" embedding class. We argue that in any context like the one illustrated in our running example of Bayesian model selection of binary phylogenetic trees there is no such embedding.