Lack of confidence in ABC model choice

TL;DR

This paper critically examines the use of Approximate Bayesian Computation (ABC) for model choice, highlighting theoretical limitations and emphasizing the need for empirical validation due to potential information loss.

Contribution

It provides a theoretical critique of ABC model choice, showing its reliance on insufficient statistics and the resulting unquantified approximation errors.

Findings

ABC model choice depends on unknown information loss

The approximation error may be unrelated to computational effort

Empirical validation is necessary for reliable model choice

Abstract

Approximate Bayesian computation (ABC) have become a essential tool for the analysis of complex stochastic models. Earlier, Grelaud et al. (2009) advocated the use of ABC for Bayesian model choice in the specific case of Gibbs random fields, relying on a inter-model sufficiency property to show that the approximation was legitimate. Having implemented ABC-based model choice in a wide range of phylogenetic models in the DIY-ABC software (Cornuet et al., 2008), we now present theoretical background as to why a generic use of ABC for model choice is ungrounded, since it depends on an unknown amount of information loss induced by the use of insufficient summary statistics. The approximation error of the posterior probabilities of the models under comparison may thus be unrelated with the computational effort spent in running an ABC algorithm. We then conclude that additional empirical verifications of the performances of the ABC procedure as those available in DIYABC are necessary to conduct model choice.