The power of Bayesian evidence in astronomy

TL;DR

This paper examines the Bayesian evidence ratio in astronomy, highlighting its variability and limitations as a model selection tool, and discusses when it can be effectively used for preliminary analysis.

Contribution

It provides an analysis of the statistical properties of the Bayesian evidence ratio, emphasizing its noise and dependence on data quality, priors, and model set.

Findings

Evidence ratio is a noisy statistic and not always reliable for model decision-making.

The performance of evidence-based tests depends on signal-to-noise ratio, priors, and thresholds.

Simple models can help assess the usefulness of the evidence ratio before complex analysis.

Abstract

We discuss the use of the Bayesian evidence ratio, or Bayes factor, for model selection in astronomy. We treat the evidence ratio as a statistic and investigate its distribution over an ensemble of experiments, considering both simple analytical examples and some more realistic cases, which require numerical simulation. We find that the evidence ratio is a noisy statistic, and thus it may not be sensible to decide to accept or reject a model based solely on whether the evidence ratio reaches some threshold value. The odds suggested by the evidence ratio bear no obvious relationship to the power or Type I error rate of a test based on the evidence ratio. The general performance of such tests is strongly affected by the signal to noise ratio in the data, the assumed priors, and the threshold in the evidence ratio that is taken as `decisive'. The comprehensiveness of the model suite under consideration is also very important. The usefulness of the evidence ratio approach in a given problem can be assessed in advance of the experiment, using simple models and numerical approximations. In many cases, this approach can be as informative as a much more costly full-scale Bayesian analysis of a complex problem.