Comparing Bayes factors and hierarchical inference for testing general relativity with gravitational waves

TL;DR

This paper compares Bayesian methods for testing general relativity with gravitational wave data, highlighting issues with Bayes factors and advocating hierarchical inference for more reliable conclusions.

Contribution

It demonstrates the limitations of multiplicative Bayes factors and advocates for hierarchical inference as a more robust approach in gravitational wave tests of general relativity.

Findings

Multiplicative Bayes factors can lead to incorrect conclusions due to prior dependence.

Hierarchical inference converges to correct conclusions and is less sensitive to priors.

Toy models show the pitfalls of multiplying Bayes factors in practice.

Abstract

In the context of testing general relativity with gravitational waves, constraints obtained with multiple events are typically combined either through a hierarchical formalism or though a combined multiplicative Bayes factor. We show that the well-known dependence of Bayes factors on the analysis priors in regions of the parameter space without likelihood support can lead to strong confidence in favor of incorrect conclusions when one employs the multiplicative Bayes factor. Bayes factors $\mathcal{O}(1)$ are ambivalent as they depend sensitively on the analysis priors, which are rarely set in a principled way; additionally, combined Bayes factors $>\mathcal{O}(10^3)$ can be obtained in favor of the incorrect conclusion depending on the analysis priors when many $\mathcal{O}(1)$ Bayes factors are multiplied, and specifically when the priors are much wider than the underlying population. The hierarchical analysis that instead infers the ensemble distribution of the individual beyond-general-relativity constraints does not suffer from this problem, and generically converges to favor the correct conclusion. Rather than a naive multiplication, a more reliable Bayes factor can be computed from the hierarchical analysis. We present a number of toy models showing that the practice of multiplying Bayes Factors can lead to incorrect conclusions.