Robust model selection between population growth and multiple merger coalescents

TL;DR

This paper develops a statistical method using the singleton-tail statistic to accurately distinguish between different coalescent models, including those affected by biological confounders like recombination and selection.

Contribution

It introduces a robust, computationally efficient approach for model selection among population coalescent models, extending previous work to account for complex biological factors.

Findings

Singleton-tail statistic effectively distinguishes models with high power.

Cryptic recombination and selection do not reduce test power.

Moderate success in differentiating multiple merger causes, with up to 30% power.

Abstract

We study the effect of biological confounders on the model selection problem between Kingman coalescents with population growth, and Xi-coalescents involving simultaneous multiple mergers. We use a low dimensional, computationally tractable summary statistic, dubbed the singleton-tail statistic, to carry out approximate likelihood ratio tests between these model classes. The singleton-tail statistic has been shown to distinguish between them with high power in the simple setting of neutrally evolving, panmictic populations without recombination. We extend this work by showing that cryptic recombination and selection do not diminish the power of the test, but that misspecifying population structure does. Furthermore, we demonstrate that the singleton-tail statistic can also solve the more challenging model selection problem between multiple mergers due to selective sweeps, and multiple mergers due to high fecundity with moderate power of up to 30%.