Clades, clans and reciprocal monophyly under neutral evolutionary models

TL;DR

This paper derives exact formulas for the probability of observing monophyletic groups in phylogenetic trees under neutral models, aiding statistical significance assessments in evolutionary studies.

Contribution

It extends previous work by providing exact probability formulas for clades and clans under the Yule and coalescent models, linking these models' predictions.

Findings

Exact formulas for clade probabilities in rooted trees.

Exact formulas for clan probabilities in unrooted trees.

Results applicable for statistical testing of monophyly.

Abstract

The Yule model and the coalescent model are two neutral stochastic models for generating trees in phylogenetics and population genetics, respectively. Although these models are quite different, they lead to identical distributions concerning the probability that pre-specified groups of taxa form monophyletic groups (clades) in the tree. We extend earlier work to derive exact formulae for the probability of finding one or more groups of taxa as clades in a rooted tree, or as `clans' in an unrooted tree. Our findings are relevant for calculating the statistical significance of observed monophyly and reciprocal monophyly in phylogenetics.