The space of phylogenetic mixtures for equivariant models

TL;DR

This paper characterizes the space of phylogenetic mixtures under equivariant models using linear equations, providing a new tool for model selection and insights into identifiability issues.

Contribution

It proves that distributions satisfying certain linear equations correspond exactly to mixtures of phylogenetic trees under equivariant models and provides a complete set of defining equations.

Findings

Linear equations characterize phylogenetic mixture distributions.

The set of equations defines the space of mixtures for any number of taxa.

Results aid in model selection and understanding identifiability.

Abstract

The selection of the most suitable evolutionary model to analyze the given molecular data is usually left to biologist's choice. In his famous book, J Felsenstein suggested that certain linear equations satisfied by the expected probabilities of patterns observed at the leaves of a phylogenetic tree could be used for model selection. It remained open the question regarding whether these equations were enough for characterizing the evolutionary model. Here we prove that, for equivariant models of evolution, the space of distributions satisfying these linear equations coincides with the space of distributions arising from mixtures of trees on a set of taxa. In other words, we prove that an alignment is produced from a mixture of phylogenetic trees under an equivariant evolutionary model if and only if its distribution of column patterns satisfies the linear equations mentioned above. Moreover, for each equivariant model and for any number of taxa, we provide a set of linearly independent equations defining this space of phylogenetic mixtures. This is a powerful tool that has already been successfully used in model selection. We also use the results obtained to study identifiability issues for phylogenetic mixtures.