When Do Phylogenetic Mixture Models Mimic Other Phylogenetic Models?

TL;DR

This paper investigates when phylogenetic mixture models can produce data indistinguishable from non-mixture models on different trees, providing conditions under which such mimicking is unlikely or possible, thus guiding their reliable use.

Contribution

It offers new theoretical results delineating when phylogenetic mixture models can mimic other models, especially regarding the number of components and parameter constraints.

Findings

Mixture models with limited components cannot mimic different tree models if branch lengths are constrained.

Over-parameterized local mixtures can mimic other tree models, leading to potential inference issues.

Results inform when mixture models can be reliably used in phylogenetic analysis.

Abstract

Phylogenetic mixture models, in which the sites in sequences undergo different substitution processes along the same or different trees, allow the description of heterogeneous evolutionary processes. As data sets consisting of longer sequences become available, it is important to understand such models, for both theoretical insights and use in statistical analyses. Some recent articles have highlighted disturbing "mimicking" behavior in which a distribution from a mixture model is identical to one arising on a different tree or trees. Other works have indicated such problems are unlikely to occur in practice, as they require very special parameter choices. After surveying some of these works on mixture models, we give several new results. In general, if the number of components in a generating mixture is not too large and we disallow zero or infinite branch lengths, then it cannot mimic the behavior of a non-mixture on a different tree. On the other hand, if the mixture model is locally over-parameterized, it is possible for a phylogenetic mixture model to mimic distributions of another tree model. Though theoretical questions remain, these sorts of results can serve as a guide to when the use of mixture models in either ML or Bayesian frameworks is likely to lead to statistically consistent inference, and when mimicking due to heterogeneity should be considered a realistic possibility.