The shape of the one-dimensional phylogenetic likelihood function

TL;DR

This paper introduces a mathematical framework to analyze the shapes of one-dimensional phylogenetic likelihood functions, revealing conditions for unimodality and demonstrating potential for multiple maxima in complex models.

Contribution

It provides a novel algebraic framework for characterizing likelihood function shapes and identifies models where multiple stationary points can occur, challenging common assumptions.

Findings

Likelihood functions are unimodal under simple models like JC69 and Felsenstein 1981.

The Kimura 2-parameter model can produce likelihood functions with multiple stationary points.

The space of likelihood functions under the Kimura 2-parameter model is dense in all non-negative continuous functions.

Abstract

By fixing all parameters in a phylogenetic likelihood model except for one branch length, one obtains a one-dimensional likelihood function. In this work, we introduce a mathematical framework to characterize the shapes of such one-dimensional phylogenetic likelihood functions. This framework is based on analyses of algebraic structures on the space of all frequency patterns with respect to a polynomial representation of the likelihood functions. Using this framework, we provide conditions under which the one-dimensional phylogenetic likelihood functions are guaranteed to have at most one stationary point, and this point is the maximum likelihood branch length. These conditions are satisfied by common simple models including all binary models, the Jukes-Cantor model and the Felsenstein 1981 model. We then prove that for the simplest model that does not satisfy our conditions, namely, the Kimura 2-parameter model, the one-dimensional likelihood functions may have multiple stationary points. As a proof of concept, we construct a non-degenerate example in which the phylogenetic likelihood function has two local maxima and a local minimum. To construct such examples, we derive a general method of constructing a tree and sequence data with a specified frequency pattern at the root. We then extend the result to prove that the space of all rescaled and translated one-dimensional phylogenetic likelihood functions under the Kimura 2-parameter model is dense in the space of all non-negative continuous functions on $[0, \infty)$ with finite limits. These results indicate that one-dimensional likelihood functions under advanced evolutionary models can be more complex than it is typically assumed by phylogenetic inference algorithms; however, these complexities can be effectively captured by the Kimura 2-parameter model.