Ancestral state reconstruction with large numbers of sequences and edge-length estimation

TL;DR

This paper investigates the accuracy and consistency of likelihood-based ancestral state reconstruction methods with many sequences and uncertain edge lengths, revealing conditions where these methods succeed or fail.

Contribution

It provides theoretical analysis of maximum likelihood and empirical Bayes estimators' consistency under large datasets and unknown edge lengths, highlighting limitations and proposing a simple alternative estimator.

Findings

Likelihood-based methods are consistent under symmetric models.

Likelihood methods can be inconsistent under non-symmetric models.

A simple consistent estimator exists for non-symmetric models.

Abstract

Likelihood-based methods are widely considered the best approaches for reconstructing ancestral states. Although much effort has been made to study properties of these methods, previous works often assume that both the tree topology and edge lengths are known. In some scenarios the tree topology might be reasonably well known for the taxa under study. When sequence length is much smaller than the number of species, however, edge lengths are not likely to be accurately estimated. We study the consistency of the maximum likelihood and empirical Bayes estimators of ancestral state of discrete traits in such settings under a star tree. We prove that the likelihood-based reconstruction is consistent under symmetric models but can be inconsistent under non-symmetric models. We show, however, that a simple consistent estimator for the ancestral states is available under non-symmetric models. The results illustrate that likelihood methods can unexpectedly have undesirable properties as the number of sequences considered get very large. Broader implications of the results are discussed.