Fast coalescent-based computation of local branch support from quartet frequencies

TL;DR

This paper introduces a fast algorithm for calculating local branch support and length in species trees using quartet frequencies, improving accuracy and efficiency over existing methods, especially with low gene tree error.

Contribution

The paper presents a novel, efficient algorithm for computing quartet-based support and branch lengths within the multi-species coalescent framework, integrated into ASTRAL.

Findings

High precision of local posterior probability in simulated and biological data

Improved recall over multi-locus bootstrapping methods

Accurate branch length estimation with low gene tree error

Abstract

Species tree reconstruction is complicated by effects of Incomplete Lineage Sorting (ILS), commonly modeled by the multi-species coalescent model. While there has been substantial progress in developing methods that estimate a species tree given a collection of gene trees, less attention has been paid to fast and accurate methods of quantifying support. In this paper, we propose a fast algorithm to compute quartet-based support for each branch of a given species tree with regard to a given set of gene trees. We then show how the quartet support can be used in the context of the multi-species coalescent model to compute i) the local posterior probability that the branch is in the species tree and ii) the length of the branch in coalescent units. We evaluate the precision and recall of the local posterior probability on a wide set of simulated and biological data, and show that it has very high precision and improved recall compared to multi-locus bootstrapping. The estimated branch lengths are highly accurate when gene trees have little error, but are underestimated when gene tree estimation error increases. Computation of both branch length and local posterior probability is implemented as a new feature in ASTRAL.