Estimating Species Trees from Quartet Gene Tree Distributions under the Coalescent Model

TL;DR

This paper introduces 'quartet neighbor joining', a new statistically consistent method for estimating unrooted species trees from gene tree distributions on all four-taxon subsets, demonstrated through simulations and real data.

Contribution

The paper presents a novel quartet-NJ algorithm for species tree inference that is similar to neighbor joining and proven to be statistically consistent under the coalescent model.

Findings

Quartet-NJ accurately reconstructs species trees in simulations.

The method is applicable to real biological data from prokaryotes.

Quartet-NJ outperforms some existing methods in accuracy.

Abstract

In this article we propose a new method, which we name 'quartet neighbor joining', or 'quartet-NJ', to infer an unrooted species tree on a given set of taxa T from empirical distributions of unrooted quartet gene trees on all four-taxon subsets of T. In particular, quartet-NJ can be used to estimate a species tree on T from distributions of gene trees on T. The quartet-NJ algorithm is conceptually very similar to classical neighbor joining, and its statistical consistency under the multispecies coalescent model is proved by a variant of the classical 'cherry picking'-theorem. In order to demonstrate the suitability of quartet-NJ, coalescent processes on two different species trees (on five resp. nine taxa) were simulated, and quartet-NJ was applied to the simulated gene tree distributions. Further, quartet-NJ was applied to quartet distributions obtained from multiple sequence alignments of 28 proteins of nine prokaryotes.