Species tree inference from gene splits by Unrooted STAR methods

TL;DR

This paper proves the statistical consistency of the NJ_st method for species tree inference from gene splits across arbitrarily large trees, extending prior results and clarifying its theoretical foundations.

Contribution

It establishes the statistical consistency of NJ_st for large species trees and links it to a generalized STAR method, enhancing theoretical understanding.

Findings

Proves consistency of NJ_st for any size species tree.

Shows NJ_st uses only gene tree split distributions.

Discusses handling multiple samples per taxon.

Abstract

The $\text{NJ}_{st}$ method was proposed by Liu and Yu to infer a species tree topology from unrooted topological gene trees. While its statistical consistency under the multispecies coalescent model was established only for a 4-taxon tree, simulations demonstrated its good performance on gene trees inferred from sequences for many taxa. Here we prove the statistical consistency of the method for an arbitrarily large species tree. Our approach connects $\text{NJ}_{st}$ to a generalization of the STAR method of Liu, Pearl and Edwards, and a previous theoretical analysis of it. We further show $\text{NJ}_{st}$ utilizes only the distribution of splits in the gene trees, and not their individual topologies. Finally, we discuss how multiple samples per taxon per gene should be handled for statistical consistency.