Testing for polytomies in phylogenetic species trees using quartet frequencies

TL;DR

This paper introduces a statistical test based on quartet frequencies within the multi-species coalescent model to detect polytomies in species trees, aiding in resolving complex evolutionary relationships.

Contribution

It presents a novel hypothesis test for identifying polytomies in species trees, implemented in the ASTRAL software, considering gene tree discordance.

Findings

Test rejects polytomies for short branches in simulated data.

Test retains true polytomies in most cases.

Applicable to biological datasets to assess resolution confidence.

Abstract

Phylogenetic species trees typically represent the speciation history as a bifurcating tree. Speciation events that simultaneously create more than two descendants, thereby creating polytomies in the phylogeny, are possible. Moreover, the inability to resolve relationships is often shown as a (soft) polytomy. Both types of polytomies have been traditionally studied in the context of gene tree reconstruction from sequence data. However, polytomies in the species tree cannot be detected or ruled out without considering gene tree discordance. In this paper, we describe a statistical test based on properties of the multi-species coalescent model to test the null hypothesis that a branch in an estimated species tree should be replaced by a polytomy. On both simulated and biological datasets, we show that the null hypothesis is rejected for all but the shortest branches, and in most cases, it is retained for true polytomies. The test, available as part of the ASTRAL package, can help systematists decide whether their datasets are sufficient to resolve specific relationships of interest.