How to make any method "fail": BAMM at the kangaroo court of false equivalency

TL;DR

This paper critiques a previous study on BAMM, a phylogenetic analysis tool, revealing that the prior comparison was flawed due to unequal information and demonstrating BAMM's superior performance when properly evaluated.

Contribution

The paper identifies and corrects conceptual errors in a prior comparison of BAMM with simpler estimators, emphasizing the importance of fair evaluation conditions.

Findings

BAMM outperforms MS estimators when given equal prior information.

The previous study's conclusions were invalid due to flawed experimental design.

Statistical errors in method comparisons are common across various biological data analysis frameworks.

Abstract

The software program BAMM has been widely used to study the dynamics of speciation, extinction, and phenotypic evolution on phylogenetic trees. The program implements a model-based clustering algorithm to identify clades that share common macroevolutionary rate dynamics and to estimate rate parameters. A recent simulation study published in Evolution (2017) by Meyer and Wiens (M&W) claimed that (i) simple ("MS") estimators of diversification rates perform much better than BAMM, and (ii) evolutionary rates inferred with BAMM are weakly correlated with the true rates in the generating model. I demonstrate that their assessment suffers from two major conceptual errors that invalidate both primary conclusions. These statistical considerations are not specific to BAMM and apply to all methods for estimating parameters from empirical data where the true grouping structure of the data is unknown. First, M&W\'s comparisons between BAMM and MS estimators suffer from false equivalency because the MS estimators are given perfect prior knowledge of the locations of rate shifts on the simulated phylogenies. BAMM is given no such information and must simultaneously estimate the number and location of rate shifts from the data, thus resulting in a massive degrees-of-freedom advantage for the MS estimators.When both methods are given equivalent information, BAMM dramatically outperforms the MS estimators. Second, M&W\'s experimental design is unable to assess parameter reliability because their analyses conflate small effect sizes across treatment groups with error in parameter estimates. Nearly all model-based frameworks for partitioning data are susceptible to the statistical mistakes in M&W, including popular clustering algorithms in population genetics, phylogenetics, and comparative methods.