Is your phylogeny informative? Measuring the power of comparative methods

TL;DR

This paper introduces a Monte Carlo-based approach to measure the statistical power of phylogenetic comparative methods, addressing limitations of traditional model choice criteria and emphasizing the importance of data structure in inference accuracy.

Contribution

It presents a novel Monte Carlo method for assessing the power of comparative methods, improving error estimation and providing confidence intervals, with an open-source implementation.

Findings

Information criteria can have high error rates in model choice

Power depends on data structure and number of taxa

Monte Carlo method reduces errors and quantifies confidence

Abstract

Phylogenetic comparative methods may fail to produce meaningful results when either the underlying model is inappropriate or the data contain insufficient information to inform the inference. The ability to measure the statistical power of these methods has become crucial to ensure that data quantity keeps pace with growing model complexity. Through simulations, we show that commonly applied model choice methods based on information criteria can have remarkably high error rates; this can be a problem because methods to estimate the uncertainty or power are not widely known or applied. Furthermore, the power of comparative methods can depend significantly on the structure of the data. We describe a Monte Carlo based method which addresses both of these challenges, and show how this approach both quantifies and substantially reduces errors relative to information criteria. The method also produces meaningful confidence intervals for model parameters. We illustrate how the power to distinguish different models, such as varying levels of selection, varies both with number of taxa and structure of the phylogeny. We provide an open-source implementation in the pmc ("Phylogenetic Monte Carlo") package for the R programming language. We hope such power analysis becomes a routine part of model comparison in comparative methods.