How trustworthy is your tree? Bayesian phylogenetic effective sample size through the lens of Monte Carlo error

TL;DR

This paper introduces tree-specific effective sample size measures to evaluate the Monte Carlo error in Bayesian phylogenetic inference, highlighting the importance of assessing both within-chain mixing and between-chain convergence.

Contribution

It proposes new tree ESS measures tailored for phylogenetic summaries and provides visualization tools to better assess MCMC convergence and error in phylogenetic analyses.

Findings

Tree ESS measures effectively capture Monte Carlo error in phylogenetic summaries.

Common workflows often underestimate the Monte Carlo error in Bayesian phylogenetics.

Visualization tools improve comparison and convergence assessment across multiple MCMC runs.

Abstract

Bayesian inference is a popular and widely-used approach to infer phylogenies (evolutionary trees). However, despite decades of widespread application, it remains difficult to judge how well a given Bayesian Markov chain Monte Carlo (MCMC) run explores the space of phylogenetic trees. In this paper, we investigate the Monte Carlo error of phylogenies, focusing on high-dimensional summaries of the posterior distribution, including variability in estimated edge/branch (known in phylogenetics as "split") probabilities and tree probabilities, and variability in the estimated summary tree. Specifically, we ask if there is any measure of effective sample size (ESS) applicable to phylogenetic trees which is capable of capturing the Monte Carlo error of these three summary measures. We find that there are some ESS measures capable of capturing the error inherent in using MCMC samples to approximate the posterior distributions on phylogenies. We term these tree ESS measures, and identify a set of three which are useful in practice for assessing the Monte Carlo error. Lastly, we present visualization tools that can improve comparisons between multiple independent MCMC runs by accounting for the Monte Carlo error present in each chain. Our results indicate that common post-MCMC workflows are insufficient to capture the inherent Monte Carlo error of the tree, and highlight the need for both within-chain mixing and between-chain convergence assessments.