Early Warning Signals and the Prosecutor's Fallacy

TL;DR

This paper investigates the bias introduced by the Prosecutor's Fallacy in early warning signals for critical transitions, demonstrating that common statistics can produce false positives and proposing a less biased model-based alternative.

Contribution

It identifies a bias in early warning signal analysis caused by conditional selection and introduces a model-based method that reduces this bias.

Findings

Common warning signal statistics have high false positive rates due to bias.

Simulated systems show that chance transitions can lead to misleading early warning signals.

Model-based approaches are less susceptible to the Prosecutor's Fallacy bias.

Abstract

Early warning signals have been proposed to forecast the possibility of a critical transition, such as the eutrophication of a lake, the collapse of a coral reef, or the end of a glacial period. Because such transitions often unfold on temporal and spatial scales that can be difficult to approach by experimental manipulation, research has often relied on historical observations as a source of natural experiments. Here we examine a critical difference between selecting systems for study based on the fact that we have observed a critical transition and those systems for which we wish to forecast the approach of a transition. This difference arises by conditionally selecting systems known to experience a transition of some sort and failing to account for the bias this introduces -- a statistical error often known as the Prosecutor's Fallacy. By analysing simulated systems that have experienced transitions purely by chance, we reveal an elevated rate of false positives in common warning signal statistics. We further demonstrate a model-based approach that is less subject to this bias than these more commonly used summary statistics. We note that experimental studies with replicates avoid this pitfall entirely.