A recipe for irreproducible results

TL;DR

This paper demonstrates how assuming ergodicity in data analysis can lead to false confidence in results, using Brownian motion as a model to show the discrepancy between time averages and ensemble averages.

Contribution

It highlights the risk of misinterpreting data stability by unwarranted ergodic assumptions, emphasizing the importance of ensemble analysis.

Findings

Time averages can be misleading in non-ergodic systems

Ensemble measurements reveal variability hidden in time averages

Assumption of ergodicity can cause false confidence in results

Abstract

Recent studies have shown that many results published in peer-reviewed scientific journals are not reproducible. This raises the following question: why is it so easy to fool myself into believing that a result is reliable when in fact it is not? Using Brownian motion as a toy model, we show how this can happen if ergodicity is assumed where it is unwarranted. A measured value can appear stable when judged over time, although it is not stable across the ensemble: a different result will be obtained each time the experiment is run.