If your P value looks too good to be true, it probably is: Communicating reproducibility and variability in cell biology

TL;DR

This paper highlights the common misuse of P values in cell biology, emphasizing that proper interpretation requires considering experimental replicates rather than individual cell data, and provides guidance for clearer communication of reproducibility.

Contribution

It clarifies the correct statistical approach for reporting P values in cell biology and offers practical tutorials for better data visualization of variability and reproducibility.

Findings

Misuse of P values based on cell counts is widespread.

Proper interpretation of P values depends on experimental replicates.

Guidelines for effective data visualization are provided.

Abstract

The cell biology literature is littered with erroneously tiny P values, often the result of evaluating individual cells as independent samples. Because readers use P values and error bars to infer whether a reported difference would likely recur if the experiment were repeated, the sample size N used for statistical tests should actually be the number of times an experiment is performed, not the number of cells (or subcellular structures) analyzed across all experiments. P values calculated using the number of cells do not reflect the reproducibility of the result and are thus highly misleading. To help authors avoid this mistake, we provide examples and practical tutorials for creating figures that communicate both the cell-level variability and the experimental reproducibility.