The Most Difference in Means: A Statistic for the Strength of Null and Near-Zero Results

TL;DR

This paper introduces the most difference in means ($oldsymbol{ ext{δ}_M}$), a new statistic to quantify and test the strength of null or near-zero effect results, aiding interpretation and comparison across studies.

Contribution

The paper proposes $ ext{δ}_M$, a novel two-sample statistic that quantifies null strength and enables hypothesis testing for negligible effect sizes, improving interpretation of null results.

Findings

$ ext{δ}_M$ outperforms other statistics in comparing null strength.

The relative $ ext{δ}_M$ allows comparison across different treatments and methods.

Reporting relative $ ext{δ}_M$ may encourage publication of null results.

Abstract

Statistical insignificance does not suggest the absence of effect, yet scientists must often use null results as evidence of negligible (near-zero) effect size to falsify scientific hypotheses. Doing so must assess a result's null strength, defined as the evidence for a negligible effect size. Such an assessment would differentiate strong null results that suggest a negligible effect size from weak null results that suggest a broad range of potential effect sizes. We propose the most difference in means ($\delta_M$) as a two-sample statistic that can both quantify null strength and perform a hypothesis test for negligible effect size. To facilitate consensus when interpreting results, our statistic allows scientists to conclude that a result has negligible effect size using different thresholds with no recalculation required. To assist with selecting a threshold, $\delta_M$ can also compare null strength between related results. Both $\delta_M$ and the relative form of $\delta_M$ outperform other candidate statistics in comparing null strength. We compile broadly related results and use the relative $\delta_M$ to compare null strength across different treatments, measurement methods, and experiment models. Reporting the relative $\delta_M$ may provide a technical solution to the file drawer problem by encouraging the publication of null and near-zero results.