The Least Difference in Means: A Statistic for Effect Size Strength and Practical Significance

TL;DR

This paper introduces the least difference in means ($oldsymbol{ ext{δ}_L}$) as a new statistic to quantify and compare effect size strength, aiding in prioritizing scientifically meaningful results with practical significance.

Contribution

It proposes $ ext{δ}_L$ as a novel, conservative effect size measure and hypothesis test, enabling comparison across related results without recalculations.

Findings

$ ext{δ}_L$ outperforms existing statistics in identifying meaningful effects.

The relative $ ext{δ}_L$ allows comparison of effect strength across experiments.

Demonstrated utility of $ ext{δ}_L$ in real data for research prioritization.

Abstract

With limited resources, scientific inquiries must be prioritized for further study, funding, and translation based on their practical significance: whether the effect size is large enough to be meaningful in the real world. Doing so must evaluate a result's effect strength, defined as a conservative assessment of practical significance. We propose the least difference in means ($\delta_L$) as a two-sample statistic that can quantify effect strength and perform a hypothesis test to determine if a result has a meaningful effect size. To facilitate consensus, $\delta_L$ allows scientists to compare effect strength between related results and choose different thresholds for hypothesis testing without recalculation. Both $\delta_L$ and the relative $\delta_L$ outperform other candidate statistics in identifying results with higher effect strength. We use real data to demonstrate how the relative $\delta_L$ compares effect strength across broadly related experiments. The relative $\delta_L$ can prioritize research based on the strength of their results.