Statistical sensitiveness for science

TL;DR

This paper introduces a sensitivity analysis method for estimating the minimum sample size needed to detect a specified effect size in significance testing, addressing limitations of traditional rules-of-thumb and power calculations.

Contribution

It presents a new procedure for sensitivity analysis tailored to Fisher's significance tests, with a comprehensive tutorial and empirical validation through simulations.

Findings

The method accurately estimates minimum sample sizes for desired effect detection.

Sensitivity analysis can be extended to determine minimum effects for given sample sizes.

Empirical simulations support the effectiveness of the proposed procedure.

Abstract

Research often necessitates of samples, yet obtaining large enough samples is not always possible. When it is, the researcher may use one of two methods for deciding upon the required sample size: rules-of-thumb, quick yet uncertain, and estimations for power, mathematically precise yet with the potential to overestimate or underestimate sample sizes when effect sizes are unknown. Misestimated sample sizes have negative repercussions in the form of increased costs, abandoned projects or abandoned publication of non-significant results. Here I describe a procedure for estimating sample sizes adequate for the testing approach which is most common in the behavioural, social, and biomedical sciences, that of tests of significance developed by Fisher. The procedure focuses on a desired minimum effect size for the research at hand and finds the minimum sample size required for capturing such effect size as a statistically significant result. In a similar fashion than power analyses, sensitiveness analyses can also be extended to finding the minimum effect for a given sample size a priori as well as to calculating sensitiveness a posteriori. The article provides a full tutorial for carrying out a sensitiveness analysis, as well as empirical support via simulation