A Proposed Hybrid Effect Size Plus $p$-Value Criterion: A Comment on Goodman et al. (2019)

TL;DR

This paper verifies and discusses a hybrid effect size and p-value criterion for statistical significance, confirming its low error rates and exploring its limitations in controlling false discovery rates through simulation and software tools.

Contribution

It replicates Goodman et al.'s simulation results and introduces an additional decision method to control false positive rates, providing accessible software for customization.

Findings

Hybrid criterion has low error rates in checked settings.

False discovery rate is difficult to control with the hybrid method.

Software implementation is available for replication and customization.

Abstract

In a recent simulation study, Goodman et al. (2019) compare several methods with regard to their type I and type II error rates in case of a thick null hypothesis that includes all values that are practically equivalent to the point null hypothesis. They propose a hybrid decision criterion only declaring a result "significant" if both a small $p$-value and a sufficiently large effect size are obtained. We successfully verify the results using our own software code in R and discuss an additional decision method that is tailored to maintain a pre-defined false positive rate. We confirm that the hybrid decision criterion has comparably low error rates in settings one can check for but point out that the false discovery rate cannot be easily controlled by the researcher. Our analyses are readily accessible and customizable on https://github.com/drehero/goodman-replication.