Accurate confidence interval estimation for non-centrality parameters and effect size indices

TL;DR

This paper introduces a bootstrap confidence interval method for the robust effect size index (RESI), related to the non-centrality parameter, providing accurate and consistent effect size estimation even when model assumptions are violated.

Contribution

It develops a bootstrap confidence interval procedure for RESI, a robust effect size measure, and proposes a general framework for effect size reporting in ANOVA analyses.

Findings

Robust estimator is consistent for true effect size.

Common CI procedures often fail to cover the true effect size.

Bootstrap CI provides accurate and valid inference for RESI.

Abstract

We recently proposed a robust effect size index (RESI) that is related to the non-centrality parameter of a test statistic. RESI is advantageous over common indices because (1) it is widely applicable to many types of data; (2) it can rely on a robust covariance estimate; (3) it can accommodate the existence of nuisance parameters. We provided a consistent estimator for the RESI, however, there is no established confidence interval (CI) estimation procedure for the RESI. Here, we use statistical theory and simulations to evaluate several CI estimation procedures for three estimators of the RESI. Our findings show (1) in contrast to common effect sizes, the robust estimator is consistent for the true effect size; (2) common CI procedures for effect sizes that are non-centrality parameters fail to cover the true effect size at the nominal level. Using the robust estimator along with the proposed bootstrap CI is generally accurate and applicable to conduct consistent estimation and valid inference for the RESI, especially when model assumptions may be violated. Based on the RESI, we propose a general framework for the analysis of effect size (ANOES), such that effect sizes and confidence intervals can be easily reported in an analysis of variance (ANOVA) table format for a wide range of models