Resampling-based inference methods for comparing two coefficient alpha

TL;DR

This paper develops resampling-based permutation and bootstrap tests to compare two Cronbach's alpha coefficients, providing more accurate control of type I error in small samples and non-normal data.

Contribution

It introduces novel resampling methods for two-sample comparison of reliability coefficients, improving small sample performance over existing asymptotic tests.

Findings

Resampling tests outperform asymptotic tests in small samples.

Proposed methods maintain validity under non-normality.

Real data applications demonstrate practical utility.

Abstract

The two-sample problem for Cronbach's coefficient $\alpha_C$, as an estimate of test or composite score reliability, has attracted little attention, compared to the extensive treatment of the one-sample case. It is necessary to compare the reliability of a test for different subgroups, for different tests or the short and long forms of a test. In this paper, we study statistically how to compare two coefficients $\alpha_{C,1}$ and $\alpha_{C,2}$. The null hypothesis of interest is $H_0 : \alpha_{C,1} = \alpha_{C,2}$, which we test against one-or two-sided alternatives. For this purpose, resampling-based permutation and bootstrap tests are proposed. These statistical tests ensure a better control of the type I error, in finite or very small sample sizes, when the state-of-affairs \textit{asymptotically distribution-free} (ADF) large-sample test may fail to properly attain the nominal significance level. We introduce the permutation and bootstrap tests for the two-group multivariate non-normal models under the general ADF setting, thereby improving on the small sample properties of the well-known ADF asymptotic test. By proper choice of a studentized test statistic, the resampling tests are modified such that they are still asymptotically valid, if the data may not be exchangeable. The usefulness of the proposed resampling-based testing strategies is demonstrated in an extensive simulation study and illustrated by real data applications.