The Spearman-Brown Formula and Reliabilities of Random Test Forms

TL;DR

This paper demonstrates that test reliability converges to 1 with increasing test length under broad conditions, and explores how the Spearman-Brown formula predicts reliability, revealing biases in short multidimensional tests.

Contribution

It establishes the asymptotic behavior of test reliability under general conditions and evaluates the Spearman-Brown formula's accuracy in different test models.

Findings

Reliability converges to 1 as test length increases.

The Spearman-Brown formula accurately predicts reliability for long tests.

Biases occur in short multidimensional tests, overestimating reliability.

Abstract

It is shown that the psychometric test reliability, based on any true-score model with randomly sampled items and conditionally independent errors, converges to 1 as the test length goes to infinity, assuming some fairly general regularity conditions. The asymptotic rate of convergence is given by the Spearman-Brown formula, and for this it is not needed that the items are parallel, or latent unidimensional, or even finite dimensional. Simulations with the 2-parameter logistic item response theory model reveal that there can be a positive bias in the reliability of short multidimensional tests, meaning that applying the Spearman-Brown formula in these cases would lead to overprediction of the reliability that will result from lengthening the tests. For short unidimensional tests under the 2-parameter logistic model the reliabilities are almost unbiased, meaning that application of the Spearman-Brown formula in these cases leads to predictions that are approximately unbiased.