Uncertainty in Test Score Data and Classically Defined Reliability of Tests and Test Batteries, using a New Method for Test Dichotomisation

TL;DR

This paper introduces a new fast method for estimating test reliability based on classical test theory by splitting tests into parallel halves, enabling accurate reliability measurement from a single test administration.

Contribution

It presents a novel, efficient method for splitting tests into parallel halves based on difficulty, allowing reliable estimation from one test administration, applicable to large datasets and test batteries.

Findings

The method achieves near-coincident empirical distributions of test halves.

It enables fast computation of test reliability for large datasets.

The approach extends to reliability estimation of test batteries.

Abstract

As with all measurements, the measurement of examinee ability, in terms of scores that the examinee obtains in a test, is also error-ridden. The quantification of such error or uncertainty in the test score data--or rather the complementary test reliability--is pursued within the paradigm of Classical Test Theory in a variety of ways, with no existing method of finding reliability, isomorphic to the theoretical definition that parametrises reliability as the ratio of the true score variance and observed (i.e. error-ridden) score variance. Thus, multiple reliability coefficients for the same test have been advanced. This paper describes a much needed method of obtaining reliability of a test as per its theoretical definition, via a single administration of the test, by using a new fast method of splitting of a given test into parallel halves, achieving near-coincident empirical distributions of the two halves. The method has the desirable property of achieving splitting on the basis of difficulty of the questions (or items) that constitute the test, thus allowing for fast computation of reliability even for very large test data sets, i.e. test data obtained by a very large examinee sample. An interval estimate for the true score is offered, given an examinee score, subsequent to the determination of the test reliability. This method of finding test reliability as per the classical definition can be extended to find reliability of a set or battery of tests; a method for determination of the weights implemented in the computation of the weighted battery score is discussed. We perform empirical illustration of our method on real and simulated tests, and on a real test battery comprising two constituent tests.