On a General Theoretical Framework of Reliability

TL;DR

This paper develops a comprehensive theoretical framework for reliability, emphasizing the measurement of association between latent and observed scores, extending existing models and proposing new desiderata for reliability measures.

Contribution

It introduces a general framework inspired by McDonald's regression approach, extending it with four key desiderata and illustrating various reliability measures through examples and simulations.

Findings

Different reliability measures behave distinctly in numerical studies.

The proposed framework clarifies the properties and applications of reliability coefficients.

Future research directions are outlined for advancing reliability measurement.

Abstract

Reliability is an essential measure of how closely observed scores represent latent scores (reflecting constructs), assuming some latent variable measurement model. We present a general theoretical framework of reliability, placing emphasis on measuring the association between latent and observed scores. This framework was inspired by McDonald's (2011) regression framework, which highlighted the coefficient of determination as a measure of reliability. We extend McDonald's (2011) framework beyond coefficients of determination and introduce four desiderata for reliability measures (estimability, normalization, symmetry, and invariance). We also present theoretical examples to illustrate distinct measures of reliability and report on a numerical study that demonstrates the behavior of different reliability measures. We conclude with a discussion on the use of reliability coefficients and outline future avenues of research.