Measurement That Matches Theory: Theory-Driven Identification in IRT Models

TL;DR

This paper introduces a semi-supervised Bayesian IRT method that produces meaningful, reliable latent dimensions aligned with theoretical concepts, overcoming traditional limitations of IRT models in social sciences.

Contribution

It develops and validates a novel semi-supervised Bayesian IRT approach that captures meaningful latent dimensions without relying on exogenous assumptions.

Findings

Produces conceptually meaningful latent dimensions

Reliable across different data sources

Validated on simulated and real data

Abstract

Measurement bridges theory and empirics. Without measures that appropriately capture theoretical concepts, description will fail to represent reality and true causal inference will be impossible. Yet, the social sciences traffic in complex concepts and their measurement is difficult. Item Response Theory (IRT) models reduce variation in multiple variables to continuous variation along one or more latent dimensions intended to capture key theoretical concepts. Unfortunately, those latent dimensions have no intrinsic conceptual meaning. Partial solutions to that problem include limiting the number of dimensions to one or assigning meaning post-analysis, but either can lead to potential bias and a lack of reliability across data sources. We propose, detail, and validate a semi-supervised approach employing Bayesian Item Response Theory on multiple latent dimensions and binary data. Our approach, which we validate on simulated and real data, yields conceptually meaningful latent dimensions that are reliable across different data sources without additional exogenous assumptions.