Heterogeneous item populations across individuals: Consequences for the factor model, item inter-correlations, and scale validity

TL;DR

This paper investigates how random selection of items from different populations affects factor analysis and scale validity, revealing that correlations can be misleading and proposing a method to detect subpopulations with distinct item responses.

Contribution

It introduces a novel method to detect subpopulations responding to different item populations, addressing scale validity issues caused by heterogeneous item responses.

Findings

Correlations can be high even with uncorrelated item populations across subgroups.

The proposed method effectively detects subpopulations with different item response patterns.

Recommendations for identifying heterogeneous subpopulations in scale validation.

Abstract

The paper is devoted to the consequences of blind random selection of items from different item populations that might be based on completely uncorrelated factors for item inter-correlations and corresponding factor loadings. Based on the model of essentially parallel measurements, we explore the consequences of presenting items from different populations across individuals and items from identical populations within each individual for the factor model and item inter-correlations in the total population of individuals. Moreover, we explore the consequences of presenting items from different as well as identical item populations across and within individuals. We show that correlations can be substantial in the total population of individuals even when -- in subpopulations of individuals -- items are drawn from populations with uncorrelated factors. In order to address this challenge for the validity of a scale, we propose a method that helps to detect whether item inter-correlations result from different item populations in different subpopulations of individuals and evaluate the method by means of a simulation study. Based on the analytical results and on results from a simulation study, we provide recommendations for the detection of subpopulations of individuals responding to items from different item populations.