Disentangling individual differences in cognitive response mechanisms for rating scale items: A flexible-mixture multidimensional IRTree approach

TL;DR

This paper introduces a new model to better understand how people respond to rating scales by accounting for different thinking strategies.

Contribution

The novel contribution is a mixture IRTree model (MixTree) that allows for heterogeneous response strategies among individuals.

Findings

MixTree identifies latent classes of respondents with distinct response mechanisms.

Simulation studies confirm the model's ability to recover classes and parameters accurately.

Empirical analysis reveals two classes: one driven by traits and another by response styles.

Abstract

The accuracy of our inferences from rating-scale items can be improved with IRTree models, which consider heuristic response strategies like response styles (RS). IRTree models break down ordinal responses into pseudo-items (nodes), each representing a distinct decision-making process. These nodes are then modeled using an item response model. In the case of four-point items, a response is split into two nodes: 1) response direction, where the trait influences the overall agreement with items, and 2) response extremity, where both the trait and extreme RS (ERS) impact the choice of relative (dis)agreement categories. However, traditional models, despite addressing RS effects, assume that all respondents follow an identical response strategy, where the selection of relative (dis)agreement categories is influenced by the trait and ERS to the same degree for all respondents. Given that respondents may vary in the extent to which they adopt heuristic-driven strategies (e.g., fatigue, motivation, expertise), this assumption of homogeneous response processes is unlikely to be satisfied, potentially leading to inaccurate inferences. To accommodate different response strategies, we introduce the mixture IRTree model (MixTree). In MixTree, participants are assigned to different latent classes, each associated with distinct response processes. Based on their class memberships, varying weights are assigned to individuals’ trait and ERS scores. Additionally, MixTree simultaneously examines extraneous variables to explore sources of heterogeneity. A simulation study validates the MixTree’s performance in recovering classes and model parameters. Empirical data analysis identifies two latent classes, one linked to a trait-driven and the other to RS-driven mechanisms.

Figures