Explanatory Cognitive Diagnosis Models Incorporating Item Features

TL;DR

This paper introduces a new method to improve cognitive diagnosis models by incorporating item features to better understand and enhance item quality.

Contribution

The novel contribution is the development of item explanatory cognitive diagnosis models that link item parameters to item features.

Findings

The proposed models effectively relate item parameters to linguistic features extracted from item stems.

Simulation studies confirm the validity of inferences drawn from empirical data analysis.

The approach provides insights into improving low-quality items in cognitive diagnosis.

Abstract

Item quality is crucial to psychometric analyses for cognitive diagnosis. In cognitive diagnosis models (CDMs), item quality is often quantified in terms of item parameters (e.g., guessing and slipping parameters). Calibrating the item parameters with only item response data, as a common practice, could result in challenges in identifying the cause of low-quality items (e.g., the correct answer is easy to be guessed) or devising an effective plan to improve the item quality. To resolve these challenges, we propose the item explanatory CDMs where the CDM item parameters are explained with item features such that item features can serve as an additional source of information for item parameters. The utility of the proposed models is demonstrated with the Trends in International Mathematics and Science Study (TIMSS)-released items and response data: around 20 item linguistic features were extracted from the item stem with natural language processing techniques, and the item feature engineering process is elaborated in the paper. The proposed models are used to examine the relationships between the guessing/slipping item parameters of the higher-order DINA model and eight of the item features. The findings from a follow-up simulation study are presented, which corroborate the validity of the inferences drawn from the empirical data analysis. Finally, future research directions are discussed.

Figures