Assessing the Performance of Diagnostic Classification Models in Small Sample Contexts with Different Estimation Methods

TL;DR

This study evaluates how different estimation methods perform in small sample settings for diagnostic classification models, revealing nuanced differences in respondent classification accuracy and item parameter recovery.

Contribution

It provides a comprehensive simulation analysis comparing ML, Bayesian, and nonparametric methods for DCMs in classroom-like small samples, highlighting their strengths and limitations.

Findings

Bayesian method slightly better in simple DCMs

ML method slightly better in complex DCMs

Item parameter recovery was generally poor

Abstract

Fueled by the call for formative assessments, diagnostic classification models (DCMs) have recently gained popularity in psychometrics. Despite their potential for providing diagnostic information that aids in classroom instruction and students' learning, empirical applications of DCMs to classroom assessments have been highly limited. This is partly because how DCMs with different estimation methods perform in small sample contexts is not yet well-explored. Hence, this study aims to investigate the performance of respondent classification and item parameter estimation with a comprehensive simulation design that resembles classroom assessments using different estimation methods. The key findings are the following: (1) although the marked difference in respondent classification accuracy was not observed among the maximum likelihood (ML), Bayesian, and nonparametric methods, the Bayesian method provided slightly more accurate respondent classification in parsimonious DCMs than the ML method, and in complex DCMs, the ML method yielded the slightly better result than the Bayesian method; (2) while item parameter recovery was poor in both Bayesian and ML methods, the Bayesian method exhibited unstable slip values owing to the multimodality of their posteriors under complex DCMs, and the ML method produced irregular estimates that appear to be well-estimated due to a boundary problem under parsimonious DCMs.