Evaluating WAIC and PSIS-LOO for Bayesian Diagnostic Classification Model Selection

TL;DR

This paper compares Bayesian model fit indices, WAIC and PSIS-LOO, to DIC in diagnostic classification models, finding that WAIC and PSIS-LOO generally perform better in model detection but have some inconsistencies.

Contribution

It introduces and evaluates the performance of WAIC and PSIS-LOO for Bayesian DCMs, providing practical recommendations for model selection.

Findings

WAIC and PSIS-LOO favor the true model more often than DIC.

Occasional inconsistencies were observed with WAIC and PSIS-LOO.

Recommendations depend on model complexity and prior informativeness.

Abstract

Bayesian diagnostic classification models (Bayesian DCMs) are effective for diagnosing students' skills. Research on the evaluation of relative model fit indices for DCMs using Bayesian estimation, however, is deficient. This study introduces the performance of Bayesian relative model fit indices, the widely applicable information criterion (WAIC) and leave-one-out cross-validation using Pareto-smoothed importance sampling (PSIS-LOO), in comparison to simpler and more widely used deviance information criterion (DIC). The simulation study evaluates the performance of WAIC and PSIS-LOO by detecting the true model with varying sample sizes, item qualities, and prior information levels. The results of the study indicate that WAIC and PSIS-LOO primarily favored the generating model; however, occasional inconsistencies were observed. This study recommends using WAIC and PSIS-LOO when the data is assumed to follow a simpler model and the models are estimated under uninformative priors, and DIC when the data is assumed to follow a more complex model.