Assessment of Misspecification in CDMs Using a Generalized Information Matrix Test

TL;DR

This paper introduces a new statistical test, GIMT_Det, for detecting misspecification in cognitive diagnostic models, demonstrating promising results in simulation studies but requiring further validation.

Contribution

The paper proposes the GIMT_Det test for assessing misspecification in CDMs, providing an alternative to existing methods with promising initial performance.

Findings

GIMT_Det effectively detects larger levels of misspecification.

GIMT_Det does not falsely detect misspecification when none exists.

Performance varies with different misspecification strategies.

Abstract

If the probability model is correctly specified, then we can estimate the covariance matrix of the asymptotic maximum likelihood estimate distribution using either the first or second derivatives of the likelihood function. Therefore, if the determinants of these two different covariance matrix estimation formulas differ this indicates model misspecification. This misspecification detection strategy is the basis of the Determinant Information Matrix Test ($GIMT_{Det}$). To investigate the performance of the $GIMT_{Det}$, a Deterministic Input Noisy And gate (DINA) Cognitive Diagnostic Model (CDM) was fit to the Fraction-Subtraction dataset. Next, various misspecified versions of the original DINA CDM were fit to bootstrap data sets generated by sampling from the original fitted DINA CDM. The $GIMT_{Det}$ showed good discrimination performance for larger levels of misspecification. In addition, the $GIMT_{Det}$ did not detect model misspecification when model misspecification was not present and additionally did not detect model misspecification when the level of misspecification was very low. However, the $GIMT_{Det}$ discrimation performance was highly variable across different misspecification strategies when the misspecification level was moderately sized. The proposed new misspecification detection methodology is promising but additional empirical studies are required to further characterize its strengths and limitations.