Validated Intraclass Correlation Statistics to Test Item Performance Models

TL;DR

This paper introduces a Matlab-based method using intraclass correlation statistics to evaluate item performance models on empirical data, effectively testing model fit and avoiding over- or under-fitting issues.

Contribution

The paper presents a novel statistical testing approach for item performance models that explicitly assesses model-data fit using intraclass correlation, validated on multiple behavioral datasets.

Findings

Method successfully tests model fit on three datasets.

Effective in detecting under-fitting and over-fitting.

Provides a practical alternative to traditional model selection criteria.

Abstract

A new method, with an application program in Matlab code, is proposed for testing item performance models on empirical databases. This method uses data intraclass correlation statistics as expected correlations to which one compares simple functions of correlations between model predictions and observed item performance. The method rests on a data population model whose validity for the considered data is suitably tested, and has been verified for three behavioural measure databases. Contrarily to usual model selection criteria, this method provides an effective way of testing under-fitting and over-fitting, answering the usually neglected question "does this model suitably account for these data?"