A New Item Response Theory Model for Open-Ended Online Homework with Multiple Allowed Attempts

TL;DR

This paper introduces an enhanced IRT model for online homework that accounts for copying and guessing behaviors, improving the accuracy of ability assessment and item quality evaluation.

Contribution

It proposes a new IRT model incorporating copying and guessing traits, addressing limitations of traditional models in formative online assessments.

Findings

Inclusion of copying and guessing traits improves ability estimation.

Copying answers has a larger impact than guessing on initial attempts.

Model traits correlate with self-reported student behaviors.

Abstract

Item Response Theory (IRT) was originally developed in traditional exam settings, and it has been shown that the model does not readily transfer to formative assessment in the form of online homework. We investigate if this is mostly due to learner traits that do not become apparent in exam settings, namely random guessing due to lack of diligence or dedication, and copying work from other students or resources. Both of these traits mask the true ability of the learner, which is the only trait considered in most mainstream unidimensional IRT models. We find that indeed the introduction of these traits allows to better assess the true ability of the learners, as well as to better gauge the quality of assessment items. Correspondence of the model traits to self-reported behavior is investigated and confirmed. We find that of these two traits, copying answers has a larger influence on initial homework attempts than random guessing.