Leveraging AI Graders for Missing Score Imputation to Achieve Accurate Ability Estimation in Constructed-Response Tests

TL;DR

This paper introduces a novel approach that uses AI-based automated scoring to impute missing responses in constructed-response tests, significantly improving ability estimation accuracy and reducing manual grading efforts.

Contribution

It presents a new method combining AI grading with item response theory to accurately estimate abilities from incomplete data, addressing limitations of previous imputation techniques.

Findings

High accuracy in ability estimation with AI-based score imputation

Significant reduction in manual grading workload

Effective handling of sparse and heterogeneous data

Abstract

Evaluating the abilities of learners is a fundamental objective in the field of education. In particular, there is an increasing need to assess higher-order abilities such as expressive skills and logical thinking. Constructed-response tests such as short-answer and essay-based questions have become widely used as a method to meet this demand. Although these tests are effective, they require substantial manual grading, making them both labor-intensive and costly. Item response theory (IRT) provides a promising solution by enabling the estimation of ability from incomplete score data, where human raters grade only a subset of answers provided by learners across multiple test items. However, the accuracy of ability estimation declines as the proportion of missing scores increases. Although data augmentation techniques for imputing missing scores have been explored in order to address this limitation, they often struggle with inaccuracy for sparse or heterogeneous data. To overcome these challenges, this study proposes a novel method for imputing missing scores by leveraging automated scoring technologies for accurate IRT-based ability estimation. The proposed method achieves high accuracy in ability estimation while markedly reducing manual grading workload.