A Differential Index Measuring Rater's Capability in Educational Assessment

TL;DR

This paper introduces a new single-value index to measure rater accuracy across different student ability levels, derived from partial derivatives of passing rates, with efficient estimation methods validated through simulations and real data.

Contribution

It proposes a novel capability index for raters that accounts for varying student abilities, derived mathematically and estimated efficiently for large datasets.

Findings

Raters show different capabilities across topics.

The index accurately recovers rater ability in simulations.

Empirical results reveal topic-dependent rater performance.

Abstract

A rater's ability to assign accurate scores can significantly impact the outcomes of educational assessments. However, common indices for evaluating rater characteristics typically focus on either their severity or their discrimination ability (i.e., skills to differentiate between students). Additionally, these indices are often developed without considering the rater's accuracy in scoring students at different ability levels. To address the limitations, this study proposes a single-value measure to assess a rater's capability of assigning accurate scores to students with varying ability levels. The measure is derived from the partial derivatives of each rater's passing rate concerning student ability. Mathematical derivations of the index under generalized multi-facet models and hierarchical rater models are provided. To ease the implementation of the index, this study develops parameter estimation using marginal likelihood and its Laplacian approximation which allows for efficient evaluation and processing of large datasets involving numerous students and raters. Simulation studies demonstrate the accuracy of parameter recovery using the approximate likelihood and show how the capability indices vary with different levels of rater severity. An empirical study further tests the practical applicability of the new measure, where raters evaluate essays on four topics: "family," "school," "sport," and "work." Results show that raters are most capable when rating the topic of family and least capable when rating sport, with individual raters displaying different capabilities across the various topics.