Unfair items detection in educational measurement

TL;DR

This paper introduces a continuous, distribution-free measure of item unfairness in educational testing, capable of identifying unfair items beyond traditional methods and aligning with expert judgments.

Contribution

A novel, assumption-free unfairness measure for test items that improves detection of unfair items in educational assessments.

Findings

The measure aligns with expert judgments.

It detects unfair items missed by conventional methods.

Applicable to any educational test without distribution assumptions.

Abstract

Measurement professionals cannot come to an agreement on the definition of the term 'item fairness'. In this paper a continuous measure of item unfairness is proposed. The more the unfairness measure deviates from zero, the less fair the item is. If the measure exceeds the cutoff value, the item is identified as definitely unfair. The new approach can identify unfair items that would not be identified with conventional procedures. The results are in accord with experts' judgments on the item qualities. Since no assumptions about scores distributions and/or correlations are assumed, the method is applicable to any educational test. Its performance is illustrated through application to scores of a real test.