Toward Sufficient Statistical Power in Algorithmic Bias Assessment: A Test for ABROCA

TL;DR

This paper examines the ABROCA metric for fairness in educational data mining, revealing its distributional challenges and proposing nonparametric tests to improve bias detection reliability in typical sample sizes.

Contribution

It introduces robust significance testing methods for ABROCA, addressing distributional issues and power limitations in bias assessment within EDM.

Findings

ABROCA does not follow standard distributions, including skewed ones.

Nonparametric randomization tests improve bias detection reliability.

Large sample sizes or effect sizes are needed for effective bias detection.

Abstract

Algorithmic bias is a pressing concern in educational data mining (EDM), as it risks amplifying inequities in learning outcomes. The Area Between ROC Curves (ABROCA) metric is frequently used to measure discrepancies in model performance across demographic groups to quantify overall model fairness. However, its skewed distribution--especially when class or group imbalances exist--makes significance testing challenging. This study investigates ABROCA's distributional properties and contributes robust methods for its significance testing. Specifically, we address (1) whether ABROCA follows any known distribution, (2) how to reliably test for algorithmic bias using ABROCA, and (3) the statistical power achievable with ABROCA-based bias assessments under typical EDM sample specifications. Simulation results confirm that ABROCA does not match standard distributions, including those suited to accommodate skewness. We propose nonparametric randomization tests for ABROCA and demonstrate that reliably detecting bias with ABROCA requires large sample sizes or substantial effect sizes, particularly in imbalanced settings. Findings suggest that ABROCA-based bias evaluations based on sample sizes common in EDM tend to be underpowered, undermining the reliability of conclusions about model fairness. By offering open-source code to simulate power and statistically test ABROCA, this paper aims to foster more reliable statistical testing in EDM research. It supports broader efforts toward replicability and equity in educational modeling.