Identifying Causes of Test Unfairness: Manipulability and Separability

TL;DR

This paper introduces a causal framework for understanding and identifying the sources of test unfairness related to non-manipulable group characteristics, using interventionist methods and causal machine learning tools.

Contribution

It proposes a novel interventionist approach to decompose and identify causal sources of test unfairness, with formal strategies and detection methods applied to real exam data.

Findings

Decomposed ELL status into vocabulary and learning barriers affecting test outcomes.

Developed causal detection methods using causal forests and Bayesian additive regression trees.

Demonstrated effectiveness of methods through simulation and real data analysis.

Abstract

Differential item functioning (DIF) is a widely used statistical notion for identifying items that may disadvantage specific groups of test-takers. These groups are often defined by non-manipulable characteristics, e.g., gender, race/ethnicity, or English-language learner (ELL) status. While DIF can be framed as a causal fairness problem by treating group membership as the treatment variable, this invokes the long-standing controversy over the interpretation of causal effects for non-manipulable treatments. To better identify and interpret causal sources of DIF, this study leverages an interventionist approach using treatment decomposition proposed by Robins and Richardson (2010). Under this framework, we can decompose a non-manipulable treatment into intervening variables. For example, ELL status can be decomposed into English vocabulary unfamiliarity and classroom learning barriers, each of which influences the outcome through different causal pathways. We formally define separable DIF effects associated with these decomposed components, depending on the absence or presence of item impact, and provide causal identification strategies for each effect. We then apply the framework to biased test items in the SAT and Regents exams. We also provide formal detection methods using causal machine learning methods, namely causal forests and Bayesian additive regression trees, and demonstrate their performance through a simulation study. Finally, we discuss the implications of adopting interventionist approaches in educational testing practices.