Mutatis Mutandis: Revisiting the Comparator in Discrimination Testing

TL;DR

This paper critically examines the comparator role in discrimination testing, proposing two types—CP and MM—and discusses their implications for causal modeling and machine learning applications.

Contribution

It introduces a novel classification of comparators in discrimination testing, emphasizing the causal modeling aspect and the potential for machine learning implementation.

Findings

The MM comparator accounts for appropriate adjustments, unlike the standard CP comparator.

The MM comparator allows for dissimilar non-protected attributes in pair comparisons.

Illustrated impact of comparator types using a real-world discrimination testing example.

Abstract

Testing for individual discrimination involves deriving a profile, the comparator, similar to the one making the discrimination claim, the complainant, based on a protected attribute, such as race or gender, and comparing their decision outcomes. The complainant-comparator pair is central to discrimination testing. Most discrimination testing tools rely on this pair to establish evidence for discrimination. In this work, we revisit the role of the comparator in discrimination testing. We first argue for the inherent causal modeling nature of deriving the comparator. We then introduce a two-kind classification for the comparator: the ceteris paribus, or "with all else equal," (CP) comparator and the mutatis mutandis, or "with the appropriate adjustments being made," (MM) comparator. The CP comparator is the standard comparator, representing an idealized comparison for establishing discrimination as it aims for a complainant-comparator pair that only differs in membership in the protected attribute. As an alternative to the CP comparator, we define the MM comparator, which requires a comparator that represents the ``what would have been'' of the complainant without the effects of the protected attribute on the non-protected attributes. Under the MM comparator, the complainant-comparator pair can be dissimilar in terms of the non-protected attributes, departing from the idealized comparison imposed by the CP comparator. Notably, the MM comparator denotes a more complex object and its implementation offers an impactful venue for machine learning methods. We illustrate these two comparators and their impact on discrimination testing using a real-world example.