Exposing the Illusion of Fairness: Auditing Vulnerabilities to Distributional Manipulation Attacks

TL;DR

This paper reveals how malicious actors can craft samples that appear fair and representative to deceive fairness audits in AI systems, highlighting vulnerabilities and proposing detection strategies.

Contribution

It formalizes the problem of fairness manipulation as a distributional projection, introduces manipulation strategies, and evaluates detection methods to improve audit robustness.

Findings

Manipulation can satisfy fairness constraints with minimal distributional shift.

Statistical tests can detect some manipulated samples but may fail under certain conditions.

Guidelines are provided to strengthen verification against distributional manipulation.

Abstract

The rapid deployment of AI systems in high-stakes domains, including those classified as high-risk under the The EU AI Act (Regulation (EU) 2024/1689), has intensified the need for reliable compliance auditing. For binary classifiers, regulatory risk assessment often relies on global fairness metrics such as the Disparate Impact ratio, widely used to evaluate potential discrimination. In typical auditing settings, the auditee provides a subset of its dataset to an auditor, while a supervisory authority may verify whether this subset is representative of the full underlying distribution. In this work, we investigate to what extent a malicious auditee can construct a fairness-compliant yet representative-looking sample from a non-compliant original distribution, thereby creating an illusion of fairness. We formalize this problem as a constrained distributional projection task and introduce mathematically grounded manipulation strategies based on entropic and optimal transport projections. These constructions characterize the minimal distributional shift required to satisfy fairness constraints. To counter such attacks, we formalize representativeness through distributional distance based statistical tests and systematically evaluate their ability to detect manipulated samples. Our analysis highlights the conditions under which fairness manipulation can remain statistically undetected and provides practical guidelines for strengthening supervisory verification. We validate our theoretical findings through experiments on standard tabular datasets for bias detection. Code is publicly available at https://github.com/ValentinLafargue/Inspection.