Measuring, Interpreting, and Improving Fairness of Algorithms using Causal Inference and Randomized Experiments

TL;DR

This paper introduces a practical, algorithm-agnostic framework that uses causal inference and randomized experiments to measure, interpret, and improve fairness in AI systems, addressing real-world application challenges.

Contribution

The authors develop MIIF, a systematic, easy-to-implement framework combining randomized experiments and interpretability techniques to assess and enhance algorithm fairness without requiring access to model code.

Findings

Effective measurement of disparate treatment and impact using randomized experiments

Accurate interpretation of blackbox algorithms with explainable models

Demonstrated applicability in e-commerce and advertising scenarios

Abstract

Algorithm fairness has become a central problem for the broad adoption of artificial intelligence. Although the past decade has witnessed an explosion of excellent work studying algorithm biases, achieving fairness in real-world AI production systems has remained a challenging task. Most existing works fail to excel in practical applications since either they have conflicting measurement techniques and/ or heavy assumptions, or require code-access of the production models, whereas real systems demand an easy-to-implement measurement framework and a systematic way to correct the detected sources of bias. In this paper, we leverage recent advances in causal inference and interpretable machine learning to present an algorithm-agnostic framework (MIIF) to Measure, Interpret, and Improve the Fairness of an algorithmic decision. We measure the algorithm bias using randomized experiments, which enables the simultaneous measurement of disparate treatment, disparate impact, and economic value. Furthermore, using modern interpretability techniques, we develop an explainable machine learning model which accurately interprets and distills the beliefs of a blackbox algorithm. Altogether, these techniques create a simple and powerful toolset for studying algorithm fairness, especially for understanding the cost of fairness in practical applications like e-commerce and targeted advertising, where industry A/B testing is already abundant.