Survey on Fairness Notions and Related Tensions

TL;DR

This survey reviews various fairness notions in machine learning, discusses their tensions with privacy and accuracy, and analyzes methods to balance fairness with other model properties through experimental case studies.

Contribution

It provides a comprehensive overview of fairness definitions, explores their inherent tensions, and evaluates methods to mitigate fairness-accuracy trade-offs with empirical analysis.

Findings

Fairness notions often conflict with each other and with privacy and accuracy.

Different mitigation methods have varying effectiveness depending on the context.

Empirical results illustrate the complex relationship between fairness measures and accuracy.

Abstract

Automated decision systems are increasingly used to take consequential decisions in problems such as job hiring and loan granting with the hope of replacing subjective human decisions with objective machine learning (ML) algorithms. However, ML-based decision systems are prone to bias, which results in yet unfair decisions. Several notions of fairness have been defined in the literature to capture the different subtleties of this ethical and social concept (e.g., statistical parity, equal opportunity, etc.). Fairness requirements to be satisfied while learning models created several types of tensions among the different notions of fairness and other desirable properties such as privacy and classification accuracy. This paper surveys the commonly used fairness notions and discusses the tensions among them with privacy and accuracy. Different methods to address the fairness-accuracy trade-off (classified into four approaches, namely, pre-processing, in-processing, post-processing, and hybrid) are reviewed. The survey is consolidated with experimental analysis carried out on fairness benchmark datasets to illustrate the relationship between fairness measures and accuracy in real-world scenarios.