On the relation between accuracy and fairness in binary classification

TL;DR

This paper examines the accuracy-fairness tradeoff in binary classification, emphasizing the importance of accounting for positive prediction rates when comparing classifiers to avoid misleading conclusions.

Contribution

It introduces methodological guidelines for fair comparison of classifiers and analyzes the tradeoffs between accuracy and non-discrimination both theoretically and empirically.

Findings

Comparison of classifiers must consider positive prediction rates.

Naive baselines' accuracy and discrimination vary with positive prediction rates.

Proper evaluation methods are essential for accurate fairness assessments.

Abstract

Our study revisits the problem of accuracy-fairness tradeoff in binary classification. We argue that comparison of non-discriminatory classifiers needs to account for different rates of positive predictions, otherwise conclusions about performance may be misleading, because accuracy and discrimination of naive baselines on the same dataset vary with different rates of positive predictions. We provide methodological recommendations for sound comparison of non-discriminatory classifiers, and present a brief theoretical and empirical analysis of tradeoffs between accuracy and non-discrimination.