Aggregating Concepts of Fairness and Accuracy in Prediction Algorithms

TL;DR

This paper proposes a method to balance fairness and accuracy in predictive algorithms by using a linear combination of metrics, grounded in preference aggregation theory, and applies it to real-world data.

Contribution

It introduces a formal framework for aggregating fairness and accuracy measures in prediction algorithms using Harsanyi's preference aggregation result.

Findings

Linear combination of fairness and accuracy metrics effectively balances trade-offs.

Application to COMPAS dataset demonstrates practical utility.

Provides normative guidelines for fairness-accuracy trade-offs.

Abstract

An algorithm that outputs predictions about the state of the world will almost always be designed with the implicit or explicit goal of outputting accurate predictions (i.e., predictions that are likely to be true). In addition, the rise of increasingly powerful predictive algorithms brought about by the recent revolution in artificial intelligence has led to an emphasis on building predictive algorithms that are fair, in the sense that their predictions do not systematically evince bias or bring about harm to certain individuals or groups. This state of affairs presents two conceptual challenges. First, the goals of accuracy and fairness can sometimes be in tension, and there are no obvious normative guidelines for managing the trade-offs between these two desiderata when they arise. Second, there are many distinct ways of measuring both the accuracy and fairness of a predictive algorithm; here too, there are no obvious guidelines on how to aggregate our preferences for predictive algorithms that satisfy disparate measures of fairness and accuracy to various extents. The goal of this paper is to address these challenges by arguing that there are good reasons for using a linear combination of accuracy and fairness metrics to measure the all-things-considered value of a predictive algorithm for agents who care about both accuracy and fairness. My argument depends crucially on a classic result in the preference aggregation literature due to Harsanyi. After making this formal argument, I apply my result to an analysis of accuracy-fairness trade-offs using the COMPAS dataset compiled by Angwin et al.