Exploring Equality: An Investigation into Custom Loss Functions for Fairness Definitions

TL;DR

This paper investigates the tradeoffs between fairness metrics and accuracy in COMPAS, introducing a novel fairness-driven framework (GAP) and analytical methods to improve fairness while acknowledging external biases.

Contribution

It develops the first implementation of the Group Accuracy Parity (GAP) framework for fairness in COMPAS and proposes a comprehensive analytical procedure for fairness comparison.

Findings

GAP improves fairness-accuracy balance over traditional COMPAS models.

Custom loss functions can enhance fairness metrics in predictive models.

External biases and lack of transparency still challenge fairness improvements.

Abstract

This paper explores the complex tradeoffs between various fairness metrics such as equalized odds, disparate impact, and equal opportunity and predictive accuracy within COMPAS by building neural networks trained with custom loss functions optimized to specific fairness criteria. This paper creates the first fairness-driven implementation of the novel Group Accuracy Parity (GAP) framework, as theoretically proposed by Gupta et al. (2024), and applies it to COMPAS. To operationalize and accurately compare the fairness of COMPAS models optimized to differing fairness ideals, this paper develops and proposes a combinatory analytical procedure that incorporates Pareto front and multivariate analysis, leveraging data visualizations such as violin graphs. This paper concludes that GAP achieves an enhanced equilibrium between fairness and accuracy compared to COMPAS's current nationwide implementation and alternative implementations of COMPAS optimized to more traditional fairness definitions. While this paper's algorithmic improvements of COMPAS significantly augment its fairness, external biases undermine the fairness of its implementation. Practices such as predictive policing and issues such as the lack of transparency regarding COMPAS's internal workings have contributed to the algorithm's historical injustice. In conjunction with developments regarding COMPAS's predictive methodology, legal and institutional changes must happen for COMPAS's just deployment.