Evaluating Fairness in Transaction Fraud Models: Fairness Metrics, Bias Audits, and Challenges

TL;DR

This paper conducts the first bias audit of transaction fraud detection models, revealing significant biases, the limitations of unawareness approaches, and emphasizing the need for specialized fairness metrics and methods.

Contribution

It introduces a comprehensive fairness evaluation framework for fraud models using synthetic datasets, highlighting unique challenges and limitations of existing bias mitigation strategies.

Findings

Fairness metrics reveal bias only after normalization due to class imbalance.

Bias affects both service quality and fraud protection metrics.

Removing sensitive attributes does not reduce bias, due to correlated proxies.

Abstract

Ensuring fairness in transaction fraud detection models is vital due to the potential harms and legal implications of biased decision-making. Despite extensive research on algorithmic fairness, there is a notable gap in the study of bias in fraud detection models, mainly due to the field's unique challenges. These challenges include the need for fairness metrics that account for fraud data's imbalanced nature and the tradeoff between fraud protection and service quality. To address this gap, we present a comprehensive fairness evaluation of transaction fraud models using public synthetic datasets, marking the first algorithmic bias audit in this domain. Our findings reveal three critical insights: (1) Certain fairness metrics expose significant bias only after normalization, highlighting the impact of class imbalance. (2) Bias is significant in both service quality-related parity metrics and fraud protection-related parity metrics. (3) The fairness through unawareness approach, which involved removing sensitive attributes such as gender, does not improve bias mitigation within these datasets, likely due to the presence of correlated proxies. We also discuss socio-technical fairness-related challenges in transaction fraud models. These insights underscore the need for a nuanced approach to fairness in fraud detection, balancing protection and service quality, and moving beyond simple bias mitigation strategies. Future work must focus on refining fairness metrics and developing methods tailored to the unique complexities of the transaction fraud domain.