When Fairness Isn't Statistical: The Limits of Machine Learning in Evaluating Legal Reasoning

TL;DR

This paper critically examines the limitations of machine learning methods in assessing fairness within legal decision-making, emphasizing the importance of legal reasoning over purely statistical approaches.

Contribution

It demonstrates that current ML techniques often produce inconsistent results and fail to capture legal reasoning, highlighting the need for context-aware fairness evaluation methods.

Findings

ML methods produce divergent signals on legal fairness

Predictive models rely on procedural rather than legal features

Semantic clustering does not capture legal reasoning

Abstract

Legal decisions are increasingly evaluated for fairness, consistency, and bias using machine learning (ML) techniques. In high-stakes domains like refugee adjudication, such methods are often applied to detect disparities in outcomes. Yet it remains unclear whether statistical methods can meaningfully assess fairness in legal contexts shaped by discretion, normative complexity, and limited ground truth. In this paper, we empirically evaluate three common ML approaches (feature-based analysis, semantic clustering, and predictive modeling) on a large, real-world dataset of 59,000+ Canadian refugee decisions (AsyLex). Our experiments show that these methods produce divergent and sometimes contradictory signals, that predictive modeling often depends on contextual and procedural features rather than legal features, and that semantic clustering fails to capture substantive legal reasoning. We show limitations of statistical fairness evaluation, challenge the assumption that statistical regularity equates to fairness, and argue that current computational approaches fall short of evaluating fairness in legally discretionary domains. We argue that evaluating fairness in law requires methods grounded not only in data, but in legal reasoning and institutional context.