The Judge Variable: Challenging Judge-Agnostic Legal Judgment Prediction

TL;DR

This paper investigates whether individual judges' decision patterns significantly influence legal outcomes, using machine learning models trained on French appellate court rulings to compare judge-specific and judge-agnostic approaches.

Contribution

It introduces a hybrid ML approach combining LLMs and traditional models to assess judge-specific decision patterns, challenging the assumption of judge neutrality in legal predictions.

Findings

Specialist models outperform generalist models in accuracy.

Judge-specific models capture stable individual decision patterns.

Empirical evidence supports the influence of judicial identity on outcomes.

Abstract

This study examines the role of human judges in legal decision-making by using machine learning to predict child physical custody outcomes in French appellate courts. Building on the legal realism-formalism debate, we test whether individual judges' decision-making patterns significantly influence case outcomes, challenging the assumption that judges are neutral variables that apply the law uniformly. To ensure compliance with French privacy laws, we implement a strict pseudonymization process. Our analysis uses 18,937 living arrangements rulings extracted from 10,306 cases. We compare models trained on individual judges' past rulings (specialist models) with a judge-agnostic model trained on aggregated data (generalist models). The prediction pipeline is a hybrid approach combining large language models (LLMs) for structured feature extraction and ML models for outcome prediction (RF, XGB and SVC). Our results show that specialist models consistently achieve higher predictive accuracy than the general model, with top-performing models reaching F1 scores as high as 92.85%, compared to the generalist model's 82.63% trained on 20x to 100x more samples. Specialist models capture stable individual patterns that are not transferable to other judges. In-Domain and Cross-Domain validity tests provide empirical support for legal realism, demonstrating that judicial identity plays a measurable role in legal outcomes. All data and code used will be made available.