LLMs on Trial: Evaluating Judicial Fairness for Large Language Models

TL;DR

This paper evaluates the fairness of large language models in judicial contexts, revealing widespread bias and inconsistency, and provides a comprehensive framework and toolkit for assessing and improving LLM fairness in high-stakes decision-making.

Contribution

It introduces a novel framework for measuring LLM judicial fairness, constructs a large dataset, and develops evaluation metrics and tools to analyze and address biases in LLMs used as judges.

Findings

LLMs exhibit pervasive inconsistency, bias, and imbalanced inaccuracy in judicial tasks.

Biases are more pronounced on demographic labels than on procedural or substantive labels.

Increasing model accuracy can worsen biases, while adjusting temperature influences fairness.

Abstract

Large Language Models (LLMs) are increasingly used in high-stakes fields where their decisions impact rights and equity. However, LLMs' judicial fairness and implications for social justice remain underexplored. When LLMs act as judges, the ability to fairly resolve judicial issues is a prerequisite to ensure their trustworthiness. Based on theories of judicial fairness, we construct a comprehensive framework to measure LLM fairness, leading to a selection of 65 labels and 161 corresponding values. Applying this framework to the judicial system, we compile an extensive dataset, JudiFair, comprising 177,100 unique case facts. To achieve robust statistical inference, we develop three evaluation metrics, inconsistency, bias, and imbalanced inaccuracy, and introduce a method to assess the overall fairness of multiple LLMs across various labels. Through experiments with 16 LLMs, we uncover pervasive inconsistency, bias, and imbalanced inaccuracy across models, underscoring severe LLM judicial unfairness. Particularly, LLMs display notably more pronounced biases on demographic labels, with slightly less bias on substance labels compared to procedure ones. Interestingly, increased inconsistency correlates with reduced biases, but more accurate predictions exacerbate biases. While we find that adjusting the temperature parameter can influence LLM fairness, model size, release date, and country of origin do not exhibit significant effects on judicial fairness. Accordingly, we introduce a publicly available toolkit containing all datasets and code, designed to support future research in evaluating and improving LLM fairness.