Questioning Biases in Case Judgment Summaries: Legal Datasets or Large Language Models?

TL;DR

This paper investigates biases in legal case judgment summaries generated by datasets and large language models, highlighting concerns about fairness and implications for justice systems.

Contribution

It provides an analysis of biases related to gender, race, crime against women, country, and religion in LLM-generated legal summaries, revealing significant biases.

Findings

Biases are present in LLM-generated summaries.

Biases relate to gender, race, and religion keywords.

Further research needed to understand bias origins.

Abstract

The evolution of legal datasets and the advent of large language models (LLMs) have significantly transformed the legal field, particularly in the generation of case judgment summaries. However, a critical concern arises regarding the potential biases embedded within these summaries. This study scrutinizes the biases present in case judgment summaries produced by legal datasets and large language models. The research aims to analyze the impact of biases on legal decision making. By interrogating the accuracy, fairness, and implications of biases in these summaries, this study contributes to a better understanding of the role of technology in legal contexts and the implications for justice systems worldwide. In this study, we investigate biases wrt Gender-related keywords, Race-related keywords, Keywords related to crime against women, Country names and religious keywords. The study shows interesting evidences of biases in the outputs generated by the large language models and pre-trained abstractive summarization models. The reasoning behind these biases needs further studies.