ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US

TL;DR

This paper introduces a new challenging dataset for legal judgment prediction in US class action cases, highlighting the difficulty of the task and demonstrating the effectiveness of a Longformer model over human experts.

Contribution

It releases the first US common law class action dataset with complaints as input and provides a benchmark showing model performance surpassing human accuracy.

Findings

Longformer outperforms human experts with 63% accuracy.

Humans only achieve 53% accuracy on this dataset.

The model is better calibrated than human predictions.

Abstract

The research field of Legal Natural Language Processing (NLP) has been very active recently, with Legal Judgment Prediction (LJP) becoming one of the most extensively studied tasks. To date, most publicly released LJP datasets originate from countries with civil law. In this work, we release, for the first time, a challenging LJP dataset focused on class action cases in the US. It is the first dataset in the common law system that focuses on the harder and more realistic task involving the complaints as input instead of the often used facts summary written by the court. Additionally, we study the difficulty of the task by collecting expert human predictions, showing that even human experts can only reach 53% accuracy on this dataset. Our Longformer model clearly outperforms the human baseline (63%), despite only considering the first 2,048 tokens. Furthermore, we perform a detailed error analysis and find that the Longformer model is significantly better calibrated than the human experts. Finally, we publicly release the dataset and the code used for the experiments.