The Legal Argument Reasoning Task in Civil Procedure

TL;DR

This paper introduces a complex legal reasoning NLP task with a new dataset based on U.S. civil procedure, highlighting the challenges in enabling models to accurately infer legal arguments.

Contribution

It presents a novel dataset and task for legal argument reasoning, and evaluates the performance of legal transformers on this challenging benchmark.

Findings

Legal transformers outperform random baselines

Legal reasoning remains a difficult challenge for current models

The dataset captures complex legal argumentation

Abstract

We present a new NLP task and dataset from the domain of the U.S. civil procedure. Each instance of the dataset consists of a general introduction to the case, a particular question, and a possible solution argument, accompanied by a detailed analysis of why the argument applies in that case. Since the dataset is based on a book aimed at law students, we believe that it represents a truly complex task for benchmarking modern legal language models. Our baseline evaluation shows that fine-tuning a legal transformer provides some advantage over random baseline models, but our analysis reveals that the actual ability to infer legal arguments remains a challenging open research question.