Identifying Legal Holdings with LLMs: A Systematic Study of Performance, Scale, and Memorization

TL;DR

This study evaluates large language models' ability to identify legal case holdings, showing performance improves with size and is not solely due to memorization, highlighting their potential and limitations in legal applications.

Contribution

It systematically assesses LLMs on a legal benchmark, introduces a novel citation anonymization test, and demonstrates that performance scales with model size without fine-tuning.

Findings

Performance improves with model size.

Models perform well even without fine-tuning.

Performance remains strong after anonymization, indicating limited memorization.

Abstract

As large language models (LLMs) continue to advance in capabilities, it is essential to assess how they perform on established benchmarks. In this study, we present a suite of experiments to assess the performance of modern LLMs (ranging from 3B to 90B+ parameters) on CaseHOLD, a legal benchmark dataset for identifying case holdings. Our experiments demonstrate scaling effects - performance on this task improves with model size, with more capable models like GPT4o and AmazonNovaPro achieving macro F1 scores of 0.744 and 0.720 respectively. These scores are competitive with the best published results on this dataset, and do not require any technically sophisticated model training, fine-tuning or few-shot prompting. To ensure that these strong results are not due to memorization of judicial opinions contained in the training data, we develop and utilize a novel citation anonymization test that preserves semantic meaning while ensuring case names and citations are fictitious. Models maintain strong performance under these conditions (macro F1 of 0.728), suggesting the performance is not due to rote memorization. These findings demonstrate both the promise and current limitations of LLMs for legal tasks with important implications for the development and measurement of automated legal analytics and legal benchmarks.