Bye-bye, Bluebook? Automating Legal Procedure with Large Language Models

TL;DR

This paper evaluates the ability of large language models to accurately adhere to complex legal citation rules in The Bluebook, revealing limitations in their reliability for legal procedural automation.

Contribution

The study introduces a new dataset of 866 Bluebook citation tasks and systematically assesses multiple leading LLMs' compliance with intricate legal formatting rules.

Findings

Models achieve 69%-74% compliance without fine-tuning

In-context learning improves accuracy to 77%

Off-the-shelf LLMs are unreliable for high-fidelity legal procedures

Abstract

Legal practice requires careful adherence to procedural rules. In the United States, few are more complex than those found in The Bluebook: A Uniform System of Citation. Compliance with this system's 500+ pages of byzantine formatting instructions is the raison d'etre of thousands of student law review editors and the bete noire of lawyers everywhere. To evaluate whether large language models (LLMs) are able to adhere to the procedures of such a complicated system, we construct an original dataset of 866 Bluebook tasks and test flagship LLMs from OpenAI, Anthropic, Google, Meta, and DeepSeek. We show (1) that these models produce fully compliant Bluebook citations only 69%-74% of the time and (2) that in-context learning on the Bluebook's underlying system of rules raises accuracy only to 77%. These results caution against using off-the-shelf LLMs to automate aspects of the law where fidelity to procedure is paramount.