Lawma: The Power of Specialization for Legal Annotation

TL;DR

This paper introduces CaselawQA, a benchmark for legal annotation, and shows that fine-tuned open-source models outperform commercial large language models like GPT-4.5 in legal text annotation tasks.

Contribution

It presents a new benchmark for legal annotation and demonstrates the effectiveness of fine-tuned open-source models over commercial models.

Findings

Commercial models have variable accuracy in legal annotation.

Fine-tuned models outperform commercial models with limited labeled data.

A few hundred labeled examples suffice for higher accuracy.

Abstract

Annotation and classification of legal text are central components of empirical legal research. Traditionally, these tasks are often delegated to trained research assistants. Motivated by the advances in language modeling, empirical legal scholars are increasingly turning to prompting commercial models, hoping that it will alleviate the significant cost of human annotation. Despite growing use, our understanding of how to best utilize large language models for legal annotation remains limited. To bridge this gap, we introduce CaselawQA, a benchmark comprising 260 legal annotation tasks, nearly all new to the machine learning community. We demonstrate that commercial models, such as GPT-4.5 and Claude 3.7 Sonnet, achieve non-trivial yet highly variable accuracy, generally falling short of the performance required for legal work. We then demonstrate that small, lightly fine-tuned models outperform commercial models. A few hundred to a thousand labeled examples are usually enough to achieve higher accuracy. Our work points to a viable alternative to the predominant practice of prompting commercial models. For concrete legal annotation tasks with some available labeled data, researchers are likely better off using a fine-tuned open-source model.