Benchmarking Legal RAG: The Promise and Limits of AI Statutory Surveys

TL;DR

This paper evaluates the performance of retrieval-augmented generation (RAG) models in legal AI, demonstrating that specialized tools like STARA outperform commercial platforms and analyzing the sources of errors to guide future system design.

Contribution

The study introduces a comprehensive benchmark for legal RAG, evaluates new tools including STARA and commercial platforms, and provides design principles for improving legal AI retrieval systems.

Findings

STARA achieves 83% accuracy, with an adjusted 92% considering attorney omissions.

Commercial tools like Westlaw AI and LexisNexis perform poorly, with 58% and 64% accuracy.

Error analysis reveals reasoning mistakes and retrieval failures, with some errors due to omissions by attorneys.

Abstract

Retrieval-augmented generation (RAG) offers significant potential for legal AI, yet systematic benchmarks are sparse. Prior work introduced LaborBench to benchmark RAG models based on ostensible ground truth from an exhaustive, multi-month, manual enumeration of all U.S. state unemployment insurance requirements by U.S. Department of Labor (DOL) attorneys. That prior work found poor performance of standard RAG (70% accuracy on Boolean tasks). Here, we assess three emerging tools not previously evaluated on LaborBench: the Statutory Research Assistant (STARA), a custom statutory research tool, and two commercial tools by Westlaw and LexisNexis marketing AI statutory survey capabilities. We make five main contributions. First, we show that STARA achieves substantial performance gains, boosting accuracy to 83%. Second, we show that commercial platforms fare poorly, with accuracy of 58% (Westlaw AI) and 64% (Lexis+ AI), even worse than standard RAG. Third, we conduct a comprehensive error analysis, comparing our outputs to those compiled by DOL attorneys, and document both reasoning errors, such as confusion between related legal concepts and misinterpretation of statutory exceptions, and retrieval failures, where relevant statutory provisions are not captured. Fourth, we discover that many apparent errors are actually significant omissions by DOL attorneys themselves, such that STARA's actual accuracy is 92%. Fifth, we chart the path forward for legal RAG through concrete design principles, offering actionable guidance for building AI systems capable of accurate multi-jurisdictional legal research.