AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark

TL;DR

This paper introduces LaborBench, a new benchmark dataset for evaluating AI's ability to simplify complex statutory language, and assesses the performance of large language models in this domain, revealing current limitations.

Contribution

It presents LaborBench and StateCodes datasets for systematic evaluation of AI in statutory simplification, and benchmarks LLMs, highlighting their current shortcomings.

Findings

LLMs show promise but lack sufficient accuracy for regulatory simplification.

LaborBench enables systematic evaluation of AI in legal code simplification.

Current models are not yet reliable as end-to-end solutions for legal code simplification.

Abstract

One of the emerging use cases of AI in law is for code simplification: streamlining, distilling, and simplifying complex statutory or regulatory language. One U.S. state has claimed to eliminate one third of its state code using AI. Yet we lack systematic evaluations of the accuracy, reliability, and risks of such approaches. We introduce LaborBench, a question-and-answer benchmark dataset designed to evaluate AI capabilities in this domain. We leverage a unique data source to create LaborBench: a dataset updated annually by teams of lawyers at the U.S. Department of Labor, who compile differences in unemployment insurance laws across 50 states for over 101 dimensions in a six-month process, culminating in a 200-page publication of tables. Inspired by our collaboration with one U.S. state to explore using large language models (LLMs) to simplify codes in this domain, where complexity is particularly acute, we transform the DOL publication into LaborBench. This provides a unique benchmark for AI capacity to conduct, distill, and extract realistic statutory and regulatory information. To assess the performance of retrieval augmented generation (RAG) approaches, we also compile StateCodes, a novel and comprehensive state statute and regulatory corpus of 8.7 GB, enabling much more systematic research into state codes. We then benchmark the performance of information retrieval and state-of-the-art large LLMs on this data and show that while these models are helpful as preliminary research for code simplification, the overall accuracy is far below the touted promises for LLMs as end-to-end pipelines for regulatory simplification.