A Reasoning-Focused Legal Retrieval Benchmark

TL;DR

This paper introduces two realistic legal RAG benchmarks, Bar Exam QA and Housing Statute QA, to evaluate retrieval-augmented LLMs in complex legal question-answering tasks, highlighting ongoing challenges.

Contribution

It presents novel legal RAG benchmarks that mimic real-world legal research, facilitating evaluation of retrieval-augmented LLMs in legal contexts.

Findings

Existing retriever pipelines perform poorly on legal RAG tasks.

Legal RAG remains a challenging application for current models.

Benchmarks reflect real-world legal research complexity.

Abstract

As the legal community increasingly examines the use of large language models (LLMs) for various legal applications, legal AI developers have turned to retrieval-augmented LLMs ("RAG" systems) to improve system performance and robustness. An obstacle to the development of specialized RAG systems is the lack of realistic legal RAG benchmarks which capture the complexity of both legal retrieval and downstream legal question-answering. To address this, we introduce two novel legal RAG benchmarks: Bar Exam QA and Housing Statute QA. Our tasks correspond to real-world legal research tasks, and were produced through annotation processes which resemble legal research. We describe the construction of these benchmarks and the performance of existing retriever pipelines. Our results suggest that legal RAG remains a challenging application, thus motivating future research.