Automating Legal Interpretation with LLMs: Retrieval, Generation, and Evaluation

TL;DR

This paper introduces ATRIE, a novel framework using large language models to automate legal interpretation tasks, including retrieval, interpretation, and evaluation, aiming to reduce reliance on time-consuming legal expert annotations.

Contribution

The paper presents ATRIE, the first framework that automates legal concept interpretation with LLMs, including a new evaluation method based on legal concept entailment.

Findings

Interpretations are comparable to legal experts in quality.

Automated approach improves efficiency of legal interpretation.

Slight accuracy gap remains but shows promising potential.

Abstract

Interpreting the law is always essential for the law to adapt to the ever-changing society. It is a critical and challenging task even for legal practitioners, as it requires meticulous and professional annotations and summarizations by legal experts, which are admittedly time-consuming and expensive to collect at scale. To alleviate the burden on legal experts, we propose a method for automated legal interpretation. Specifically, by emulating doctrinal legal research, we introduce a novel framework, ATRIE, to address Legal Concept Interpretation, a typical task in legal interpretation. ATRIE utilizes large language models (LLMs) to AuTomatically Retrieve concept-related information, Interpret legal concepts, and Evaluate generated interpretations, eliminating dependence on legal experts. ATRIE comprises a legal concept interpreter and a legal concept interpretation evaluator. The interpreter uses LLMs to retrieve relevant information from previous cases and interpret legal concepts. The evaluator uses performance changes on Legal Concept Entailment, a downstream task we propose, as a proxy of interpretation quality. Automated and multifaceted human evaluations indicate that the quality of our interpretations is comparable to those written by legal experts, with superior comprehensiveness and readability. Although there remains a slight gap in accuracy, it can already assist legal practitioners in improving the efficiency of legal interpretation.