CitaLaw: Enhancing LLM with Citations in Legal Domain

TL;DR

CitaLaw introduces a benchmark and evaluation framework for assessing legal language models' ability to generate legally sound responses with accurate citations, improving response quality and alignment with legal standards.

Contribution

This paper presents the first benchmark for evaluating LLMs' legal citation accuracy and introduces syllogism-inspired evaluation methods for legal alignment assessment.

Findings

Legal references improve LLM response quality

Syllogism-based evaluation aligns well with human judgments

Legal-specific LLMs benefit from citation integration

Abstract

In this paper, we propose CitaLaw, the first benchmark designed to evaluate LLMs' ability to produce legally sound responses with appropriate citations. CitaLaw features a diverse set of legal questions for both laypersons and practitioners, paired with a comprehensive corpus of law articles and precedent cases as a reference pool. This framework enables LLM-based systems to retrieve supporting citations from the reference corpus and align these citations with the corresponding sentences in their responses. Moreover, we introduce syllogism-inspired evaluation methods to assess the legal alignment between retrieved references and LLM-generated responses, as well as their consistency with user questions. Extensive experiments on 2 open-domain and 7 legal-specific LLMs demonstrate that integrating legal references substantially enhances response quality. Furthermore, our proposed syllogism-based evaluation method exhibits strong agreement with human judgments.