LegalCiteBench: Evaluating Citation Reliability in Legal Language Models

TL;DR

LegalCiteBench is a new benchmark for evaluating legal language models' ability to accurately recover, verify, and match citations in a closed-book setting, revealing significant challenges and high rates of fabricated citations.

Contribution

The paper introduces LegalCiteBench, a comprehensive benchmark with 24K instances for studying citation-related tasks in legal LLMs, highlighting their limitations in authority generation.

Findings

Models perform poorly on citation retrieval and completion, scoring below 7/100.

High Misleading Answer Rates (MAR) over 94% indicate frequent incorrect citations.

Explicit uncertainty instructions reduce confident fabrication but do not improve correctness.

Abstract

Large language models (LLMs) are increasingly integrated into legal drafting and research workflows, where incorrect citations or fabricated precedents can cause serious professional harm. Existing legal benchmarks largely emphasize statutory reasoning, contract understanding, or general legal question answering, but they do not directly study a central common-law failure mode: when asked to provide case authorities without external grounding, models may return plausible-looking but incorrect citations or cases. We introduce LegalCiteBench, a benchmark for studying closed-book citation recovery, citation verification, and case matching in legal language models. LegalCiteBench contains approximately 24K evaluation instances constructed from 1,000 real U.S. judicial opinions from the Case Law Access Project. The benchmark covers five citation-centric tasks: citation retrieval, citation completion, citation error detection, case matching, and case verification and correction. Across 21 LLMs, exact citation recovery remains highly challenging in this closed-book setting: even the strongest models score below 7/100 on citation retrieval and completion. Within the evaluated models, scale and legal-domain pretraining provide limited gains and do not resolve this difficulty. Models also frequently provide concrete but incorrect or low-overlap authorities under our evaluation protocol, with Misleading Answer Rates (MAR) exceeding 94% for 20 of 21 evaluated models on retrieval-heavy tasks. A prompt-only abstention experiment shows that explicit uncertainty instructions reduce some confident fabrication but do not improve citation correctness. LegalCiteBench is intended as a diagnostic framework for studying authority generation failures, verification behavior, and abstention when external grounding is absent, incomplete, or bypassed.