CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval

TL;DR

CaseFacts is a new benchmark dataset designed to evaluate legal fact-checking and precedent retrieval systems by challenging them to verify claims against U.S. Supreme Court cases, considering semantic and temporal complexities.

Contribution

This paper introduces CaseFacts, a novel benchmark dataset for legal fact-checking that incorporates complex claim synthesis and a semantic similarity heuristic for verifying legal overrulings.

Findings

State-of-the-art LLMs find the task challenging.

Web search augmentation degrades performance due to noisy data.

The dataset enables research into legal fact verification systems.

Abstract

Automated Fact-Checking has largely focused on verifying general knowledge against static corpora, overlooking high-stakes domains like law where truth is evolving and technically complex. We introduce CaseFacts, a benchmark for verifying colloquial legal claims against U.S. Supreme Court precedents. Unlike existing resources that map formal texts to formal texts, CaseFacts challenges systems to bridge the semantic gap between layperson assertions and technical jurisprudence while accounting for temporal validity. The dataset consists of 6,294 claims categorized as Supported, Refuted, or Overruled. We construct this benchmark using a multi-stage pipeline that leverages Large Language Models (LLMs) to synthesize claims from expert case summaries, employing a novel semantic similarity heuristic to efficiently identify and verify complex legal overrulings. Experiments with state-of-the-art LLMs reveal that the task remains challenging; notably, augmenting models with unrestricted web search degrades performance compared to closed-book baselines due to the retrieval of noisy, non-authoritative precedents. We release CaseFacts to spur research into legal fact verification systems.