LexTime: A Benchmark for Temporal Ordering of Legal Events

TL;DR

LexTime is a specialized benchmark dataset for evaluating large language models' ability to understand and order legal events in temporal sequences, addressing a gap in legal NLP evaluation tools.

Contribution

The paper introduces LexTime, a novel dataset with annotated legal event pairs, and provides insights into LLMs' performance and challenges in legal temporal reasoning.

Findings

LLMs perform better on legal event ordering than narrative texts (+10.5%)

Longer contexts and implicit events improve accuracy (up to 80.8%)

Legal linguistic complexities and nested clauses pose challenges

Abstract

Understanding temporal relationships and accurately reconstructing the event timeline is important for case law analysis, compliance monitoring, and legal summarization. However, existing benchmarks lack specialized language evaluation, leaving a gap in understanding how LLMs handle event ordering in legal contexts. We introduce LexTime, a dataset designed to evaluate LLMs' event ordering capabilities in legal language, consisting of 512 instances from U.S. Federal Complaints with annotated event pairs and their temporal relations. Our findings show that (1) LLMs are more accurate on legal event ordering than on narrative texts (up to +10.5%); (2) longer input contexts and implicit events boost accuracy, reaching 80.8% for implicit-explicit event pairs; (3) legal linguistic complexities and nested clauses remain a challenge. While performance is promising, specific features of legal texts remain a bottleneck for legal temporal event reasoning, and we propose concrete modeling directions to better address them.