Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization

TL;DR

This paper introduces Gavel-Ref, a comprehensive evaluation framework for assessing large language models on complex long-context legal summarization tasks, revealing current limitations and proposing an autonomous agent approach to improve efficiency.

Contribution

The paper presents Gavel-Ref, a multi-value checklist evaluation framework for legal summarization, and Gavel-Agent, an autonomous tool-enhanced LLM system that reduces token usage with minimal performance loss.

Findings

Even the best models score only around 50 on Gavel-Ref.

Models excel at simple checklist items but struggle with complex ones.

Gavel-Agent reduces token usage by 36% with only 7% performance drop.

Abstract

Large language models (LLMs) now support contexts of up to 1M tokens, but their effectiveness on complex long-context tasks remains unclear. In this paper, we study multi-document legal case summarization, where a single case often spans many documents totaling 100K-500K tokens. We introduce Gavel-Ref, a reference-based evaluation framework with multi-value checklist evaluation over 26 items, as well as residual fact and writing-style evaluations. Using Gavel-Ref, we go beyond the single aggregate scores reported in prior work and systematically evaluate 12 frontier LLMs on 100 legal cases ranging from 32K to 512K tokens, primarily from 2025. Our results show that even the strongest model, Gemini 2.5 Pro, achieves only around 50 of $S_{\text{Gavel-Ref}}$, highlighting the difficulty of the task. Models perform well on simple checklist items (e.g., filing date) but struggle on multi-value or rare ones such as settlements and monitor reports. As LLMs continue to improve and may surpass human-written summaries -- making human references less reliable -- we develop Gavel-Agent, an efficient and autonomous agent scaffold that equips LLMs with six tools to navigate and extract checklists directly from case documents. With Qwen3, Gavel-Agent reduces token usage by 36% while resulting in only a 7% drop in $S_{\text{checklist}}$ compared to end-to-end extraction with GPT-4.1.