AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios

TL;DR

The paper introduces AppealCase, a comprehensive dataset and benchmark for analyzing civil case appeal scenarios, addressing a gap in LegalAI research by focusing on appellate review processes.

Contribution

It provides a large, annotated dataset for civil appeals and proposes five new LegalAI tasks, enabling systematic evaluation of models in appellate case analysis.

Findings

Models achieve less than 50% F1 on judgment reversal prediction

The dataset covers 91 civil case categories with detailed annotations

Current models struggle with the complexity of appeal scenarios

Abstract

Recent advances in LegalAI have primarily focused on individual case judgment analysis, often overlooking the critical appellate process within the judicial system. Appeals serve as a core mechanism for error correction and ensuring fair trials, making them highly significant both in practice and in research. To address this gap, we present the AppealCase dataset, consisting of 10,000 pairs of real-world, matched first-instance and second-instance documents across 91 categories of civil cases. The dataset also includes detailed annotations along five dimensions central to appellate review: judgment reversals, reversal reasons, cited legal provisions, claim-level decisions, and whether there is new information in the second instance. Based on these annotations, we propose five novel LegalAI tasks and conduct a comprehensive evaluation across 20 mainstream models. Experimental results reveal that all current models achieve less than 50% F1 scores on the judgment reversal prediction task, highlighting the complexity and challenge of the appeal scenario. We hope that the AppealCase dataset will spur further research in LegalAI for appellate case analysis and contribute to improving consistency in judicial decision-making.