MultiJustice: A Chinese Dataset for Multi-Party, Multi-Charge Legal Prediction

TL;DR

This paper introduces MultiJustice, a new Chinese dataset for multi-party, multi-charge legal prediction, and evaluates large language models across various complex legal scenarios to understand their performance and challenges.

Contribution

The paper presents a novel dataset for multi-party, multi-charge legal prediction and systematically evaluates LLMs on complex legal scenarios, highlighting their strengths and limitations.

Findings

Multi-defendant, multi-charge scenario (S4) is the most challenging for models.

Model performance varies significantly across different legal scenarios.

Certain models like InternLM2 and Lawformer show notable differences in F1-score and LogD across scenarios.

Abstract

Legal judgment prediction offers a compelling method to aid legal practitioners and researchers. However, the research question remains relatively under-explored: Should multiple defendants and charges be treated separately in LJP? To address this, we introduce a new dataset namely multi-person multi-charge prediction (MPMCP), and seek the answer by evaluating the performance of several prevailing legal large language models (LLMs) on four practical legal judgment scenarios: (S1) single defendant with a single charge, (S2) single defendant with multiple charges, (S3) multiple defendants with a single charge, and (S4) multiple defendants with multiple charges. We evaluate the dataset across two LJP tasks, i.e., charge prediction and penalty term prediction. We have conducted extensive experiments and found that the scenario involving multiple defendants and multiple charges (S4) poses the greatest challenges, followed by S2, S3, and S1. The impact varies significantly depending on the model. For example, in S4 compared to S1, InternLM2 achieves approximately 4.5% lower F1-score and 2.8% higher LogD, while Lawformer demonstrates around 19.7% lower F1-score and 19.0% higher LogD. Our dataset and code are available at https://github.com/lololo-xiao/MultiJustice-MPMCP.