LexRel: Benchmarking Legal Relation Extraction for Chinese Civil Cases

TL;DR

This paper introduces LexRel, a comprehensive benchmark for legal relation extraction in Chinese civil law, highlighting current LLM limitations and potential improvements in legal AI tasks.

Contribution

It presents a new schema and benchmark for legal relations in Chinese civil cases, and evaluates LLM performance on this task.

Findings

Current LLMs have significant limitations in identifying civil legal relations.

Explicitly incorporating legal relation information improves downstream legal AI tasks.

LexRel provides a structured dataset for Chinese civil law relation extraction.

Abstract

Legal relations serve as an important analytical framework for dispute resolution in civil cases. However, legal relations in Chinese civil cases remain underexplored in the field of legal AI, largely due to the absence of comprehensive schemas. In this work, we first introduce a comprehensive schema for legal relations in civil cases, which contains a hierarchical taxonomy and definitions of arguments. Based on this schema, we formulate a legal relation extraction task and present LexRel, an expert-annotated benchmark for legal relation extraction in the Chinese civil law domain. We use LexRel to evaluate state-of-the-art large language models (LLMs) on legal relation extraction, showing that current LLMs exhibit significant limitations in accurately identifying civil legal relations. Furthermore, we demonstrate that explicitly incorporating information about legal relations leads to promising performance gains on other downstream legal AI tasks.