Chinese Labor Law Large Language Model Benchmark

TL;DR

This paper introduces LabourLawLLM, a specialized Chinese labor law large language model, and LabourLawBench, a comprehensive benchmark for evaluating legal AI performance in labor law tasks, demonstrating superior results over general models.

Contribution

The paper presents a tailored legal LLM for Chinese labor law and a new benchmark, advancing specialized legal AI capabilities and evaluation methods.

Findings

LabourLawLLM outperforms general-purpose models in labor law tasks.

The benchmark effectively evaluates legal LLMs across diverse tasks.

Methodology scalable to other legal subfields.

Abstract

Recent advances in large language models (LLMs) have led to substantial progress in domain-specific applications, particularly within the legal domain. However, general-purpose models such as GPT-4 often struggle with specialized subdomains that require precise legal knowledge, complex reasoning, and contextual sensitivity. To address these limitations, we present LabourLawLLM, a legal large language model tailored to Chinese labor law. We also introduce LabourLawBench, a comprehensive benchmark covering diverse labor-law tasks, including legal provision citation, knowledge-based question answering, case classification, compensation computation, named entity recognition, and legal case analysis. Our evaluation framework combines objective metrics (e.g., ROUGE-L, accuracy, F1, and soft-F1) with subjective assessment based on GPT-4 scoring. Experiments show that LabourLawLLM consistently outperforms general-purpose and existing legal-specific LLMs across task categories. Beyond labor law, our methodology provides a scalable approach for building specialized LLMs in other legal subfields, improving accuracy, reliability, and societal value of legal AI applications.