Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models

TL;DR

This paper introduces MSLR, a Chinese multi-step legal reasoning dataset based on real judicial decisions, and demonstrates how self-initiated Chain-of-Thought prompts improve LLM reasoning in complex legal tasks.

Contribution

It presents the first Chinese legal reasoning dataset grounded in real cases and proposes a scalable annotation pipeline, also showing the effectiveness of autonomous Chain-of-Thought prompts.

Findings

LLMs show moderate performance on complex legal reasoning tasks.

Self-initiated Chain-of-Thought prompts outperform human-designed prompts.

MSLR dataset and annotation pipeline facilitate future research in legal reasoning.

Abstract

Large language models (LLMs) have demonstrated strong reasoning abilities across specialized domains, motivating research into their application to legal reasoning. However, existing legal benchmarks often conflate factual recall with genuine inference, fragment the reasoning process, and overlook the quality of reasoning. To address these limitations, we introduce MSLR, the first Chinese multi-step legal reasoning dataset grounded in real-world judicial decision making. MSLR adopts the IRAC framework (Issue, Rule, Application, Conclusion) to model structured expert reasoning from official legal documents. In addition, we design a scalable Human-LLM collaborative annotation pipeline that efficiently produces fine-grained step-level reasoning annotations and provides a reusable methodological framework for multi-step reasoning datasets. Evaluation of multiple LLMs on MSLR shows only moderate performance, highlighting the challenges of adapting to complex legal reasoning. Further experiments demonstrate that Self-Initiated Chain-of-Thought prompts generated by models autonomously improve reasoning coherence and quality, outperforming human-designed prompts. MSLR contributes to advancing LLM reasoning and Chain-of-Thought strategies and offers open resources for future research. The dataset and code are available at https://github.com/yuwenhan07/MSLR-Bench and https://law.sjtu.edu.cn/flszyjzx/index.html.