Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task

TL;DR

This paper introduces a new dataset and expert evaluation framework to assess large language models' open-ended legal reasoning capabilities in the Japanese legal context, highlighting current limitations and challenges.

Contribution

It presents the first dataset for evaluating LLMs' legal reasoning on Japanese bar exam questions and provides expert analysis of model-generated responses.

Findings

LLMs show limitations in legal reasoning accuracy.

Models often hallucinate unsupported content.

Expert evaluations reveal specific reasoning challenges.

Abstract

Large language models (LLMs) have shown strong performance on legal benchmarks, including multiple-choice components of bar exams. However, their capacity for generating open-ended legal reasoning in realistic scenarios remains insufficiently explored. Notably, to our best knowledge, there are no prior studies or datasets addressing this issue in the Japanese context. This study presents the first dataset designed to evaluate the open-ended legal reasoning performance of LLMs within the Japanese jurisdiction. The dataset is based on the writing component of the Japanese bar examination, which requires examinees to identify multiple legal issues from long narratives and to construct structured legal arguments in free text format. Our key contribution is the manual evaluation of LLMs' generated responses by legal experts, which reveals limitations and challenges in legal reasoning. Moreover, we conducted a manual analysis of hallucinations to characterize when and how the models introduce content not supported by precedent or law. Our real exam questions, model-generated responses, and expert evaluations reveal the milestones of current LLMs in the Japanese legal domain. Our dataset and relevant resources will be available online.