LegalRikai: Open Benchmark -- Benchmark for Complex Japanese Corporate Legal Tasks

TL;DR

LegalRikai: Open Benchmark is a new, expert-created benchmark with complex Japanese legal tasks, evaluated by humans and LLMs, highlighting challenges in document editing and automated assessment.

Contribution

It introduces a legally grounded benchmark for Japanese corporate legal tasks and evaluates LLM performance and evaluation methods in this domain.

Findings

Automated evaluation aligns well with human judgment on clear criteria.

Model weaknesses in document-level editing are identified.

Automated evaluation can serve as an effective screening tool.

Abstract

This paper introduces LegalRikai: Open Benchmark, a new benchmark comprising four complex tasks that emulate Japanese corporate legal practices. The benchmark was created by legal professionals under the supervision of an attorney. This benchmark has 100 samples that require long-form, structured outputs, and we evaluated them against multiple practical criteria. We conducted both human and automated evaluations using leading LLMs, including GPT-5, Gemini 2.5 Pro, and Claude Opus 4.1. Our human evaluation revealed that abstract instructions prompted unnecessary modifications, highlighting model weaknesses in document-level editing that were missed by conventional short-text tasks. Furthermore, our analysis reveals that automated evaluation aligns well with human judgment on criteria with clear linguistic grounding, and assessing structural consistency remains a challenge. The result demonstrates the utility of automated evaluation as a screening tool when expert availability is limited. We propose a dataset evaluation framework to promote more practice-oriented research in the legal domain.