Self-Verification is All You Need To Pass The Japanese Bar Examination

TL;DR

This paper demonstrates that a self-verification approach enables a large language model to pass the Japanese bar exam without modifying question formats, emphasizing the importance of format-faithful supervision and consistency checks.

Contribution

The study introduces a self-verification model trained on a new dataset that faithfully replicates the exam format, achieving passing scores on the actual Japanese bar exam without altering question structure or scoring.

Findings

The model exceeds the official passing score on the real exam.

Self-verification outperforms multi-agent inference and decomposition methods.

Format-faithful supervision is crucial for high-stakes professional reasoning.

Abstract

Despite rapid advances in large language models (LLMs), achieving reliable performance on highly professional and structured examinations remains a significant challenge. The Japanese bar examination is a particularly demanding benchmark, requiring not only advanced legal reasoning but also strict adherence to complex answer formats that involve joint evaluation of multiple propositions. While recent studies have reported improvements by decomposing such questions into simpler true--false judgments, these approaches have not been systematically evaluated under the original exam format and scoring scheme, leaving open the question of whether they truly capture exam-level competence. In this paper, we present a self-verification model trained on a newly constructed dataset that faithfully replicates the authentic format and evaluation scale of the exam. Our model is able to exceed the official passing score when evaluated on the actual exam scale, marking the first demonstration, to our knowledge, of an LLM passing the Japanese bar examination without altering its original question structure or scoring rules. We further conduct extensive comparisons with alternative strategies, including multi-agent inference and decomposition-based supervision, and find that these methods fail to achieve comparable performance. Our results highlight the importance of format-faithful supervision and consistency verification, and suggest that carefully designed single-model approaches can outperform more complex systems in high-stakes professional reasoning tasks. Our dataset and codes are publicly available.