Ebisu: Benchmarking Large Language Models in Japanese Finance

TL;DR

Ebisu is a new benchmark designed to evaluate large language models' understanding of Japanese financial language, highlighting current models' struggles with linguistic and cultural complexities.

Contribution

The paper introduces Ebisu, a culturally and linguistically grounded benchmark with expert-annotated tasks for Japanese financial NLP, and evaluates diverse LLMs revealing significant performance gaps.

Findings

State-of-the-art models perform poorly on Ebisu tasks.

Increasing model size offers limited improvements.

Domain-specific adaptation does not reliably enhance performance.

Abstract

Japanese finance combines agglutinative, head-final linguistic structure, mixed writing systems, and high-context communication norms that rely on indirect expression and implicit commitment, posing a substantial challenge for LLMs. We introduce Ebisu, a benchmark for native Japanese financial language understanding, comprising two linguistically and culturally grounded, expert-annotated tasks: JF-ICR, which evaluates implicit commitment and refusal recognition in investor-facing Q&A, and JF-TE, which assesses hierarchical extraction and ranking of nested financial terminology from professional disclosures. We evaluate a diverse set of open-source and proprietary LLMs spanning general-purpose, Japanese-adapted, and financial models. Results show that even state-of-the-art systems struggle on both tasks. While increased model scale yields limited improvements, language- and domain-specific adaptation does not reliably improve performance, leaving substantial gaps unresolved. Ebisu provides a focused benchmark for advancing linguistically and culturally grounded financial NLP. All datasets and evaluation scripts are publicly released.