Construction of a Japanese Financial Benchmark for Large Language Models

TL;DR

This paper introduces a Japanese financial domain benchmark for large language models, demonstrating its effectiveness in evaluating model performance and highlighting GPT-4's superior capabilities.

Contribution

It presents a new domain-specific benchmark for Japanese financial LLMs and validates its ability to differentiate model performance across various levels.

Findings

GPT-4 outperforms other models in the benchmark

The benchmark effectively distinguishes performance differences among models

The constructed tasks cover a range of difficulties to evaluate models comprehensively

Abstract

With the recent development of large language models (LLMs), models that focus on certain domains and languages have been discussed for their necessity. There is also a growing need for benchmarks to evaluate the performance of current LLMs in each domain. Therefore, in this study, we constructed a benchmark comprising multiple tasks specific to the Japanese and financial domains and performed benchmark measurements on some models. Consequently, we confirmed that GPT-4 is currently outstanding, and that the constructed benchmarks function effectively. According to our analysis, our benchmark can differentiate benchmark scores among models in all performance ranges by combining tasks with different difficulties.