MTFinEval:A Multi-domain Chinese Financial Benchmark with Eurypalynous questions

TL;DR

This paper introduces MTFinEval, a comprehensive benchmark based on fundamental economic knowledge from textbooks and exams, to evaluate LLMs' generalization and theoretical understanding across multiple economic disciplines.

Contribution

The paper presents MTFinEval, a new multi-domain Chinese financial benchmark built on foundational economic questions, addressing limitations of existing benchmarks in assessing LLMs' basic knowledge and generalization.

Findings

LLMs perform poorly on MTFinEval, indicating gaps in basic economic knowledge.

The benchmark covers six major economic disciplines for comprehensive evaluation.

Results highlight the need for improving LLMs' foundational understanding.

Abstract

With the emergence of more and more economy-specific LLMS, how to measure whether they can be safely invested in production becomes a problem. Previous research has primarily focused on evaluating the performance of LLMs within specific application scenarios. However, these benchmarks cannot reflect the theoretical level and generalization ability, and the backward datasets are increasingly unsuitable for problems in real scenarios. In this paper, we have compiled a new benchmark, MTFinEval, focusing on the LLMs' basic knowledge of economics, which can always be used as a basis for judgment. To examine only theoretical knowledge as much as possible, MTFinEval is build with foundational questions from university textbooks,and exam papers in economics and management major. Aware of the overall performance of LLMs do not depend solely on one subdiscipline of economics, MTFinEval comprise 360 questions refined from six major disciplines of economics, and reflect capabilities more comprehensively. Experiment result shows all LLMs perform poorly on MTFinEval, which proves that our benchmark built on basic knowledge is very successful. Our research not only offers guidance for selecting the appropriate LLM for specific use cases, but also put forward increase the rigor reliability of LLMs from the basics.