FLAME: Financial Large-Language Model Assessment and Metrics Evaluation

TL;DR

The paper introduces FLAME, a comprehensive evaluation system for Chinese financial large-language models, including benchmarks for certifications and business scenarios, to assess and advance their capabilities.

Contribution

It presents a new, detailed evaluation framework and benchmarks specifically designed for Chinese financial LLMs, covering certifications and real-world financial scenarios.

Findings

Baichuan4-Finance outperforms other LLMs in most tasks.

The evaluation system covers 14 financial certifications and 31 financial scenarios.

FLAME facilitates systematic assessment and development of financial LLMs in Chinese.

Abstract

LLMs have revolutionized NLP and demonstrated potential across diverse domains. More and more financial LLMs have been introduced for finance-specific tasks, yet comprehensively assessing their value is still challenging. In this paper, we introduce FLAME, a comprehensive financial LLMs evaluation system in Chinese, which includes two core evaluation benchmarks: FLAME-Cer and FLAME-Sce. FLAME-Cer covers 14 types of authoritative financial certifications, including CPA, CFA, and FRM, with a total of approximately 16,000 carefully selected questions. All questions have been manually reviewed to ensure accuracy and representativeness. FLAME-Sce consists of 10 primary core financial business scenarios, 21 secondary financial business scenarios, and a comprehensive evaluation set of nearly 100 tertiary financial application tasks. We evaluate 6 representative LLMs, including GPT-4o, GLM-4, ERNIE-4.0, Qwen2.5, XuanYuan3, and the latest Baichuan4-Finance, revealing Baichuan4-Finance excels other LLMs in most tasks. By establishing a comprehensive and professional evaluation system, FLAME facilitates the advancement of financial LLMs in Chinese contexts. Instructions for participating in the evaluation are available on GitHub: https://github.com/FLAME-ruc/FLAME.