FIRE: A Comprehensive Benchmark for Financial Intelligence and Reasoning Evaluation

TL;DR

FIRE is a comprehensive benchmark for evaluating large language models' financial knowledge and reasoning abilities through theoretical exams and practical financial scenarios, aiding future research.

Contribution

Introduces FIRE, a new benchmark combining theoretical financial exams and real-world scenarios to assess LLMs' financial intelligence and reasoning capabilities.

Findings

State-of-the-art LLMs show varying performance across financial tasks.

The benchmark reveals current limitations in LLMs' financial reasoning.

Our financial-domain model XuanYuan 4.0 performs strongly on the benchmark.

Abstract

We introduce FIRE, a comprehensive benchmark designed to evaluate both the theoretical financial knowledge of LLMs and their ability to handle practical business scenarios. For theoretical assessment, we curate a diverse set of examination questions drawn from widely recognized financial qualification exams, enabling evaluation of LLMs deep understanding and application of financial knowledge. In addition, to assess the practical value of LLMs in real-world financial tasks, we propose a systematic evaluation matrix that categorizes complex financial domains and ensures coverage of essential subdomains and business activities. Based on this evaluation matrix, we collect 3,000 financial scenario questions, consisting of closed-form decision questions with reference answers and open-ended questions evaluated by predefined rubrics. We conduct comprehensive evaluations of state-of-the-art LLMs on the FIRE benchmark, including XuanYuan 4.0, our latest financial-domain model, as a strong in-domain baseline. These results enable a systematic analysis of the capability boundaries of current LLMs in financial applications. We publicly release the benchmark questions and evaluation code to facilitate future research.