FinGAIA: A Chinese Benchmark for AI Agents in Real-World Financial Domain

TL;DR

FinGAIA is a comprehensive benchmark with 407 financial tasks designed to evaluate AI agents' abilities in real-world financial scenarios, highlighting current limitations and guiding future research.

Contribution

This paper introduces the first detailed benchmark for AI agents in finance, covering multiple sub-domains and hierarchical task levels to assess practical capabilities.

Findings

ChatGPT achieved 48.9% accuracy in zero-shot evaluation.

AI agents lag behind financial experts by over 35 percentage points.

Identified key failure patterns to guide future improvements.

Abstract

The booming development of AI agents presents unprecedented opportunities for automating complex tasks across various domains. However, their multi-step, multi-tool collaboration capabilities in the financial sector remain underexplored. This paper introduces FinGAIA, an end-to-end benchmark designed to evaluate the practical abilities of AI agents in the financial domain. FinGAIA comprises 407 meticulously crafted tasks, spanning seven major financial sub-domains: securities, funds, banking, insurance, futures, trusts, and asset management. These tasks are organized into three hierarchical levels of scenario depth: basic business analysis, asset decision support, and strategic risk management. We evaluated 10 mainstream AI agents in a zero-shot setting. The best-performing agent, ChatGPT, achieved an overall accuracy of 48.9\%, which, while superior to non-professionals, still lags financial experts by over 35 percentage points. Error analysis has revealed five recurring failure patterns: Cross-modal Alignment Deficiency, Financial Terminological Bias, Operational Process Awareness Barrier, among others. These patterns point to crucial directions for future research. Our work provides the first agent benchmark closely related to the financial domain, aiming to objectively assess and promote the development of agents in this crucial field. Partial data is available at https://github.com/SUFE-AIFLM-Lab/FinGAIA.