FinMaster: A Holistic Benchmark for Mastering Full-Pipeline Financial Workflows with LLMs

TL;DR

FinMaster is a comprehensive benchmark designed to evaluate large language models across full-pipeline financial workflows, highlighting significant gaps in reasoning and accuracy that need addressing for real-world financial applications.

Contribution

This paper introduces FinMaster, the first holistic benchmark covering diverse financial tasks and workflows to systematically assess LLM capabilities in finance.

Findings

LLMs show high accuracy on simple financial tasks but struggle with complex multi-step reasoning.

Accuracy drops from over 90% on basic tasks to around 40% on complex scenarios.

Error propagation significantly impacts multi-metric financial calculations.

Abstract

Financial tasks are pivotal to global economic stability; however, their execution faces challenges including labor intensive processes, low error tolerance, data fragmentation, and tool limitations. Although large language models (LLMs) have succeeded in various natural language processing tasks and have shown potential in automating workflows through reasoning and contextual understanding, current benchmarks for evaluating LLMs in finance lack sufficient domain-specific data, have simplistic task design, and incomplete evaluation frameworks. To address these gaps, this article presents FinMaster, a comprehensive financial benchmark designed to systematically assess the capabilities of LLM in financial literacy, accounting, auditing, and consulting. Specifically, FinMaster comprises three main modules: i) FinSim, which builds simulators that generate synthetic, privacy-compliant financial data for companies to replicate market dynamics; ii) FinSuite, which provides tasks in core financial domains, spanning 183 tasks of various types and difficulty levels; and iii) FinEval, which develops a unified interface for evaluation. Extensive experiments over state-of-the-art LLMs reveal critical capability gaps in financial reasoning, with accuracy dropping from over 90% on basic tasks to merely 40% on complex scenarios requiring multi-step reasoning. This degradation exhibits the propagation of computational errors, where single-metric calculations initially demonstrating 58% accuracy decreased to 37% in multimetric scenarios. To the best of our knowledge, FinMaster is the first benchmark that covers full-pipeline financial workflows with challenging tasks. We hope that FinMaster can bridge the gap between research and industry practitioners, driving the adoption of LLMs in real-world financial practices to enhance efficiency and accuracy.