Evaluation and Benchmarking Suite for Financial Large Language Models and Agents

TL;DR

This paper introduces a comprehensive evaluation and benchmarking suite for financial large language models and agents, aiming to improve their reliability, governance, and application in the financial industry.

Contribution

It presents an open platform with evaluation tools, governance frameworks, and leaderboards specifically designed for FinLLMs and FinAgents, advancing financial AI research and deployment.

Findings

Development of an evaluation pipeline and governance framework

Launch of a FinLLM Leaderboard with HuggingFace

Facilitation of quantitative and qualitative analysis of FinLLMs and FinAgents

Abstract

Over the past three years, the financial services industry has witnessed Large Language Models (LLMs) and agents transitioning from the exploration stage to readiness and governance stages. Financial large language models (FinLLMs), such as open FinGPT and proprietary BloombergGPT , have great potential in financial applications, including retrieving real-time data, tutoring, analyzing sentiment of social media, analyzing SEC filings, and agentic trading. However, general-purpose LLMs and agents lack financial expertise and often struggle to handle complex financial reasoning. This paper presents an evaluation and benchmarking suite that covers the lifecycle of FinLLMs and FinAgents. This suite led by SecureFinAI Lab includes an evaluation pipeline and a governance framework collaborating with Linux Foundation and PyTorch Foundation, a FinLLM Leaderboard with HuggingFace, an AgentOps framework with Red Hat, and a documentation website with Rensselear Center of Open Source. Our collaborative development evolves through three stages: FinLLM Exploration (2023), FinLLM Readiness (2024), and FinAI Governance (2025). The proposed suite serves as an open platform that enables researchers and practitioners to perform both quantitative and qualitative analysis of different FinLLMs and FinAgents, fostering a more robust and reliable FinAI ecosystem.