FinS-Pilot: A Benchmark for Online Financial RAG System

TL;DR

FinS-Pilot is a new benchmark designed to evaluate online financial retrieval-augmented generation systems, integrating real-time data and intent classification to improve financial NLP model assessment.

Contribution

It introduces a novel financial RAG benchmark with real-world data and a comprehensive evaluation framework for financial NLP models.

Findings

Effective in identifying suitable models for financial applications

Addresses data confidentiality issues in financial RAG benchmarking

Provides a curated dataset and evaluation tools for financial NLP research

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities across various professional domains, with their performance typically evaluated through standardized benchmarks. In the financial field, the stringent demands for professional accuracy and real-time data processing often necessitate the use of retrieval-augmented generation (RAG) techniques. However, the development of financial RAG benchmarks has been constrained by data confidentiality issues and the lack of dynamic data integration. To address this issue, we introduce FinS-Pilot, a novel benchmark for evaluating RAG systems in online financial applications. Constructed from real-world financial assistant interactions, our benchmark incorporates both real-time API data and text data, organized through an intent classification framework covering critical financial domains. The benchmark enables comprehensive evaluation of financial assistants' capabilities in handling both static knowledge and time-sensitive market information.Through systematic experiments with multiple Chinese leading LLMs, we demonstrate FinS-Pilot's effectiveness in identifying models suitable for financial applications while addressing the current gap in specialized evaluation tools for the financial domain. Our work contributes both a practical evaluation framework and a curated dataset to advance research in financial NLP systems. The code and dataset are accessible on GitHub.