Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models

TL;DR

Golden Touchstone is a new bilingual benchmark for evaluating financial large language models across multiple tasks in Chinese and English, providing a standardized and comprehensive assessment tool for the financial AI community.

Contribution

It introduces Golden Touchstone, a comprehensive bilingual benchmark for financial LLMs, and presents Touchstone-GPT, a new financial LLM trained specifically for this benchmark.

Findings

Major models show varied strengths and weaknesses in financial tasks.

Touchstone-GPT performs well on the benchmark but has limitations in certain areas.

The benchmark reveals specific challenges in bilingual financial NLP evaluation.

Abstract

As large language models (LLMs) increasingly permeate the financial sector, there is a pressing need for a standardized method to comprehensively assess their performance. Existing financial benchmarks often suffer from limited language and task coverage, low-quality datasets, and inadequate adaptability for LLM evaluation. To address these limitations, we introduce Golden Touchstone, a comprehensive bilingual benchmark for financial LLMs, encompassing eight core financial NLP tasks in both Chinese and English. Developed from extensive open-source data collection and industry-specific demands, this benchmark thoroughly assesses models' language understanding and generation capabilities. Through comparative analysis of major models such as GPT-4o, Llama3, FinGPT, and FinMA, we reveal their strengths and limitations in processing complex financial information. Additionally, we open-source Touchstone-GPT, a financial LLM trained through continual pre-training and instruction tuning, which demonstrates strong performance on the bilingual benchmark but still has limitations in specific tasks. This research provides a practical evaluation tool for financial LLMs and guides future development and optimization. The source code for Golden Touchstone and model weight of Touchstone-GPT have been made publicly available at https://github.com/IDEA-FinAI/Golden-Touchstone.