AI Analyst: Framework and Comprehensive Evaluation of Large Language Models for Financial Time Series Report Generation

TL;DR

This paper presents a comprehensive framework for evaluating large language models in generating financial reports from time series data, emphasizing factual accuracy and reasoning through an innovative highlighting system.

Contribution

It introduces a novel framework with prompt engineering, model selection, and an automated highlighting system for detailed evaluation of LLMs in financial report generation.

Findings

LLMs can produce coherent financial reports from time series data

The highlighting system effectively categorizes information sources in reports

Models demonstrate varying levels of factual grounding and reasoning capabilities

Abstract

This paper explores the potential of large language models (LLMs) to generate financial reports from time series data. We propose a framework encompassing prompt engineering, model selection, and evaluation. We introduce an automated highlighting system to categorize information within the generated reports, differentiating between insights derived directly from time series data, stemming from financial reasoning, and those reliant on external knowledge. This approach aids in evaluating the factual grounding and reasoning capabilities of the models. Our experiments, utilizing both data from the real stock market indices and synthetic time series, demonstrate the capability of LLMs to produce coherent and informative financial reports.