PriceSeer: Evaluating Large Language Models in Real-Time Stock Prediction

TL;DR

PriceSeer is a new benchmark for evaluating large language models' ability to predict stock prices in real-time, considering diverse data sources and challenges like fake news, to assess their investment strategy potential.

Contribution

We introduce PriceSeer, a live, dynamic benchmark with comprehensive data and evaluation protocols for assessing LLMs in stock prediction tasks.

Findings

LLMs can generate investment strategies based on accurate price predictions.

Long-term predictions are less accurate, especially under fake news influence.

LLMs show potential but also vulnerabilities in stock prediction scenarios.

Abstract

Stock prediction, a subject closely related to people's investment activities in fully dynamic and live environments, has been widely studied. Current large language models (LLMs) have shown remarkable potential in various domains, exhibiting expert-level performance through advanced reasoning and contextual understanding. In this paper, we introduce PriceSeer, a live, dynamic, and data-uncontaminated benchmark specifically designed for LLMs performing stock prediction tasks. Specifically, PriceSeer includes 110 U.S. stocks from 11 industrial sectors, with each containing 249 historical data points. Our benchmark implements both internal and external information expansion, where LLMs receive extra financial indicators, news, and fake news to perform stock price prediction. We evaluate six cutting-edge LLMs under different prediction horizons, demonstrating their potential in generating investment strategies after obtaining accurate price predictions for different sectors. Additionally, we provide analyses of LLMs' suboptimal performance in long-term predictions, including the vulnerability to fake news and specific industries. The code and evaluation data will be open-sourced at https://github.com/BobLiang2113/PriceSeer.