Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts

TL;DR

This paper discusses recent progress and challenges in training large language models for event forecasting, proposing new methods and data strategies to achieve superforecaster-level performance and societal impact.

Contribution

It introduces novel training difficulties and solutions, and advocates for large-scale data use to advance event forecasting LLMs towards superforecaster-level accuracy.

Findings

State-of-the-art LLMs are approaching superforecaster performance.

Reinforcement learning improves forecasting accuracy.

Proposed data and training strategies can enhance model capabilities.

Abstract

Many recent papers have studied the development of superforecaster-level event forecasting LLMs. While methodological problems with early studies cast doubt on the use of LLMs for event forecasting, recent studies with improved evaluation methods have shown that state-of-the-art LLMs are gradually reaching superforecaster-level performance, and reinforcement learning has also been reported to improve future forecasting. Additionally, the unprecedented success of recent reasoning models and Deep Research-style models suggests that technology capable of greatly improving forecasting performance has been developed. Therefore, based on these positive recent trends, we argue that the time is ripe for research on large-scale training of superforecaster-level event forecasting LLMs. We discuss two key research directions: training methods and data acquisition. For training, we first introduce three difficulties of LLM-based event forecasting training: noisiness-sparsity, knowledge cut-off, and simple reward structure problems. Then, we present related ideas to mitigate these problems: hypothetical event Bayesian networks, utilizing poorly-recalled and counterfactual events, and auxiliary reward signals. For data, we propose aggressive use of market, public, and crawling datasets to enable large-scale training and evaluation. Finally, we explain how these technical advances could enable AI to provide predictive intelligence to society in broader areas. This position paper presents promising specific paths and considerations for getting closer to superforecaster-level AI technology, aiming to call for researchers' interest in these directions.