LEAF: A Living Benchmark for Event-Augmented Forecasting

TL;DR

LEAF is a novel living benchmark designed to evaluate large language models' ability to perform event-augmented forecasting in complex, real-world scenarios, incorporating dynamic, multidimensional event data.

Contribution

It introduces the first living benchmark for event-augmented forecasting, utilizing a recursive retrieval system and cross-validation to assess LLMs in real-world, complex environments.

Findings

LLMs can leverage complex event signals to improve predictions.

Models perform better on equities with higher predictability confidence.

Events are strongly correlated with target equities.

Abstract

Large Language Models (LLMs) are increasingly applied to forecasting. To evaluate this capability while mitigating pre-training data contamination, several living benchmarks have been proposed. However, existing benchmarks either lack the multidimensional events essential for accurate forecasting due to data scarcity, or focus on relatively closed environments. To assess the predictive capabilities of LLMs in complex, real-world scenarios, we propose LEAF, the first living benchmark for event-augmented forecasting tasks, including future event probabilities, trend and time series forecasting. LEAF utilizes a recursive retrieval agent system paired with dual-agent cross-validation to provide comprehensive and relevant auxiliary text for forecasting. Evaluating state-of-the-art proprietary and open-weight LLMs, we find that these models can leverage signals extracted from complex events to enhance predictive performance. In the stock domain, we find that LLMs achieve better performance on equities they confidently identify as more predictable. Furthermore, the events demonstrate a strong correlation with the target equities. To this end, LEAF provides a necessary, dynamically updating testbed to continuously track and drive progress in event-driven forecasting tasks.