LEAF: A Living Benchmark for Event-Augmented Forecasting
Mingtian Tan, Mihir Parmar, Palash Goyal, Chun-Liang Li, Nanyun Peng, Thomas Hartvigsen, Jinsung Yoon, and Tomas Pfister

TL;DR
LEAF is a novel living benchmark designed to evaluate large language models' ability to perform event-augmented forecasting in complex, real-world scenarios, incorporating dynamic, multidimensional event data.
Contribution
It introduces the first living benchmark for event-augmented forecasting, utilizing a recursive retrieval system and cross-validation to assess LLMs in real-world, complex environments.
Findings
LLMs can leverage complex event signals to improve predictions.
Models perform better on equities with higher predictability confidence.
Events are strongly correlated with target equities.
Abstract
Large Language Models (LLMs) are increasingly applied to forecasting. To evaluate this capability while mitigating pre-training data contamination, several living benchmarks have been proposed. However, existing benchmarks either lack the multidimensional events essential for accurate forecasting due to data scarcity, or focus on relatively closed environments. To assess the predictive capabilities of LLMs in complex, real-world scenarios, we propose LEAF, the first living benchmark for event-augmented forecasting tasks, including future event probabilities, trend and time series forecasting. LEAF utilizes a recursive retrieval agent system paired with dual-agent cross-validation to provide comprehensive and relevant auxiliary text for forecasting. Evaluating state-of-the-art proprietary and open-weight LLMs, we find that these models can leverage signals extracted from complex events…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
