Navigating Tomorrow: Reliably Assessing Large Language Models Performance on Future Event Prediction

TL;DR

This paper evaluates the ability of large language models to predict future events across various scenarios, highlighting their potential and limitations in predictive tasks.

Contribution

It introduces a new dataset and methodology for assessing LLMs in future event prediction, an under-explored application area.

Findings

LLMs show potential in future event prediction tasks.

Performance varies across different question types and scenarios.

Limitations identified suggest areas for future model improvements.

Abstract

Predicting future events is an important activity with applications across multiple fields and domains. For example, the capacity to foresee stock market trends, natural disasters, business developments, or political events can facilitate early preventive measures and uncover new opportunities. Multiple diverse computational methods for attempting future predictions, including predictive analysis, time series forecasting, and simulations have been proposed. This study evaluates the performance of several large language models (LLMs) in supporting future prediction tasks, an under-explored domain. We assess the models across three scenarios: Affirmative vs. Likelihood questioning, Reasoning, and Counterfactual analysis. For this, we create a dataset1 by finding and categorizing news articles based on entity type and its popularity. We gather news articles before and after the LLMs training cutoff date in order to thoroughly test and compare model performance. Our research highlights LLMs potential and limitations in predictive modeling, providing a foundation for future improvements.