A Comprehensive Evaluation of Large Language Models on Temporal Event Forecasting

TL;DR

This paper systematically evaluates large language models on temporal event forecasting, introduces a new benchmark dataset, and analyzes various methods, revealing insights into their strengths and limitations for this task.

Contribution

It constructs a high-quality benchmark dataset for temporal event forecasting and provides a comprehensive evaluation of LLM-based methods, highlighting key findings and future research directions.

Findings

Fine-tuning improves LLM performance significantly.

Retrieval modules help capture temporal patterns.

Popularity bias and long-tail issues persist in RAG methods.

Abstract

Recently, Large Language Models (LLMs) have demonstrated great potential in various data mining tasks, such as knowledge question answering, mathematical reasoning, and commonsense reasoning. However, the reasoning capability of LLMs on temporal event forecasting has been under-explored. To systematically investigate their abilities in temporal event forecasting, we conduct a comprehensive evaluation of LLM-based methods for temporal event forecasting. Due to the lack of a high-quality dataset that involves both graph and textual data, we first construct a benchmark dataset, named MidEast-TE-mini. Based on this dataset, we design a series of baseline methods, characterized by various input formats and retrieval augmented generation (RAG) modules. From extensive experiments, we find that directly integrating raw texts into the input of LLMs does not enhance zero-shot extrapolation performance. In contrast, fine-tuning LLMs with raw texts can significantly improve performance. Additionally, LLMs enhanced with retrieval modules can effectively capture temporal relational patterns hidden in historical events. However, issues such as popularity bias and the long-tail problem persist in LLMs, particularly in the retrieval-augmented generation (RAG) method. These findings not only deepen our understanding of LLM-based event forecasting methods but also highlight several promising research directions. We consider that this comprehensive evaluation, along with the identified research opportunities, will significantly contribute to future research on temporal event forecasting through LLMs.