Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives

TL;DR

This paper introduces Narrative-of-Thought (NoT), a prompting technique that enhances small language models' temporal reasoning by generating grounded narratives, narrowing the performance gap with larger models without finetuning.

Contribution

The paper proposes a novel prompting method, Narrative-of-Thought, that significantly improves temporal reasoning in small language models by guiding them to generate grounded narratives.

Findings

NoT improves F1 scores on temporal graph generation tasks.

NoT achieves state-of-the-art results on Schema-11 evaluation set.

NoT enhances structural similarity in temporal reasoning tasks.

Abstract

Reasoning about time and temporal relations is an integral aspect of human cognition, essential for perceiving the world and navigating our experiences. Though large language models (LLMs) have demonstrated impressive performance in many reasoning tasks, temporal reasoning remains challenging due to its intrinsic complexity. In this work, we first study an essential task of temporal reasoning -- temporal graph generation, to unveil LLMs' inherent, global reasoning capabilities. We show that this task presents great challenges even for the most powerful LLMs, such as GPT-3.5/4. We also notice a significant performance gap by small models (<10B) that lag behind LLMs by 50%. Next, we study how to close this gap with a budget constraint, e.g., not using model finetuning. We propose a new prompting technique tailored for temporal reasoning, Narrative-of-Thought (NoT), that first converts the events set to a Python class, then prompts a small model to generate a temporally grounded narrative, guiding the final generation of a temporal graph. Extensive experiments showcase the efficacy of NoT in improving various metrics. Notably, NoT attains the highest F1 on the Schema-11 evaluation set, while securing an overall F1 on par with GPT-3.5. NoT also achieves the best structural similarity across the board, even compared with GPT-3.5/4. Our code is available at https://github.com/launchnlp/NoT.