Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

TL;DR

This paper introduces a comprehensive dataset and a novel learning framework to evaluate and enhance the temporal reasoning abilities of large language models across various question answering settings.

Contribution

It presents TempReason, a new dataset for probing temporal reasoning, and a learning framework based on span extraction and reinforcement learning to improve models.

Findings

The dataset covers three levels of temporal reasoning.

The proposed framework improves temporal reasoning performance.

Experimental results validate the effectiveness of the approach.

Abstract

Reasoning about time is of fundamental importance. Many facts are time-dependent. For example, athletes change teams from time to time, and different government officials are elected periodically. Previous time-dependent question answering (QA) datasets tend to be biased in either their coverage of time spans or question types. In this paper, we introduce a comprehensive probing dataset \tempreason to evaluate the temporal reasoning capability of large language models. Our dataset includes questions of three temporal reasoning levels. In addition, we also propose a novel learning framework to improve the temporal reasoning capability of large language models, based on temporal span extraction and time-sensitive reinforcement learning. We conducted experiments in closed book QA, open book QA, and reasoning QA settings and demonstrated the effectiveness of our approach. Our code and data are released on https://github.com/DAMO-NLP-SG/TempReason.