TIMEDIAL: Temporal Commonsense Reasoning in Dialog

TL;DR

This paper introduces TIMEDIAL, a new dataset and task to evaluate pre-trained language models' ability to perform temporal commonsense reasoning in dialogs, revealing significant gaps compared to human performance.

Contribution

The study presents the first benchmark for temporal reasoning in dialog contexts using pre-trained models, highlighting their limitations and guiding future research.

Findings

Models perform poorly compared to humans, with a 23-point accuracy gap.

Models rely on shallow cues rather than true temporal reasoning.

The dataset is publicly available for further research.

Abstract

Everyday conversations require understanding everyday events, which in turn, requires understanding temporal commonsense concepts interwoven with those events. Despite recent progress with massive pre-trained language models (LMs) such as T5 and GPT-3, their capability of temporal reasoning in dialogs remains largely under-explored. In this paper, we present the first study to investigate pre-trained LMs for their temporal reasoning capabilities in dialogs by introducing a new task and a crowd-sourced English challenge set, TIMEDIAL. We formulate TIME-DIAL as a multiple-choice cloze task with over 1.1K carefully curated dialogs. Empirical results demonstrate that even the best performing models struggle on this task compared to humans, with 23 absolute points of gap in accuracy. Furthermore, our analysis reveals that the models fail to reason about dialog context correctly; instead, they rely on shallow cues based on existing temporal patterns in context, motivating future research for modeling temporal concepts in text and robust contextual reasoning about them. The dataset is publicly available at: https://github.com/google-research-datasets/timedial.