Generic Temporal Reasoning with Differential Analysis and Explanation

TL;DR

This paper introduces TODAY, a new temporal reasoning task that tests models' understanding of how subtle contextual changes affect event temporal relations, revealing current models' reliance on superficial cues and proposing methods to improve reasoning capabilities.

Contribution

The paper presents TODAY, a novel benchmark for evaluating and training temporal reasoning models with differential analysis and explanation annotations, enhancing generalizability and reasoning accuracy.

Findings

Existing models, including GPT-3.5, perform at chance on TODAY.

TODAY's supervision and explanations improve model performance on benchmarks.

Models trained with TODAY's approach can leverage noisy supervision sources.

Abstract

Temporal reasoning is the task of predicting temporal relations of event pairs. While temporal reasoning models can perform reasonably well on in-domain benchmarks, we have little idea of these systems' generalizability due to existing datasets' limitations. In this work, we introduce a novel task named TODAY that bridges this gap with temporal differential analysis, which as the name suggests, evaluates whether systems can correctly understand the effect of incremental changes. Specifically, TODAY introduces slight contextual changes for given event pairs, and systems are asked to tell how this subtle contextual change would affect relevant temporal relation distributions. To facilitate learning, TODAY also annotates human explanations. We show that existing models, including GPT-3.5, drop to random guessing on TODAY, suggesting that they heavily rely on spurious information rather than proper reasoning for temporal predictions. On the other hand, we show that TODAY's supervision style and explanation annotations can be used in joint learning, encouraging models to use more appropriate signals during training and thus outperform across several benchmarks. TODAY can also be used to train models to solicit incidental supervision from noisy sources such as GPT-3.5, thus moving us more toward the goal of generic temporal reasoning systems.