Probing Language Models for Understanding of Temporal Expressions

TL;DR

This paper introduces three NLI challenge sets to evaluate language models' understanding of temporal expressions, revealing that large models lack deep comprehension of temporal relations despite some basic perception.

Contribution

The paper creates specialized NLI datasets to assess models' grasp of temporal concepts, highlighting gaps in current language model understanding.

Findings

Large models show limited understanding of temporal relations.

Fine-tuned models perceive temporal order but lack depth.

Models struggle with relations between different temporal units.

Abstract

We present three Natural Language Inference (NLI) challenge sets that can evaluate NLI models on their understanding of temporal expressions. More specifically, we probe these models for three temporal properties: (a) the order between points in time, (b) the duration between two points in time, (c) the relation between the magnitude of times specified in different units. We find that although large language models fine-tuned on MNLI have some basic perception of the order between points in time, at large, these models do not have a thorough understanding of the relation between temporal expressions.