Temporal Validity Change Prediction

TL;DR

This paper introduces a new NLP task called Temporal Validity Change Prediction, which assesses models' ability to detect how contextual information can alter the validity duration of statements, using a Twitter-based dataset.

Contribution

It proposes a novel task, creates a dataset from Twitter, benchmarks transformer models, and explores auxiliary training for improved prediction accuracy.

Findings

Transformer models can detect validity change with moderate accuracy.

Contextual information significantly influences validity duration predictions.

Auxiliary tasks improve model performance on validity change detection.

Abstract

Temporal validity is an important property of text that is useful for many downstream applications, such as recommender systems, conversational AI, or story understanding. Existing benchmarking tasks often require models to identify the temporal validity duration of a single statement. However, in many cases, additional contextual information, such as sentences in a story or posts on a social media profile, can be collected from the available text stream. This contextual information may greatly alter the duration for which a statement is expected to be valid. We propose Temporal Validity Change Prediction, a natural language processing task benchmarking the capability of machine learning models to detect contextual statements that induce such change. We create a dataset consisting of temporal target statements sourced from Twitter and crowdsource sample context statements. We then benchmark a set of transformer-based language models on our dataset. Finally, we experiment with temporal validity duration prediction as an auxiliary task to improve the performance of the state-of-the-art model.