ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning

TL;DR

ECONET is a continual pretraining method that enhances language models' ability to understand event temporal relations, significantly improving performance on related NLP tasks.

Contribution

The paper introduces a novel continual pretraining framework with self-supervised objectives specifically targeting event temporal reasoning in language models.

Findings

Improved performance on five relation extraction and question answering tasks.

Achieved new or on-par state-of-the-art results in most downstream tasks.

Reinforced models' attention to event and temporal information.

Abstract

While pre-trained language models (PTLMs) have achieved noticeable success on many NLP tasks, they still struggle for tasks that require event temporal reasoning, which is essential for event-centric applications. We present a continual pre-training approach that equips PTLMs with targeted knowledge about event temporal relations. We design self-supervised learning objectives to recover masked-out event and temporal indicators and to discriminate sentences from their corrupted counterparts (where event or temporal indicators got replaced). By further pre-training a PTLM with these objectives jointly, we reinforce its attention to event and temporal information, yielding enhanced capability on event temporal reasoning. This effective continual pre-training framework for event temporal reasoning (ECONET) improves the PTLMs' fine-tuning performances across five relation extraction and question answering tasks and achieves new or on-par state-of-the-art performances in most of our downstream tasks.