Modeling Preconditions in Text with a Crowd-sourced Dataset

TL;DR

This paper introduces PeKo, a large crowd-sourced dataset for modeling preconditions between events in text, and presents challenge tasks that reveal the difficulty of capturing precondition relations with current language models.

Contribution

The paper provides the first large-scale annotated dataset for event preconditions and proposes two new challenge tasks to evaluate modeling preconditions in text.

Findings

Precondition modeling remains challenging for current language models.

Fine-tuning language models on PeKo improves precondition relation detection.

The dataset enables new research on event reasoning beyond causality and temporal relations.

Abstract

Preconditions provide a form of logical connection between events that explains why some events occur together and information that is complementary to the more widely studied relations such as causation, temporal ordering, entailment, and discourse relations. Modeling preconditions in text has been hampered in part due to the lack of large scale labeled data grounded in text. This paper introduces PeKo, a crowd-sourced annotation of preconditions between event pairs in newswire, an order of magnitude larger than prior text annotations. To complement this new corpus, we also introduce two challenge tasks aimed at modeling preconditions: (i) Precondition Identification -- a standard classification task defined over pairs of event mentions, and (ii) Precondition Generation -- a generative task aimed at testing a more general ability to reason about a given event. Evaluation on both tasks shows that modeling preconditions is challenging even for today's large language models (LM). This suggests that precondition knowledge is not easily accessible in LM-derived representations alone. Our generation results show that fine-tuning an LM on PeKo yields better conditional relations than when trained on raw text or temporally-ordered corpora.