Are All Steps Equally Important? Benchmarking Essentiality Detection of Events

TL;DR

This paper investigates the importance of individual steps in achieving goals within event sequences, creating a new dataset and showing current models struggle to match human understanding of event essentiality.

Contribution

It introduces a high-quality annotated dataset of goal-step pairs for essentiality detection and evaluates existing models, revealing significant gaps in their understanding.

Findings

Humans have a consistent understanding of event essentiality.

Existing models underperform compared to human annotations.

The dataset and code are publicly available for further research.

Abstract

Natural language expresses events with varying granularities, where coarse-grained events (goals) can be broken down into finer-grained event sequences (steps). A critical yet overlooked aspect of understanding event processes is recognizing that not all step events hold equal importance toward the completion of a goal. In this paper, we address this gap by examining the extent to which current models comprehend the essentiality of step events in relation to a goal event. Cognitive studies suggest that such capability enables machines to emulate human commonsense reasoning about preconditions and necessary efforts of everyday tasks. We contribute a high-quality corpus of (goal, step) pairs gathered from the community guideline website WikiHow, with steps manually annotated for their essentiality concerning the goal by experts. The high inter-annotator agreement demonstrates that humans possess a consistent understanding of event essentiality. However, after evaluating multiple statistical and largescale pre-trained language models, we find that existing approaches considerably underperform compared to humans. This observation highlights the need for further exploration into this critical and challenging task. The dataset and code are available at http://cogcomp.org/page/publication_view/1023.