ESTER: A Machine Reading Comprehension Dataset for Event Semantic Relation Reasoning

TL;DR

ESTER is a new comprehensive dataset designed to evaluate machine reading comprehension in understanding complex event semantic relations, emphasizing reasoning beyond temporal and argument-based tasks.

Contribution

The paper introduces ESTER, a novel dataset for event semantic relation reasoning that captures diverse event relations and challenges current state-of-the-art systems.

Findings

Current SOTA systems perform significantly below humans on ESTER.

ESTER covers five key event semantic relations with over 6K questions.

The dataset highlights the need for improved reasoning capabilities in machines.

Abstract

Understanding how events are semantically related to each other is the essence of reading comprehension. Recent event-centric reading comprehension datasets focus mostly on event arguments or temporal relations. While these tasks partially evaluate machines' ability of narrative understanding, human-like reading comprehension requires the capability to process event-based information beyond arguments and temporal reasoning. For example, to understand causality between events, we need to infer motivation or purpose; to establish event hierarchy, we need to understand the composition of events. To facilitate these tasks, we introduce ESTER, a comprehensive machine reading comprehension (MRC) dataset for Event Semantic Relation Reasoning. The dataset leverages natural language queries to reason about the five most common event semantic relations, provides more than 6K questions and captures 10.1K event relation pairs. Experimental results show that the current SOTA systems achieve 22.1%, 63.3%, and 83.5% for token-based exact-match, F1, and event-based HIT@1 scores, which are all significantly below human performances (36.0%, 79.6%, 100% respectively), highlighting our dataset as a challenging benchmark.