A Multi-Axis Annotation Scheme for Event Temporal Relations

TL;DR

This paper introduces a multi-axis annotation scheme for event temporal relations that improves inter-annotator agreement and simplifies the task by focusing on event start-points, facilitating crowdsourcing.

Contribution

The paper proposes a novel multi-axis modeling approach and a start-point-only annotation method to enhance TempRel annotation consistency and efficiency.

Findings

Inter-annotator agreement improved from 60s to 80s Cohen's Kappa

The new scheme enables effective crowdsourcing of annotations

Better-defined annotation scheme fosters further event understanding studies

Abstract

Existing temporal relation (TempRel) annotation schemes often have low inter-annotator agreements (IAA) even between experts, suggesting that the current annotation task needs a better definition. This paper proposes a new multi-axis modeling to better capture the temporal structure of events. In addition, we identify that event end-points are a major source of confusion in annotation, so we also propose to annotate TempRels based on start-points only. A pilot expert annotation using the proposed scheme shows significant improvement in IAA from the conventional 60's to 80's (Cohen's Kappa). This better-defined annotation scheme further enables the use of crowdsourcing to alleviate the labor intensity for each annotator. We hope that this work can foster more interesting studies towards event understanding.