CLEVE: Contrastive Pre-training for Event Extraction

TL;DR

CLEVE introduces a contrastive pre-training framework that enhances event extraction by jointly learning event semantics and structures from large unsupervised data, significantly improving performance especially in unsupervised scenarios.

Contribution

This paper presents a novel contrastive pre-training approach for event extraction that models event characteristics and structures using both text and graph encoders, leveraging large-scale unsupervised data.

Findings

Significant improvements on ACE 2005 and MAVEN datasets.

Enhanced performance in unsupervised event extraction.

Effective joint learning of event semantics and structures.

Abstract

Event extraction (EE) has considerably benefited from pre-trained language models (PLMs) by fine-tuning. However, existing pre-training methods have not involved modeling event characteristics, resulting in the developed EE models cannot take full advantage of large-scale unsupervised data. To this end, we propose CLEVE, a contrastive pre-training framework for EE to better learn event knowledge from large unsupervised data and their semantic structures (e.g. AMR) obtained with automatic parsers. CLEVE contains a text encoder to learn event semantics and a graph encoder to learn event structures respectively. Specifically, the text encoder learns event semantic representations by self-supervised contrastive learning to represent the words of the same events closer than those unrelated words; the graph encoder learns event structure representations by graph contrastive pre-training on parsed event-related semantic structures. The two complementary representations then work together to improve both the conventional supervised EE and the unsupervised "liberal" EE, which requires jointly extracting events and discovering event schemata without any annotated data. Experiments on ACE 2005 and MAVEN datasets show that CLEVE achieves significant improvements, especially in the challenging unsupervised setting. The source code and pre-trained checkpoints can be obtained from https://github.com/THU-KEG/CLEVE.