ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification

TL;DR

ClarET is a versatile pre-trained transformer model designed for event-centric reasoning tasks, effectively capturing event correlations for improved generation and classification across diverse scenarios.

Contribution

The paper introduces ClarET, a novel correlation-aware pre-training framework with three event-centric objectives, enhancing event reasoning capabilities in various applications.

Findings

Outperforms existing models on 9 diverse benchmarks

Effective in zero- and few-shot learning scenarios

Demonstrates strong generalization across multiple reasoning types

Abstract

Generating new events given context with correlated ones plays a crucial role in many event-centric reasoning tasks. Existing works either limit their scope to specific scenarios or overlook event-level correlations. In this paper, we propose to pre-train a general Correlation-aware context-to-Event Transformer (ClarET) for event-centric reasoning. To achieve this, we propose three novel event-centric objectives, i.e., whole event recovering, contrastive event-correlation encoding and prompt-based event locating, which highlight event-level correlations with effective training. The proposed ClarET is applicable to a wide range of event-centric reasoning scenarios, considering its versatility of (i) event-correlation types (e.g., causal, temporal, contrast), (ii) application formulations (i.e., generation and classification), and (iii) reasoning types (e.g., abductive, counterfactual and ending reasoning). Empirical fine-tuning results, as well as zero- and few-shot learning, on 9 benchmarks (5 generation and 4 classification tasks covering 4 reasoning types with diverse event correlations), verify its effectiveness and generalization ability.